Three-dimensional human body model parameter fitting method and device and electronic equipment

Abstract

The embodiment of the invention discloses a three-dimensional human body model parameter fitting method and device and electronic equipment. A specific embodiment of the method comprises the following steps: acquiring a multi-view human body image sequence and sensor data in an image acquisition process; determining a two-dimensional coordinate of each joint point in the multi-view human body image sequence; determining a three-dimensional coordinate of each joint point based on the two-dimensional coordinate of each joint point; and determining parameter information of the three-dimensional human body model based on the three-dimensional coordinates of the joint points and the sensor data. According to the embodiment, the multi-view human body image and the sensor data in the image acquisition process are combined, and the parameters of the three-dimensional human body model can be fitted more accurately.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, specifically to a method, apparatus, and electronic device for fitting parameters of a three-dimensional human body model. Background Technology

[0002] Using data-driven deep learning methods to estimate human pose in first-person view of VR (Virtual Reality) cameras has become a trend in the field. Therefore, it is necessary to construct large-scale, high-precision human motion datasets to support the model training process. Existing human motion capture datasets mostly use SMPL (Skinned Multi-Person Linear Model, a parametric model for 3D human motion capture) to represent human pose. This model can represent human poses of arbitrary body shape and movement, and the pose parameters can be used to drive the virtual avatar. Summary of the Invention

[0003] This disclosure is provided to briefly introduce the concepts, which will be described in detail in the subsequent Detailed Description section. This disclosure is not intended to identify key or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0004] In a first aspect, embodiments of this disclosure provide a method for fitting parameters of a three-dimensional human body model, comprising: acquiring a multi-view human body image sequence and sensor data during the image acquisition process; determining the two-dimensional coordinates of each joint point in the multi-view human body image sequence; determining the three-dimensional coordinates of each joint point based on the two-dimensional coordinates of each joint point; and determining the parameter information of the three-dimensional human body model based on the three-dimensional coordinates of each joint point and the sensor data.

[0005] Secondly, embodiments of this disclosure provide a three-dimensional human body model parameter fitting device, comprising: an acquisition unit for acquiring a multi-view human body image sequence and sensor data during the image acquisition process; a first determination unit for determining the two-dimensional coordinates of each joint point in the multi-view human body image sequence; a second determination unit for determining the three-dimensional coordinates of each joint point based on the two-dimensional coordinates of each joint point; and a third determination unit for determining the parameter information of the three-dimensional human body model based on the three-dimensional coordinates of each joint point and the sensor data.

[0006] Thirdly, embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the three-dimensional human body model parameter fitting method as described in the first aspect.

[0007] Fourthly, embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon that, when executed by a processor, implements the steps of the three-dimensional human body model parameter fitting method as described in the first aspect.

[0008] The three-dimensional human body model parameter fitting method, apparatus, and electronic device provided in this disclosure acquire multi-view human body image sequences and sensor data during the image acquisition process. Then, the two-dimensional coordinates of each joint point in the multi-view human body image sequence can be determined. Next, based on the two-dimensional coordinates of each joint point, the three-dimensional coordinates of each joint point can be determined. Finally, based on the three-dimensional coordinates of each joint point and the sensor data, the parameter information of the three-dimensional human body model can be determined. This method, combining multi-view human body images and sensor data during the image acquisition process, can more accurately fit the parameters of the three-dimensional human body model. Attached Figure Description

[0009] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0010] Figure 1 This is a flowchart of an embodiment of the three-dimensional human body model parameter fitting method according to the present disclosure;

[0011] Figure 2 This is a flowchart of yet another embodiment of the three-dimensional human body model parameter fitting method according to the present disclosure;

[0012] Figure 3 This is a flowchart of another embodiment of the three-dimensional human body model parameter fitting method according to the present disclosure;

[0013] Figure 4 This is a flowchart of yet another embodiment of the three-dimensional human body model parameter fitting method according to the present disclosure;

[0014] Figure 5 This is a flowchart of yet another embodiment of the three-dimensional human body model parameter fitting method according to the present disclosure;

[0015] Figure 6 This is a schematic diagram of a structure of an embodiment of the three-dimensional human body model parameter fitting device according to the present disclosure;

[0016] Figure 7 These are exemplary system architecture diagrams to which the various embodiments of this disclosure can be applied;

[0017] Figure 8This is a schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present disclosure. Detailed Implementation

[0018] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0019] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0020] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0021] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0022] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0023] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0024] Existing methods for fitting SMPL parameters include: human parametric posture fitting methods with labeled points and human parametric posture fitting methods without labeled points.

[0025] The marker-based human parametric pose fitting method requires the subject to wear a close-fitting motion capture suit with reflective spheres attached. During the initialization phase, the subject is required to perform specific movements, and the positional relationship between the human SMPL model and the reflective spheres is established. During motion capture, a light-based capture system tracks the position of each reflective sphere, thereby optimizing and fitting the human pose parameters for each frame.

[0026] However, this method cannot collect data on natural clothing, and the collection environment is limited to fixed indoor scenes, which restricts the diversity of foreground and background in the data, making it unsuitable for building datasets for vision-based human motion capture tasks.

[0027] Existing markerless human pose fitting schemes mainly suffer from the following two shortcomings:

[0028] First, considering the relatively far camera placement and resolution limitations during full-body image capture, it is difficult to accurately capture the poses of the hand and facial joints. Therefore, existing methods generally ignore the estimation of these joint poses, directly setting the local pose of the wrist and head nodes to 0. Furthermore, the lack of wrist node pose estimation also affects the accuracy of elbow and shoulder joint pose estimation.

[0029] Second, during the data acquisition process, self-occlusion may occasionally occur, causing some joints to become invisible. This results in the loss of some input information during the whole-body pose optimization, leading to unreasonable fitting results.

[0030] To address the aforementioned issues, this invention combines sensor information from XR (Extended Reality) devices with multi-view human images to propose a markerless, multimodal method for acquiring human motion capture data, thereby enabling more accurate fitting of 3D human model parameters.

[0031] XR technology is a collective term encompassing VR (VR), AR (Augmented Reality), and MR (Mixed Reality). XR technology uses computer graphics and wearable devices to integrate virtual information with the real world or completely create a virtual environment to achieve an immersive experience.

[0032] Please refer to Figure 1 The diagram illustrates a flowchart 100 of an embodiment of a three-dimensional human body model parameter fitting method according to the present disclosure. This three-dimensional human body model parameter fitting method includes the following steps:

[0033] Step 101: Acquire multi-view human body image sequences and sensor data during the image acquisition process.

[0034] In this embodiment, the execution subject of the three-dimensional human body model parameter fitting method (e.g.) Figure 7 The server shown can acquire multi-view human image sequences and sensor data during the image acquisition process. The aforementioned multi-view human image sequence can be a sequence composed of human images acquired from multiple perspectives in chronological order.

[0035] Sensor data can be acquired during image acquisition. This sensor data can be data collected by an IMU (Inertial Measurement Unit), specifically data collected by the controllers of an XR device and the IMU of a head-mounted display. Controllers are typically one of the primary input devices for user interaction with the virtual environment; they are usually equipped with sensors to detect the position and movement of the user's hands, enabling precise gesture control. Head-mounted displays are usually the core output device for the XR experience, directly displaying virtual images in front of the user.

[0036] An IMU (Instrument Measurer) is a device used to measure and report specific forces, angular velocities, and, in some cases, the orientation of an object around a magnetic field. An IMU typically consists of the following main components: an accelerometer, used to measure the acceleration of an object along the X, Y, and Z axes; a gyroscope, used to measure the angular velocity or angular displacement of an object relative to an inertial reference point; and a magnetometer, used to measure the orientation of an object relative to the Earth's magnetic field, providing a reference for the yaw angle.

[0037] Step 102: Determine the two-dimensional coordinates of each joint point in the multi-view human image sequence.

[0038] In this embodiment, the aforementioned execution entity can determine the two-dimensional coordinates of each joint point in the aforementioned multi-view human image sequence. Here, the execution entity can use HRNet (High-Resolution Network) to detect the two-dimensional coordinates of each joint point of the human body in the image. HRNet is an advanced deep learning network architecture whose core feature is maintaining high-resolution representation throughout the entire network structure, and enhancing model performance by connecting convolutional stream branches of different resolutions in parallel and continuously exchanging information between these branches.

[0039] It should be noted that any other method capable of detecting the two-dimensional coordinates of joints can also be used to determine the two-dimensional coordinates of each joint in a multi-view human image sequence, which will not be elaborated here.

[0040] Step 103: Determine the three-dimensional coordinates of each joint point based on the two-dimensional coordinates of each joint point.

[0041] In this embodiment, the execution entity can determine the three-dimensional coordinates of each joint point based on its two-dimensional coordinates. During the calibration process, camera intrinsic and extrinsic parameters, as well as the correspondence between multi-view images at the same time, can be obtained. The execution entity can then use a numerical triangulation method to convert the two-dimensional coordinates of each joint point into corresponding three-dimensional coordinates. The triangulation method calculates the depth information of each point by comparing the positional differences (parallax) between points in two images, thereby converting the two-dimensional coordinates into three-dimensional coordinates.

[0042] Step 104: Determine the parameter information of the three-dimensional human body model based on the three-dimensional coordinates of each joint and sensor data.

[0043] In this embodiment, the execution entity can determine the parameter information of the 3D human body model based on the 3D coordinates of each joint and the sensor data. As an example, the execution entity can input the 3D coordinates of each joint and the sensor data into a pre-trained 3D human body model parameter fitting model to obtain the parameter information of the 3D human body model. The 3D human body model parameter fitting model can be used to characterize the correspondence between the 3D coordinates and sensor data of each joint and the parameter information of the 3D human body model.

[0044] SMPL is a parametric model for 3D human motion capture that can represent different human shapes and poses and can deform naturally to demonstrate soft tissue movement. The model is built by learning from a large amount of 3D human scan data in various poses, enabling it to capture a range of natural shapes and postures.

[0045] The method provided in the above embodiments of this disclosure acquires multi-view human body image sequences and sensor data during the image acquisition process; then, the two-dimensional coordinates of each joint point in the multi-view human body image sequence can be determined; subsequently, the three-dimensional coordinates of each joint point can be determined based on the two-dimensional coordinates of each joint point; finally, the parameter information of the three-dimensional human body model can be determined based on the three-dimensional coordinates of each joint point and the sensor data. This approach, combining multi-view human body images and sensor data during the image acquisition process, can more accurately fit the parameters of the three-dimensional human body model.

[0046] In some alternative implementations, the execution entity can determine the parameter information of the 3D human body model based on the 3D coordinates of each joint and the sensor data in the following way: The execution entity can, based on the 3D coordinates of each joint and the sensor data, use the parameter information of the 3D human body model (e.g., posture parameters) as the optimization objective, and utilize a preset loss function to determine the parameter information of the 3D human body model when the loss function reaches its minimum value. Minimizing the loss function refers to adjusting the model parameters during model training to make the value of the loss function as small as possible, thereby accurately fitting the parameters of the 3D human body model.

[0047] In some alternative implementations, the aforementioned parameter information typically includes posture parameters and body shape parameters, and the aforementioned sensor data may include the degree of freedom information of the target joints. The aforementioned target joints may include at least one of the following: wrist joints, head joints, and knee joints.

[0048] Degrees of freedom (DoF) generally refer to the number of independent ways a system can move or rotate in space, including 3DoF and 6DoF. 3DoF typically refers to the degrees of freedom of an object's motion within a plane or line, including two translational degrees of freedom (e.g., left and right, forward and backward) and one rotational degree of freedom (e.g., rotation about an axis perpendicular to that plane). 6DoF refers to the total degrees of freedom of an object's motion in three-dimensional space, including three translational degrees of freedom (up and down, left and right, forward and backward) and three rotational degrees of freedom (rotation about the X, Y, and Z axes).

[0049] The aforementioned execution entity can determine the parameter information of the three-dimensional human body model when the loss function reaches its minimum value by using the parameter information of the three-dimensional human body model as the optimization target, based on the three-dimensional coordinates of the aforementioned joints and the aforementioned sensor data, using a preset loss function: The aforementioned execution entity can first determine the first difference value based on the mapped coordinates of the aforementioned joints and the three-dimensional coordinates of the aforementioned joints. The aforementioned mapped coordinates are usually obtained by mapping the parameter information of the three-dimensional human body model to three-dimensional space.

[0050] Subsequently, a second difference value can be determined based on the global pose information and the degrees of freedom information of the target joint. The global pose information can be determined based on the local pose information of the target joint. Here, the degrees of freedom information of the target joint is typically the 3DoF of the target joint.

[0051] Local pose and global pose are commonly used to describe the position and orientation of an object in space. Specifically, local pose typically refers to the position and orientation of an object relative to its local coordinate system. For example, local pose could refer to the position and orientation of a subsequent joint relative to its preceding joint. Global pose typically refers to the position and orientation of an object relative to the world coordinate system or the entire scene. It is an absolute position, indicating the object's exact location within the environment. Pose is typically represented by rotation matrices, quaternions, or Euler angles, while position is represented by vectors. Pose transformations can be combined using matrix multiplication to compute the global pose from the local pose.

[0052] Then, based on the aforementioned first and second difference values, and using the parameter information of the 3D human body model as the optimization target, a preset loss function can be used to determine the parameter information of the 3D human body model when the loss function reaches its minimum value. Specifically, the executing entity can substitute the aforementioned first and second difference values ​​into the aforementioned loss function, adjust the parameter information of the 3D human body model to minimize the loss function, and output the parameter information of the 3D human body model when the loss function reaches its minimum value.

[0053] As an example, the loss function can be expressed as follows (1):

[0054]

[0055] Among them, E smpl Let τ1 and τ2 be the loss function values, θ be the pose parameter, β be the body shape parameter, N be the number of frames in the human image sequence, n be the nth frame, and P be the body shape parameter. 3D Let j ∈ {4, 5, 15, 20, 21} represent the 3D coordinates of this joint point, where j ∈ {4, 5, 15, 20, 21} represents the human joint point number corresponding to the IMU sensor data. Represents the 3DoF rotation matrix of the head joints. A 3DoF rotation matrix representing the wrist joint. 6DoF posture of the wrist joint. The 6DoF pose of the head joints. Φ(·) represents the 3DoF rotation matrix of the knee joint, Φ(·) represents the mapping from SMPL parameters to the 3D joints of the human body, and Π(·) represents the forward calculation process from local pose to global pose.

[0056] This method can accurately fit the postures of the wrist, head, and knee joints, further improving the accuracy of posture estimation for the elbow and shoulder joints.

[0057] In some optional implementations, since joint detection is a single-frame process, temporal jitter may occur. To increase the smoothness of joints, temporal filtering can be applied to each joint. Here, the execution entity can determine a third difference value based on the pose parameters of the next frame and the pose parameters of the current frame, thus ensuring that the difference in pose parameters between adjacent frames is not too large. Then, based on the first, second, and third difference values, and using the parameter information of the 3D human model as the optimization target, the execution entity can determine the parameter information of the 3D human model when the loss function reaches its minimum value using a preset loss function. Specifically, the execution entity can substitute the first, second, and third difference values ​​into the loss function to adjust the parameter information of the 3D human model until the loss function reaches its minimum value, and then output the parameter information of the 3D human model when the loss function reaches its minimum value.

[0058] As an example, the loss function can be expressed as follows (2):

[0059]

[0060] Among them, E smpl Here, τ1, τ2, and τ3 are the loss function values, τ1, τ2, and τ3 are constant coefficients, and θ is the attitude parameter. n Let θ be the attitude parameter corresponding to the current frame (the nth frame). n+1 Let β be the pose parameter corresponding to the frame following the nth frame, β be the body shape parameter, N be the frame number of the human image sequence, n be the nth frame, and P be the body shape parameter. 3D Let j ∈ {4, 5, 15, 20, 21} represent the 3D coordinates of this joint point, where j ∈ {4, 5, 15, 20, 21} represents the human joint point number corresponding to the IMU sensor data. Represents the 3DoF rotation matrix of the head joints. A 3DoF rotation matrix representing the wrist joint. 6DoF posture of the wrist joint. The 6DoF pose of the head joints. Φ(·) represents the 3DoF rotation matrix of the knee joint, Φ(·) represents the mapping from SMPL parameters to the 3D joints of the human body, and Π(·) represents the forward calculation process from local pose to global pose.

[0061] By performing timing filtering on each joint, the problem of timing jitter can be solved, and the smoothness of the joint can be increased.

[0062] In some optional implementations, the aforementioned execution entity can determine the three-dimensional coordinates of each joint point based on its two-dimensional coordinates as follows: The execution entity can use the two-dimensional coordinates of each joint point to determine its initial three-dimensional coordinates. During calibration, camera intrinsic and extrinsic parameters, as well as the correspondence between multi-view images at the same time, can be obtained. The execution entity can then use a numerical triangulation method to convert the two-dimensional coordinates of each joint point into their corresponding three-dimensional coordinates. The triangulation method calculates the depth information of each point by comparing the positional differences between points in two images, thereby converting the two-dimensional coordinates into three-dimensional coordinates.

[0063] Subsequently, a fifth difference value can be determined based on the projected coordinates and two-dimensional coordinates of each joint point. The projected coordinates can be obtained by projecting the initial three-dimensional coordinates onto the human body images acquired from each viewpoint. Here, the executing entity can, for each of the multiple viewpoints, project the initial three-dimensional coordinates of the joint point onto the human body image acquired from that viewpoint, and compare the projected coordinates with the two-dimensional coordinates representing the joint point in the human body image from that viewpoint to obtain the fifth difference value.

[0064] Then, based on the fifth difference value mentioned above, the three-dimensional coordinates of each joint point can be used as the optimization target, and the three-dimensional coordinates when the loss function reaches its minimum value can be determined using the preset loss function as the three-dimensional coordinates of each joint point.

[0065] As an example, the loss function can be the following formula (3):

[0066]

[0067] Among them, E 3D Let λ be the loss function value, λ1 be a constant coefficient, V represent the total number of viewpoints, v be the v-th viewpoint, N be the number of frames in the human image sequence, and n be the n-th frame. Here are the 3D coordinates of the joint point corresponding to the nth frame. Let K(·) be the 2D coordinate of the joint point in the nth frame from the vth viewpoint, and let K(·) represent the projection of the spatial 3D coordinates onto the camera's 2D coordinates.

[0068] In this way, the initial three-dimensional coordinates of the joint are projected onto the acquired human body image. The projected coordinates are compared with the two-dimensional coordinates representing the joint in the human body image to obtain the fifth difference value. The three-dimensional coordinates when the fifth difference value reaches the minimum value are determined as the three-dimensional coordinates of each joint, so that the three-dimensional coordinates of each joint can be determined more accurately.

[0069] In some optional implementations, since the joint detection process is single-frame, timing jitter may occur. To increase the smoothness of the joints, temporal filtering can be applied to each joint. Here, the execution entity can determine the sixth difference value based on the initial 3D coordinates of the next frame and the initial 3D coordinates of the current frame, thus ensuring that the difference in pose parameters between adjacent frames is not too large. Then, based on the fifth and sixth difference values, the execution entity can use a preset loss function, with the 3D coordinates of each joint as the optimization target, to determine the 3D coordinates of each joint when the loss function reaches its minimum value. Specifically, the execution entity can substitute the fifth and sixth difference values ​​into the loss function, adjust the 3D coordinates of each joint to minimize the loss function, and output the 3D coordinates of each joint when the loss function reaches its minimum value.

[0070] As an example, the loss function can be the following formula (4):

[0071]

[0072] Among them, E 3D Here, λ1, λ2, and λ3 are the loss function values, λ1, λ2, and λ3 are constant coefficients, V represents the total number of viewpoints, v is the v-th viewpoint, N is the number of frames in the human image sequence, and n is the n-th frame. Here are the 3D coordinates of the joint point corresponding to the nth frame. Let K(·) represent the 2D coordinates of the joint point in the nth frame from the vth viewpoint, and let K(·) represent the projection of the spatial 3D coordinates onto the camera's 2D coordinates. The coordinates of the joint point corresponding to the (n+1)th frame are given.

[0073] By performing timing filtering on each joint, the problem of timing jitter can be solved, and the smoothness of the joint can be increased.

[0074] Continue to refer to Figure 2 This illustrates a flowchart 200 of another embodiment of a three-dimensional human body model parameter fitting method. The flowchart 200 of this three-dimensional human body model parameter fitting method includes the following steps:

[0075] Step 201: Acquire multi-view human body image sequences and sensor data during the image acquisition process.

[0076] Step 202: Determine the two-dimensional coordinates of each joint point in the multi-view human image sequence.

[0077] Step 203: Determine the three-dimensional coordinates of each joint point based on the two-dimensional coordinates of each joint point.

[0078] In this embodiment, steps 201-203 can be performed in a similar manner to steps 101-103, and will not be described again here.

[0079] Step 204: Determine the first difference value based on the mapped coordinates and three-dimensional coordinates of each joint.

[0080] In this embodiment, the execution entity of the three-dimensional human body model parameter fitting method can determine the first difference value based on the mapped coordinates and three-dimensional coordinates of each joint point. The mapped coordinates are typically obtained by mapping the parameter information of the three-dimensional human body model to three-dimensional space.

[0081] Step 205: Determine the second difference value based on the global pose information and the degree of freedom information of the target joint.

[0082] The aforementioned parameter information typically includes posture parameters and body shape parameters. The aforementioned sensor data may include the degree of freedom information of the target joints. The aforementioned target joints may include at least one of the following: wrist joints, head joints, and knee joints.

[0083] Degrees of freedom (DOF) generally refer to the number of independent ways a system can move or rotate in space, including 3DoF and 6DoF. 3DoF typically refers to the degrees of freedom of an object's motion within a plane or along a line, including two translational degrees of freedom (e.g., left and right, forward and backward) and one rotational degree of freedom (e.g., rotation about an axis perpendicular to that plane). 6DoF refers to the total degrees of freedom of an object's motion in three-dimensional space, including three translational degrees of freedom (up and down, left and right, forward and backward) and three rotational degrees of freedom (rotation about the X, Y, and Z axes).

[0084] In this embodiment, the execution entity can determine the second difference value based on the global pose information and the degree-of-freedom information of the target joint. The global pose information can be determined based on the local pose information of the target joint. Here, the degree-of-freedom information of the target joint is typically the 3DoF of the target joint.

[0085] Local pose and global pose are commonly used to describe the position and orientation of an object in space. Specifically, local pose typically refers to the position and orientation of an object relative to its local coordinate system. For example, local pose could refer to the position and orientation of a subsequent joint relative to its preceding joint. Global pose typically refers to the position and orientation of an object relative to the world coordinate system or the entire scene. It is an absolute position, indicating the object's exact location within the environment. Pose is typically represented by rotation matrices, quaternions, or Euler angles, while position is represented by vectors. Pose transformations can be combined using matrix multiplication to compute the global pose from the local pose.

[0086] Step 206: Determine the third difference value based on the attitude parameters corresponding to the next frame and the attitude parameters corresponding to the current frame.

[0087] In this embodiment, since the joint point detection process is a single frame, timing jitter will occur. In order to increase the smoothness of the joint point, timing filtering can be performed on each joint point. Here, the execution entity can determine the third difference value based on the attitude parameters corresponding to the next frame and the attitude parameters corresponding to the current frame, so that the difference between the attitude parameters corresponding to the two adjacent frames cannot be too large.

[0088] Step 207: Determine the fourth difference value based on the length of each bone segment and the standard length of each bone segment.

[0089] In this embodiment, the execution entity can determine the fourth difference value based on the length of each bone segment and the standard length of each bone segment. The length of each bone segment can be determined using the aforementioned body shape parameters. Using the aforementioned body shape parameters, the distance between two adjacent joint points can be determined as the bone length between these two joint points. The standard length of each bone segment can be the average length of bones in a preset number of target frames. As an example, the preset number can be the first 30 frames.

[0090] The human figure in the aforementioned target frame is typically in a preset standard pose, such as a T-pose. A T-pose usually refers to the default pose of a 3D model without any animation or pose adjustments. In a T-pose, the model's arms are usually outstretched, forming a "T" shape, and the legs are straight. This aligns the model's bones and joints, making it easier for animators to create animations. The T-pose is the starting point in the animation workflow; animators begin by adding actions and expressions to the model from this pose.

[0091] Step 208: Based on the first difference value, the second difference value, the third difference value, and the fourth difference value, with the parameter information of the three-dimensional human body model as the optimization target, and using the preset loss function, determine the parameter information of the three-dimensional human body model when the loss function reaches its minimum value.

[0092] In this embodiment, the execution entity can, based on the first, second, third, and fourth difference values, and using the parameter information of the 3D human body model as the optimization target, determine the parameter information of the 3D human body model when the loss function reaches its minimum value using a preset loss function. Specifically, the execution entity can substitute the first, second, third, and fourth difference values ​​into the loss function to adjust the parameter information of the 3D human body model so that the loss function reaches its minimum value, and then output the parameter information of the 3D human body model when the loss function reaches its minimum value.

[0093] As an example, the loss function can be expressed as follows (5):

[0094]

[0095] Among them, E smpl Here, τ1, τ2, τ3, τ4 are the loss function values, θ is the pose parameter, β is the body shape parameter, N is the number of frames in the human image sequence, n is the nth frame, and P... 3D Let j ∈ {4, 5, 15, 20, 21} represent the 3D coordinates of this joint point, where j ∈ {4, 5, 15, 20, 21} represents the human joint point number corresponding to the IMU sensor data. Represents the 3DoF rotation matrix of the head joints. A 3DoF rotation matrix representing the wrist joint. 6DoF posture of the wrist joint. The 6DoF pose of the head joints. The 3DoF rotation matrix of the knee joint, β avg Φ(·) represents the standard length of the bone, Φ(·) represents the mapping from SMPL parameters to human 3D joints, and Π(·) represents the forward calculation process from local pose to global pose.

[0096] from Figure 2 It can be seen from this that, with Figure 1Compared to the corresponding embodiments, the flowchart 200 of the three-dimensional human body model parameter fitting method in this embodiment embodies the steps of using the parameter information of the three-dimensional human body model as the optimization target based on the first difference value, the second difference value, the third difference value, and the fourth difference value, and using a preset loss function to determine the parameter information of the three-dimensional human body model when the loss function reaches its minimum value. Therefore, the scheme described in this embodiment can incorporate 3DoF-IMU data from at least one of the wrist joint points, head joint points, and knee joint points into the optimization term, avoiding unreasonable posture results. Furthermore, by performing temporal filtering on each joint point, the problem of temporal jitter can be solved, increasing the smoothness of the joint points. Additionally, based on the fact that the bone length of the same data collector remains constant, a bone length constraint is added during the optimization process to ensure consistent bone length throughout the multi-view human body image sequence. This improves the optimization effect and further enhances the fitting effect of the three-dimensional human body model parameters.

[0097] In some alternative implementations, the 3D coordinates of the target joints can be the rotation values ​​from their degrees of freedom information. When self-occlusion causes wrist or head joints to be invisible in the third-view camera, the rotation values ​​from the wrist joint's degrees of freedom information can be used instead of its 3D coordinates, and the rotation values ​​from the head joint's degrees of freedom information can be used instead of its 3D coordinates. This approach can resolve the problem of unreasonable fitting results caused by some joints being invisible due to self-occlusion.

[0098] In some alternative implementations, the aforementioned sensor data may include the degrees of freedom information of the extended reality device's handle and the head-mounted device. The degrees of freedom information of the wrist joint can be determined based on the handle's degrees of freedom information. Specifically, the wrist joint's degrees of freedom information can be determined using the following formula (6):

[0099] T wrist =T h2w *T handl e (6)

[0100] Among them, T wrist For the 6DoF posture of the wrist joint, T h2w T is the transformation matrix from the handle to the wrist joint, calculated under the correct grip condition. handle This is the 6DoF orientation of the controller.

[0101] The degrees of freedom information of the aforementioned head joints can be determined based on the degrees of freedom information of the aforementioned head-mounted device. Specifically, the degrees of freedom information of the aforementioned head joints can be determined using the following formula (7):

[0102] T head =T h2h *T HMD (7)

[0103] Among them, T head For the head joint 6DoF pose, T h2h T is the transformation matrix from the head-mounted device to the head joints, calculated under the T-Pose posture. HMD The 6DoF orientation for the head-mounted device.

[0104] Because the poses of hand and head joints cannot be accurately captured during full-body motion capture, the wrist and head joint poses are inaccurate when fitting SMPL parameters. This method allows for a more accurate fitting of wrist and head joint poses.

[0105] See further Figure 3 , Figure 3 This is a flow 300 of another embodiment of the three-dimensional human body model parameter fitting method according to this embodiment. The flow 300 of this three-dimensional human body model parameter fitting method includes the following steps:

[0106] Step 301: Acquire multi-view human body image sequences and sensor data during the image acquisition process.

[0107] Step 302: Determine the two-dimensional coordinates of each joint point in the multi-view human image sequence.

[0108] Step 303: Determine the three-dimensional coordinates of each joint point based on the two-dimensional coordinates of each joint point.

[0109] Step 304: Determine the first difference value based on the mapped coordinates and three-dimensional coordinates of each joint.

[0110] Step 305: Determine the second difference value based on the global pose information and the degree of freedom information of the target joint.

[0111] Step 306: Determine the third difference value based on the attitude parameters corresponding to the next frame and the attitude parameters corresponding to the current frame.

[0112] Step 307: Determine the fourth difference value based on the length of each bone segment and the standard length of each bone segment.

[0113] Step 308: Based on the first difference value, the second difference value, the third difference value, and the fourth difference value, with the parameter information of the three-dimensional human body model as the optimization target, and using the preset loss function, determine the parameter information of the three-dimensional human body model when the loss function reaches its minimum value.

[0114] In this embodiment, steps 301-308 can be performed in a similar manner to steps 201-208, and will not be described again here.

[0115] Step 309: Using the parameter information of the three-dimensional human body model, determine the local pose information of the wrist joint points of a preset number of target frames.

[0116] Here, the target joint mentioned above is usually the wrist joint.

[0117] In this embodiment, the execution entity of the three-dimensional human body model parameter fitting method can use the parameter information of the three-dimensional human body model to determine the local posture information of the wrist joint points of a preset number of target frames.

[0118] The human pose in the target frame mentioned above is usually a preset standard pose, such as a T-pose. A T-pose typically refers to the default pose of a 3D model when no animation or pose adjustment has been performed. In a T-pose, the model's arms are usually outstretched, forming a "T" shape, and the legs are straight. This aligns the model's bones and joints, making it easier for animators to create animations.

[0119] As an example, during the initialization process, the acquisition is specified to begin with a T-Pose posture, meaning the actual LocalPose of the wrist node is 0. The LocalPose of the wrist is determined using the initial fitting results of the first 30 frames.

[0120] Step 310: Determine the compensation matrix based on the local pose information.

[0121] In this embodiment, the execution entity can determine the compensation matrix based on the aforementioned local pose information. Here, the compensation matrix can be obtained by taking the inverse average of the local pose of the wrist joint.

[0122] As an example, the compensation matrix can be determined using the following formula (8):

[0123]

[0124] Among them, R comp Characterization of the compensation matrix, R local,wrist The local pose of the wrist joint is represented by i, which is the i-th frame.

[0125] Step 311: Use the compensation matrix to calibrate the degree of freedom information of the handle.

[0126] In this embodiment, the aforementioned execution entity can use the aforementioned compensation matrix to calibrate the degree of freedom information of the handle.

[0127] As an example, the degrees of freedom information of the handle can be calibrated using the following formula (9):

[0128]

[0129] Among them, R comp Characterization compensation matrix, The 3DoF rotation matrix, R, characterizes the calibrated handle. handle The original 3DoF rotation matrix characterizing the handle.

[0130] Step 312: Using the calibrated handle's degree of freedom information, determine the calibrated wrist joint's degree of freedom information.

[0131] In this embodiment, the aforementioned execution entity can utilize the calibrated handle's degree-of-freedom information to determine the calibrated wrist joint's degree-of-freedom information. Specifically, the calibrated wrist joint's degree-of-freedom information can be determined using the following formula (10):

[0132]

[0133] in, For the calibrated 3DoF posture of the wrist joint, T h2w This is the transformation matrix from the handle to the wrist joint. The 3DoF orientation of the calibrated handle.

[0134] Step 313: Based on the global pose information of the wrist joint and the calibrated degree of freedom information of the wrist joint, redetermine the second difference value.

[0135] In this embodiment, the execution entity can redetermine the second difference value based on the global pose information of the wrist joint and the calibrated degree-of-freedom information of the wrist joint. Specifically, the 3DoF rotation matrix of the calibrated wrist joint can be substituted into formula (1), formula (2), or formula (5).

[0136] Step 314: Replace the initial second difference value with the newly determined second difference value. Using the parameter information of the three-dimensional human body model as the optimization target, and using the preset loss function, redetermine the parameter information of the three-dimensional human body model when the loss function reaches its minimum value.

[0137] In this embodiment, the execution entity can replace the initial second difference value with a newly determined second difference value, take the parameter information of the three-dimensional human body model as the optimization target, and use a preset loss function to redetermine the parameter information of the three-dimensional human body model when the loss function reaches its minimum value.

[0138] Specifically, the aforementioned executing entity can substitute the calibrated 3DoF rotation matrix of the wrist joint into formula (1), formula (2), or formula (5) above. Then, the parameter information of the three-dimensional human body model when the above formula (1), formula (2), or formula (5) reaches the minimum value is redefined, and the final three-dimensional human body model parameter result is output.

[0139] from Figure 3 It can be seen from this that, with Figure 1 Compared to the corresponding embodiment, the flow 300 of the three-dimensional human body model parameter fitting method in this embodiment embodies the steps of calibrating the IMU data of the handle and refitting the three-dimensional human body model parameters. In actual use, due to different handholding habits of data collectors, a unified transformation matrix T is used. h2w There will be calculation biases. The scheme described in this embodiment specifies that the subject starts with a T-Pose posture during the acquisition initialization process, that is, the actual LocalPose of the wrist node is 0. The freedom information of the handle is calibrated by the LocalPose of the wrist based on the initial fitting results of the first N frames, thereby solving the problem of using a unified transformation matrix T. h2w There is a problem with calculation bias. By using the calibrated handle's degree of freedom information, the calibrated wrist joint's degree of freedom information is determined, and this information is then substituted into the loss function to obtain the 3D human body model parameters, thereby further improving the fitting results of the 3D human body model parameters.

[0140] See also Figure 4 , Figure 4 This is a further embodiment of the three-dimensional human body model parameter fitting method according to this embodiment, process 400. The three-dimensional human body model parameter fitting method process 400 includes the following steps:

[0141] Step 401: Acquire multi-view human body image sequences and sensor data during the image acquisition process.

[0142] Step 402: Determine the two-dimensional coordinates of each joint point in the multi-view human body image sequence.

[0143] In this embodiment, steps 401-402 can be performed in a similar manner to steps 101-102, and will not be described again here.

[0144] Step 403: Use the two-dimensional coordinates of each joint point to determine the initial three-dimensional coordinates of each joint point.

[0145] In this embodiment, the execution entity can use the two-dimensional coordinates of each joint point to determine the initial three-dimensional coordinates of each joint point. During the calibration process, camera intrinsic and extrinsic parameters, as well as the correspondence between multi-view images at the same time, can be obtained. The execution entity can then use a numerical triangulation method to convert the two-dimensional coordinates of each joint point into corresponding three-dimensional coordinates. The triangulation method calculates the depth information of each point by comparing the positional differences between points in two images, thereby converting the two-dimensional coordinates into three-dimensional coordinates.

[0146] Step 404: Determine the fifth difference value based on the projected coordinates and two-dimensional coordinates of each joint.

[0147] In this embodiment, the execution entity can determine the fifth difference value based on the projected coordinates and two-dimensional coordinates of each joint. The projected coordinates can be obtained by projecting the initial three-dimensional coordinates onto human body images acquired from various viewpoints.

[0148] Here, the aforementioned execution entity can project the initial three-dimensional coordinates of the joint point onto the human body image acquired under each of the multiple perspectives, and compare the projected coordinates with the two-dimensional coordinates representing the joint point in the human body image under that perspective to obtain the fifth difference value.

[0149] Step 405: Determine the sixth difference value based on the initial three-dimensional coordinates corresponding to the next frame and the initial three-dimensional coordinates corresponding to the current frame.

[0150] In this embodiment, the execution entity can determine the sixth difference value based on the initial three-dimensional coordinates corresponding to the next frame and the initial three-dimensional coordinates corresponding to the current frame, so that the difference in the attitude parameters corresponding to the two adjacent frames cannot be too large.

[0151] Step 406: Determine the seventh difference value based on the length of each bone segment and the standard length of each bone segment.

[0152] In this embodiment, the execution entity can determine the seventh difference value based on the length of each bone segment and the standard length of each bone segment. The length of each bone segment can be determined using the distance between two adjacent joint points, and the standard length of each bone segment can be the average length of the bones in a preset number of frames. As an example, the preset number of frames can be the first 30 frames.

[0153] The human poses in the aforementioned preset number of frames are typically preset standard poses, such as the T-pose. A T-pose usually refers to the default pose of a 3D model without any animation or pose adjustments. In a T-pose, the model's arms are usually outstretched, forming a "T" shape, and the legs are straight. This aligns the model's bones and joints, making it easier for animators to create animations. The T-pose is the starting point in the animation workflow; animators begin by adding actions and expressions to the model from this pose.

[0154] Step 407: Based on the fifth, sixth, and seventh difference values, and using the three-dimensional coordinates of each joint as the optimization target, the three-dimensional coordinates at which the loss function reaches its minimum value are determined using a preset loss function as the three-dimensional coordinates of each joint.

[0155] In this embodiment, the execution entity can, based on the fifth, sixth, and seventh difference values, use the three-dimensional coordinates of each joint point as the optimization target and a preset loss function to determine the three-dimensional coordinates of each joint point when the loss function reaches its minimum value. Specifically, the execution entity can substitute the fifth, sixth, and seventh difference values ​​into the loss function, adjust the three-dimensional coordinates of each joint point to minimize the loss function, and output the three-dimensional coordinates of each joint point when the loss function reaches its minimum value.

[0156] As an example, the loss function can be the following formula (11):

[0157]

[0158] Among them, E 3D Here, λ1, λ2, and λ3 are the loss function values, λ1, λ2, and λ3 are constant coefficients, V represents the total number of viewpoints, v is the v-th viewpoint, N is the number of frames in the human image sequence, and n is the n-th frame. Here are the 3D coordinates of the joint point corresponding to the nth frame. Let K(·) represent the 2D coordinates of the joint point in the nth frame from the vth viewpoint, and let K(·) represent the projection of the spatial 3D coordinates onto the camera's 2D coordinates. B represents the 3D coordinates of the joint point corresponding to the (n+1)th frame. n B is the bone length of the nth frame. avg This represents the average bone length.

[0159] Step 408: Determine the parameter information of the three-dimensional human body model based on the three-dimensional coordinates of each joint and sensor data.

[0160] In this embodiment, step 408 can be performed in a similar manner to step 104, and will not be described again here.

[0161] from Figure 4 It can be seen from this that, with Figure 1 Compared to the corresponding embodiments, the flow 400 of the three-dimensional human body model parameter fitting method in this embodiment embodies the steps of using the three-dimensional coordinates of each joint point as the optimization target based on the aforementioned fifth, sixth, and seventh difference values, and using a preset loss function to determine the three-dimensional coordinates of each joint point when the loss function reaches its minimum value. Therefore, the scheme described in this embodiment can solve the problem of temporal jitter by performing temporal filtering on each joint point, increasing the smoothness of the joint points. Furthermore, based on the fact that the bone length of the same data acquisition subject remains constant, a bone length constraint is added during the optimization process to ensure that the bone length is consistent throughout the entire multi-view human body image sequence. Thus, the three-dimensional coordinates can be determined more accurately.

[0162] See also Figure 5 , Figure 5 This is a flow chart of another embodiment of the three-dimensional human body model parameter fitting method according to this embodiment. This flow chart can represent an optional flow chart when the three-dimensional human body model parameter fitting method is applied in a specific application scenario. The flow chart is summarized as follows:

[0163] First, acquire multi-view RGB images and IMU data, and perform 2D joint detection on the multi-view RGB images. Calculate 3D joints using the detected 2D joints. Then, use the 3D joints and IMU data to fit the SMPL parameters to obtain the initial SMPL parameters. After that, calibrate the IMU data, and use the calibrated IMU data and 3D joints to refit the SMPL parameter results.

[0164] Further reference Figure 6 As an implementation of the methods shown in the above figures, this application provides an embodiment of a three-dimensional human body model parameter fitting device, which is similar to... Figure 1 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0165] like Figure 6As shown, the three-dimensional human body model parameter fitting device 600 of this embodiment includes: an acquisition unit 601, a first determination unit 602, a second determination unit 603, and a third determination unit 604. The acquisition unit 601 is used to acquire multi-view human body image sequences and sensor data during the image acquisition process; the first determination unit 602 is used to determine the two-dimensional coordinates of each joint point in the multi-view human body image sequence; the second determination unit 603 is used to determine the three-dimensional coordinates of each joint point based on the two-dimensional coordinates; and the third determination unit 604 is used to determine the parameter information of the three-dimensional human body model based on the three-dimensional coordinates of each joint point and the sensor data.

[0166] In this embodiment, the specific processing of the acquisition unit 601, the first determination unit 602, the second determination unit 603, and the third determination unit 604 of the three-dimensional human body model parameter fitting device 600 can be referred to Figure 1 The corresponding steps are 101, 102, 103 and 104 in the embodiment.

[0167] In some optional implementations, the third determining unit 604 can determine the parameter information of the three-dimensional human body model based on the three-dimensional coordinates of each joint and the sensor data in the following way: based on the three-dimensional coordinates of each joint and the sensor data, with the parameter information of the three-dimensional human body model as the optimization target, and using a preset loss function, determine the parameter information of the three-dimensional human body model when the loss function reaches its minimum value.

[0168] In some optional implementations, the aforementioned parameter information includes posture parameters and body shape parameters, the aforementioned sensor data includes the degree of freedom information of the target joints, and the aforementioned target joints include at least one of the following: wrist joint, head joint, and knee joint; and the aforementioned third determining unit 604 can determine the parameter information of the three-dimensional human body model when the loss function reaches its minimum value by using a preset loss function, based on the three-dimensional coordinates of the aforementioned joints and the aforementioned sensor data, with the parameter information of the three-dimensional human body model as the optimization target: based on the mapped coordinates of the aforementioned joints and the aforementioned three-dimensional coordinates of the aforementioned joints, a first difference value is determined, wherein the aforementioned mapped coordinates are obtained by mapping the parameter information of the three-dimensional human body model to three-dimensional space; based on the aforementioned global posture information of the aforementioned target joints and the aforementioned degree of freedom information of the aforementioned target joints, a second difference value is determined, wherein the aforementioned global posture information is determined based on the aforementioned local posture information of the aforementioned target joints; based on the aforementioned first difference value and the aforementioned second difference value, with the parameter information of the three-dimensional human body model as the optimization target, the parameter information of the three-dimensional human body model when the loss function reaches its minimum value is determined by using a preset loss function.

[0169] In some optional implementations, the third determining unit 604 is further used to determine a third difference value based on the pose parameters corresponding to the next frame and the pose parameters corresponding to the current frame; and the third determining unit 604 can determine the parameter information of the three-dimensional human body model when the loss function reaches its minimum value by using the parameter information of the three-dimensional human body model as the optimization target, based on the first difference value and the second difference value, and using a preset loss function: based on the first difference value, the second difference value and the third difference value, using the parameter information of the three-dimensional human body model as the optimization target, and using a preset loss function, determine the parameter information of the three-dimensional human body model when the loss function reaches its minimum value.

[0170] In some optional implementations, the third determining unit 604 is further configured to determine a fourth difference value based on the length of each bone segment and the standard length of each bone segment, wherein the length of each bone segment is determined using the body shape parameters, and the standard length of each bone segment is the average length of bones in a preset number of target frames; and the third determining unit 604 can determine the parameter information of the three-dimensional human body model when the loss function reaches its minimum value by using a preset loss function, based on the first difference value, the second difference value, and the third difference value, with the parameter information of the three-dimensional human body model as the optimization target: based on the first difference value, the second difference value, the third difference value, and the fourth difference value, with the parameter information of the three-dimensional human body model as the optimization target, and using a preset loss function, determine the parameter information of the three-dimensional human body model when the loss function reaches its minimum value.

[0171] In some alternative implementations, the three-dimensional coordinates of the target joint are the rotation amounts in the degree of freedom information of the target joint.

[0172] In some alternative implementations, the aforementioned sensor data includes degrees of freedom information of the extended reality device's handle and degrees of freedom information of the head-mounted device, wherein the degrees of freedom information of the wrist joints is determined based on the degrees of freedom information of the handles, and the degrees of freedom information of the head joints is determined based on the degrees of freedom information of the head-mounted device.

[0173] In some optional implementations, the target joint point is the wrist joint point; and the three-dimensional human body model parameter fitting device 600 may include: a fourth determining unit (not shown in the figure), a fifth determining unit (not shown in the figure), a calibration unit (not shown in the figure), and a sixth determining unit (not shown in the figure). The fourth determining unit can be used to determine the local pose information of the wrist joint point of a preset number of target frames using the parameter information of the three-dimensional human body model, wherein the human body pose in the target frames is a preset standard pose; the fifth determining unit can be used to determine a compensation matrix based on the local pose information; the calibration unit can be used to calibrate the degree of freedom information of the handle using the compensation matrix; and the sixth determining unit can be used to determine the degree of freedom information of the calibrated wrist joint point using the calibrated degree of freedom information of the handle.

[0174] In some optional implementations, the aforementioned three-dimensional human body model parameter fitting device 600 may further include: a first re-determination unit (not shown in the figure) and a second re-determination unit (not shown in the figure). The first re-determination unit is used to re-determine the second difference value based on the global pose information of the wrist joint and the calibrated degree-of-freedom information of the wrist joint; the second re-determination unit is used to replace the initial second difference value with the re-determined second difference value, and using the parameter information of the three-dimensional human body model as the optimization target, and using a preset loss function, to re-determine the parameter information of the three-dimensional human body model when the loss function reaches its minimum value.

[0175] In some optional implementations, the second determining unit 603 can be further used to determine the three-dimensional coordinates of each joint point based on the two-dimensional coordinates of each joint point in the following manner: using the two-dimensional coordinates of each joint point to determine the initial three-dimensional coordinates of each joint point; determining a fifth difference value based on the projected coordinates and the two-dimensional coordinates of each joint point, wherein the projected coordinates are obtained by projecting the initial three-dimensional coordinates onto human body images acquired from various viewpoints; and based on the fifth difference value, using the three-dimensional coordinates of each joint point as the optimization target, using a preset loss function, determining the three-dimensional coordinates when the loss function reaches its minimum value as the three-dimensional coordinates of each joint point.

[0176] In some optional implementations, the second determining unit 603 can be used to determine the sixth difference value based on the initial three-dimensional coordinates corresponding to the next frame and the initial three-dimensional coordinates corresponding to the current frame; and the second determining unit 603 can be further used to determine the three-dimensional coordinates of each joint point as the three-dimensional coordinates of each joint point by using a preset loss function based on the fifth difference value in the following manner: based on the fifth difference value and the sixth difference value, using the three-dimensional coordinates of each joint point as the optimization target, using a preset loss function, determine the three-dimensional coordinates of each joint point as the three-dimensional coordinates of each joint point.

[0177] In some optional implementations, the second determining unit 603 can be used to determine a seventh difference value based on the length of each bone segment and the standard length of each bone segment, wherein the length of each bone segment is determined by the distance between two adjacent joint points, and the standard length of each bone segment is the average length of the bones in a preset number of frames; and the second determining unit 603 can be further used to determine the three-dimensional coordinates of each joint point as the three-dimensional coordinates of each joint point by using a preset loss function, based on the fifth difference value and the sixth difference value, with the three-dimensional coordinates of each joint point as the optimization target: based on the fifth difference value, the sixth difference value and the seventh difference value, with the three-dimensional coordinates of each joint point as the optimization target, and using a preset loss function, determine the three-dimensional coordinates of each joint point as the three-dimensional coordinates of each joint point.

[0178] See further Figure 7 , Figure 7 An exemplary system architecture 700 is shown, in which embodiments of the three-dimensional human body model parameter fitting method of this disclosure can be applied.

[0179] like Figure 7 As shown, system architecture 700 may include terminal devices 7011, 7012, and 7013, network 702, and server 703. Network 702 is used as a medium to provide a communication link between terminal devices 7011, 7012, and 7013 and server 703. Network 702 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0180] Users can use terminal devices 7011, 7012, and 7013 to interact with server 703 via network 702 to send or receive messages, etc. For example, server 703 can receive multi-view human image sequences and sensor data sent by terminal devices 7011, 7012, and 7013. Various communication client applications can be installed on terminal devices 7011, 7012, and 7013, such as image capture applications, video processing applications, video playback applications, instant messaging software, etc.

[0181] Terminal devices 7011, 7012, and 7013 can be either hardware or software. When terminal devices 7011, 7012, and 7013 are hardware, they can be various electronic devices that support information interaction, including but not limited to extended reality devices, smart cameras, smartphones, tablets, and laptops. When terminal devices 7011, 7012, and 7013 are software, they can be installed in the electronic devices listed above. They can be implemented as multiple software programs or software modules (e.g., multiple software programs or software modules used to provide distributed services) or as a single software program or software module. No specific limitations are imposed here.

[0182] Server 703 can be a server that provides various services. For example, it can be a backend server that analyzes multi-view human image sequences and sensor data during the image acquisition process. Server 703 can first acquire multi-view human image sequences and sensor data during the image acquisition process from terminal devices 7011, 7012, and 7013; then, determine the two-dimensional coordinates of each joint point in the aforementioned multi-view human image sequence; then, based on the two-dimensional coordinates of each joint point, determine the three-dimensional coordinates of each joint point; finally, based on the three-dimensional coordinates of each joint point and the aforementioned sensor data, determine the parameter information of the three-dimensional human body model.

[0183] It should be noted that a server 703 can be either hardware or software. When the server 703 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server 703 is software, it can be implemented as multiple software programs or software modules (e.g., used to provide distributed services), or as a single software program or software module. No specific limitations are made here.

[0184] It should also be noted that the three-dimensional human body model parameter fitting method provided in this embodiment is usually executed by server 703, and the three-dimensional human body model parameter fitting device is usually set in server 703.

[0185] It should be understood that Figure 7The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0186] The following is for reference. Figure 8 It illustrates an electronic device suitable for implementing embodiments of the present disclosure (e.g., Figure 7 The structural diagram of the server (800) in the middle. Figure 8 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0187] like Figure 8 As shown, the electronic device 800 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage device 808 into a random access memory (RAM) 803. The RAM 803 also stores various programs and data required for the operation of the electronic device 800. The processing device 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0188] Typically, the following devices can be connected to I / O interface 805: input devices 806 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 807 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 808 including, for example, magnetic tapes, hard disks, etc.; and communication devices 809. Communication device 809 allows electronic device 800 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 8 An electronic device 800 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 8 Each box shown can represent a device or multiple devices as needed.

[0189] Specifically, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 809, or installed from a storage device 808, or installed from a ROM 802. When the computer program is executed by a processing device 801, it performs the functions defined in the methods of embodiments of this disclosure. It should be noted that the computer-readable medium described in embodiments of this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0190] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: acquire multi-view human image sequences and sensor data during the image acquisition process; determine the two-dimensional coordinates of each joint point in the multi-view human image sequence; determine the three-dimensional coordinates of each joint point based on the two-dimensional coordinates of each joint point; and determine the parameter information of the three-dimensional human model based on the three-dimensional coordinates of each joint point and the sensor data.

[0191] Computer program code for performing the operations of embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0192] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0193] The units described in the embodiments of this disclosure can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor may be described as including an acquisition unit, a first determining unit, a second determining unit, and a third determining unit. The names of these units do not necessarily limit the specific unit; for example, the acquisition unit may also be described as "a unit that acquires multi-view human image sequences and sensor data during the image acquisition process."

[0194] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A method for fitting parameters of a three-dimensional human body model, characterized in that, include: Acquire multi-view human body image sequences and sensor data during image acquisition; Determine the two-dimensional coordinates of each joint point in the multi-view human image sequence; Based on the two-dimensional coordinates of each joint, determine the three-dimensional coordinates of each joint. Based on the three-dimensional coordinates of each joint and the sensor data, the parameter information of the three-dimensional human body model is determined.

2. The method according to claim 1, characterized in that, The determination of parameter information for the 3D human body model based on the 3D coordinates of each joint point and the sensor data includes: Based on the three-dimensional coordinates of each joint and the sensor data, with the parameter information of the three-dimensional human body model as the optimization target, the parameter information of the three-dimensional human body model when the loss function reaches its minimum value is determined using a preset loss function.

3. The method according to claim 2, characterized in that, The parameter information includes posture parameters and body shape parameters, and the sensor data includes the degree of freedom information of the target joints. The target joints include at least one of the following: wrist joints, head joints, and knee joints. as well as Based on the three-dimensional coordinates of each joint point and the sensor data, and using the parameter information of the three-dimensional human body model as the optimization target, the parameter information of the three-dimensional human body model when the loss function reaches its minimum value is determined using a preset loss function, including: Based on the mapped coordinates of each joint and the three-dimensional coordinates of each joint, a first difference value is determined, wherein the mapped coordinates are obtained by mapping the parameter information of the three-dimensional human body model to three-dimensional space; Based on the global attitude information and the degree of freedom information of the target joint, a second difference value is determined, wherein the global attitude information is determined based on the local attitude information of the target joint; Based on the first difference value and the second difference value, and taking the parameter information of the three-dimensional human body model as the optimization target, the parameter information of the three-dimensional human body model when the loss function reaches its minimum value is determined using a preset loss function.

4. The method according to claim 3, characterized in that, The method further includes: The third difference value is determined based on the pose parameters corresponding to the next frame and the pose parameters corresponding to the current frame; and The step of determining the parameter information of the 3D human body model when the loss function reaches its minimum value, based on the first difference value and the second difference value and using the parameter information of the 3D human body model as the optimization target and a preset loss function, includes: Based on the first difference value, the second difference value, and the third difference value, with the parameter information of the three-dimensional human body model as the optimization target, the parameter information of the three-dimensional human body model when the loss function reaches its minimum value is determined using a preset loss function.

5. The method according to claim 4, characterized in that, The method further includes: A fourth difference value is determined based on the length of each bone segment and the standard length of each bone segment, wherein the length of each bone segment is determined using the body shape parameters, and the standard length of each bone segment is the average length of bones in a preset number of target frames; and Based on the first difference value, the second difference value, and the third difference value, and using the parameter information of the 3D human body model as the optimization target, the step of determining the parameter information of the 3D human body model when the loss function reaches its minimum value using a preset loss function includes: Based on the first difference value, the second difference value, the third difference value, and the fourth difference value, with the parameter information of the three-dimensional human body model as the optimization target, the parameter information of the three-dimensional human body model when the loss function reaches its minimum value is determined using a preset loss function.

6. The method according to claim 3, characterized in that, The three-dimensional coordinates of the target joint are the rotation amount in the degree of freedom information of the target joint.

7. The method according to claim 6, characterized in that, The sensor data includes the degrees of freedom information of the extended reality device's handle and the head-mounted device. The degrees of freedom information of the wrist joints is determined based on the degrees of freedom information of the handles, and the degrees of freedom information of the head joints is determined based on the degrees of freedom information of the head-mounted device.

8. The method according to claim 7, characterized in that, The target joint is the wrist joint. as well as After determining the parameter information of the 3D human body model when the loss function reaches its minimum using a preset loss function, the method further includes: Using the parameter information of the three-dimensional human body model, the local pose information of the wrist joint points of a preset number of target frames is determined, wherein the human body pose in the target frames is a preset standard pose. Based on the local pose information, determine the compensation matrix; The degree-of-freedom information of the handle is calibrated using the compensation matrix. Using the calibrated handle's degree of freedom information, the calibrated wrist joint's degree of freedom information is determined.

9. The method according to claim 8, characterized in that, The method further includes: Based on the global posture information of the wrist joint and the calibrated degree-of-freedom information of the wrist joint, the second difference value is re-determined; The initial second difference value is replaced with a newly determined second difference value. Using the parameter information of the 3D human body model as the optimization target, the parameter information of the 3D human body model when the loss function reaches its minimum value is redetermined using a preset loss function.

10. The method according to claim 1, characterized in that, Determining the three-dimensional coordinates of each joint point based on its two-dimensional coordinates includes: The initial three-dimensional coordinates of each joint point are determined using the two-dimensional coordinates of each joint point. Based on the projected coordinates and two-dimensional coordinates of each joint, a fifth difference value is determined, wherein the projected coordinates are obtained by projecting the initial three-dimensional coordinates onto human body images acquired from various viewpoints. Based on the fifth difference value, with the three-dimensional coordinates of each joint as the optimization target, and using a preset loss function, the three-dimensional coordinates when the loss function reaches its minimum value are determined as the three-dimensional coordinates of each joint.

11. The method according to claim 10, characterized in that, The method further includes: The sixth difference value is determined based on the initial 3D coordinates corresponding to the next frame and the initial 3D coordinates corresponding to the current frame; and Based on the fifth difference value, and using the three-dimensional coordinates of each joint as the optimization objective, the step of determining the three-dimensional coordinates of each joint when the loss function reaches its minimum value using a preset loss function includes: Based on the fifth and sixth difference values, and taking the three-dimensional coordinates of each joint as the optimization target, the three-dimensional coordinates at which the loss function reaches its minimum value are determined using a preset loss function as the three-dimensional coordinates of each joint.

12. The method according to claim 11, characterized in that, The method further includes: A seventh difference value is determined based on the length of each bone segment and the standard length of each bone segment, wherein the length of each bone segment is determined using the distance between two adjacent joint points, and the standard length of each bone segment is the average length of the bones in a preset number of frames; and The step of determining the three-dimensional coordinates of each joint point based on the fifth and sixth difference values, using the three-dimensional coordinates of each joint point as the optimization objective and employing a preset loss function, includes: Based on the fifth, sixth, and seventh difference values, and with the three-dimensional coordinates of each joint point as the optimization target, the three-dimensional coordinates at which the loss function reaches its minimum value are determined using a preset loss function as the three-dimensional coordinates of each joint point.

13. A three-dimensional human body model parameter fitting device, characterized in that, include: The acquisition unit is used to acquire multi-view human image sequences and sensor data during the image acquisition process; The first determining unit is used to determine the two-dimensional coordinates of each joint point in the multi-view human image sequence; The second determining unit is used to determine the three-dimensional coordinates of each joint point based on the two-dimensional coordinates of each joint point; The third determining unit is used to determine the parameter information of the three-dimensional human body model based on the three-dimensional coordinates of each joint point and the sensor data.

14. An electronic device, characterized in that, include: One or more processors; Storage device, on which one or more programs are stored, When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-12.

15. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-12.

Images