A virtual human driving method, device, equipment and readable storage medium

Abstract

The application discloses a virtual person driving method and device, equipment and readable storage medium, and relates to the field of computer graphics, and comprises the following steps: acquiring video frame data, wherein the video frame data comprises at least one character; determining the three-dimensional space coordinates of the joint points of the character and the rotation angles of the joint points of the character according to the video frame data and a pose estimation model; constructing a driving file according to the three-dimensional space coordinates and the rotation angles; and driving a virtual person based on the driving file. It can be seen that the application can directly construct a driving file for driving a virtual person through video frame data, and can drive a virtual person only through a video, which greatly reduces the use cost and improves the interactivity of the virtual person compared with the method in the prior art that first records actions through a motion capture device and then uses an FBX action file to drive a virtual person.

Description

Technical Field

[0001] This application relates to the field of computer graphics, and more specifically, to a virtual human driving method, apparatus, device, and readable storage medium. Background Technology

[0002] Real-time human motion capture and transfer is a crucial and fundamental problem in computer vision and computer graphics, with wide applications in virtual reality and mixed reality. For example, in mixed reality, by introducing virtual character models and using real-time human motion for driving—that is, virtual human-driven motion—richer interactive experiences can be achieved. Because human motion is ubiquitous in daily life, human motion capture has extremely high scientific research and application value.

[0003] Current methods for driving virtual humans primarily involve recording motion using motion capture devices and then using FBX motion files to drive the virtual human. However, this method is costly and significantly limits the interactivity of the virtual human. Summary of the Invention

[0004] This application provides a virtual human driving method, apparatus, device, and readable storage medium. Compared with the prior art method of first recording actions through a motion capture device and then using FBX motion files to drive the virtual human, it greatly reduces the cost of use and improves the interactivity of the virtual human.

[0005] In view of this, embodiments of this application provide a virtual human driving method, the method comprising:

[0006] Acquire video frame data, wherein the video frame data includes at least one person;

[0007] The three-dimensional spatial coordinates of the joints of the character and the rotation angle of the joints of the character are determined based on the video frame data and the pose estimation model.

[0008] Construct a driver file based on the three-dimensional spatial coordinates and the rotation angle;

[0009] The virtual human is driven based on the aforementioned driver file.

[0010] Optionally, the pose estimation model includes a ROMP model and a Mediapipe model; determining the three-dimensional spatial coordinates of the character's joints and the rotation angle of the character's joints based on the video frame data and the pose estimation model includes:

[0011] The video frame data is input into the ROMP model to obtain the rotation angle of the torso joints and the local rotation angle of the root joint. The torso joints are the joints of the character other than the root joint.

[0012] The video frame data is input into the Mediapipe model to obtain the initial three-dimensional spatial coordinates of the character's joints;

[0013] The initial three-dimensional spatial coordinates are repositioned to obtain the target three-dimensional spatial coordinates of the character's joints;

[0014] Construct a target coordinate system with the root joint point as the origin based on the target's three-dimensional spatial coordinates;

[0015] Construct the rotation matrix of the root joint based on the local rotation angle of the root joint and the target coordinate system;

[0016] The rotation matrix is ​​converted into Euler angles of the root joint to obtain the global rotation angle of the root joint, and the global rotation angle of the root joint is used as the rotation angle of the root joint.

[0017] The three-dimensional spatial coordinates of the character's joints are calculated based on the global rotation angle and the original three-dimensional spatial coordinates of the character's joints. The original three-dimensional spatial coordinates are the three-dimensional spatial coordinates of the character when it is in a bound posture.

[0018] Optionally, a driving file is constructed based on the three-dimensional spatial coordinates and the rotation angle, including:

[0019] Create an initial driver file, which includes the original three-dimensional spatial coordinates and original rotation angles of the character's joints. The original three-dimensional spatial coordinates are the three-dimensional spatial coordinates of the character when it is in a tethered pose, and the original rotation angle is zero.

[0020] Replace the original three-dimensional spatial coordinates with the three-dimensional spatial coordinates of the character's joints;

[0021] The rotation angles of the character's joints are mirrored to obtain the mirrored rotation angles of the character's joints.

[0022] Replace the original rotation angle with the mirror rotation angle;

[0023] Set the scaling value to obtain the driver file.

[0024] Optionally, before constructing the driving file based on the three-dimensional spatial coordinates and the rotation angle, the method further includes:

[0025] The three-dimensional spatial coordinates and the rotation angle are filtered respectively to obtain the filtered three-dimensional spatial coordinates of the character's joints and the filtered rotation angle of the character's joints.

[0026] Therefore, the step of constructing the driving file based on the three-dimensional spatial coordinates and the rotation angle specifically involves:

[0027] A driver file is constructed based on the filtered three-dimensional spatial coordinates of the character's joints and the filtered rotation angles of the character's joints.

[0028] Optionally, the step of filtering the three-dimensional spatial coordinates and the rotation angle to obtain the filtered three-dimensional spatial coordinates and the filtered rotation angle of the character's joints includes:

[0029] The three-dimensional spatial coordinates and the rotation angle are initially filtered by a one-euro filter to obtain the initially filtered three-dimensional spatial coordinates and the initially filtered rotation angle.

[0030] The three-dimensional spatial coordinates and rotation angles obtained from the initial filtering are further filtered using a weighted mean filter to obtain the filtered three-dimensional spatial coordinates and rotation angles of the character's joints.

[0031] This application embodiment also provides a virtual human driving device, the device comprising:

[0032] An acquisition unit is used to acquire video frame data, wherein the video frame data includes at least one person;

[0033] The determining unit is used to determine the three-dimensional spatial coordinates of the joints of the character and the rotation angle of the joints of the character based on the video frame data and the pose estimation model.

[0034] A construction unit is used to construct a driver file based on the three-dimensional spatial coordinates and the rotation angle;

[0035] A driving unit is used to drive the virtual human based on the driving file.

[0036] Optionally, the attitude estimation model includes the ROMP model and the Mediapipe model; the determining unit is specifically used for:

[0037] The video frame data is input into the ROMP model to obtain the rotation angle of the torso joints and the local rotation angle of the root joint. The torso joints are the joints of the character other than the root joint.

[0038] The video frame data is input into the Mediapipe model to obtain the initial three-dimensional spatial coordinates of the character's joints;

[0039] The initial three-dimensional spatial coordinates are repositioned to obtain the target three-dimensional spatial coordinates of the character's joints;

[0040] Construct a target coordinate system with the root joint point as the origin based on the target's three-dimensional spatial coordinates;

[0041] Construct the rotation matrix of the root joint based on the local rotation angle of the root joint and the target coordinate system;

[0042] The rotation matrix is ​​converted into Euler angles of the root joint to obtain the global rotation angle of the root joint, and the global rotation angle of the root joint is used as the rotation angle of the root joint.

[0043] The three-dimensional spatial coordinates of the character's joints are calculated based on the global rotation angle and the original three-dimensional spatial coordinates of the character's joints. The original three-dimensional spatial coordinates are the three-dimensional spatial coordinates of the character when it is in a bound posture.

[0044] Optionally, the building unit is specifically used for:

[0045] Create an initial driver file, which includes the original three-dimensional spatial coordinates and original rotation angles of the character's joints. The original three-dimensional spatial coordinates are the three-dimensional spatial coordinates of the character when it is in a tethered pose, and the original rotation angle is zero.

[0046] Replace the original three-dimensional spatial coordinates with the three-dimensional spatial coordinates of the character's joints;

[0047] The rotation angles of the character's joints are mirrored to obtain the mirrored rotation angles of the character's joints.

[0048] Replace the original rotation angle with the mirror rotation angle;

[0049] Set the scaling value to obtain the driver file.

[0050] This application also provides a computer device, including: a memory, a processor, and a bus system;

[0051] The memory is used to store programs;

[0052] The processor is used to execute the program in the memory to implement any of the virtual human driving methods described above;

[0053] The bus system is used to connect the memory and the processor to enable communication between the memory and the processor.

[0054] This application also provides a computer-readable storage medium storing instructions that, when run on a computer, cause the computer to execute any of the virtual human driving methods described above.

[0055] This application provides a virtual human driving method, comprising: acquiring video frame data, the video frame data including at least one human figure; determining the three-dimensional spatial coordinates of the joints of the human figure and the rotation angle of the joints of the human figure based on the video frame data and a pose estimation model; constructing a driving file based on the three-dimensional spatial coordinates and the rotation angle; and driving the virtual human figure based on the driving file. It is evident that this application can directly construct a driving file for driving a virtual human figure from video frame data, enabling the virtual human figure to be driven solely by video. Compared to the prior art method of first recording actions using a motion capture device and then using FBX motion files to drive the virtual human figure, this significantly reduces the cost of use and improves the interactivity of the virtual human figure. Attached Figure Description

[0056] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0057] Figure 1 A flowchart illustrating a virtual human driving method provided in an embodiment of this application;

[0058] Figure 2 This is a schematic diagram of the structure of a skeletal node template provided in an embodiment of this application;

[0059] Figure 3 This is a schematic diagram of another skeletal node template provided in an embodiment of this application;

[0060] Figure 4 This is a schematic diagram of the structure of a virtual human driving device provided in an embodiment of this application. Detailed Implementation

[0061] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0062] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0063] Real-time human motion capture and transfer is a crucial and fundamental problem in computer vision and computer graphics, with wide applications in virtual reality and mixed reality. For example, in mixed reality, by introducing virtual character models and using real-time human motion for driving—that is, virtual human driving—richer interactive experiences can be achieved. Because human motion is ubiquitous in daily life, human motion capture has extremely high research and application value. Existing virtual human driving methods primarily involve recording movements using motion capture devices and then using FBX motion files to drive the virtual human. However, this method is costly and significantly limits the interactivity of the virtual human.

[0064] Therefore, to address the above-mentioned problems, this application provides a virtual human driving method, apparatus, device, and readable storage medium. Compared with the prior art method of first recording actions through a motion capture device and then using FBX motion files to drive the virtual human, this greatly reduces the cost of use and improves the interactivity of the virtual human.

[0065] Please see Figure 1 The virtual human driving method provided in this application includes the following steps.

[0066] S101. Obtain video frame data, wherein the video frame data includes at least one person.

[0067] In this embodiment, to enable the virtual human to be driven solely by video, video frame data can be acquired first. This video frame data includes at least one person. It is understood that the video frame data can be frame data from a pre-recorded video or frame data from a video recorded in real-time by a camera. Furthermore, the video frame data should include at least one person so that the virtual human can be driven based on the person's movements and posture. The video frame data can be accessed through OpenCV, a cross-platform computer vision and machine learning software library released under the Apache 2.0 license (open source).

[0068] S102. Determine the three-dimensional spatial coordinates of the joints of the character and the rotation angle of the joints of the character based on the video frame data and the pose estimation model.

[0069] In this embodiment, after acquiring video frame data, the three-dimensional spatial coordinates of the character's joints and the rotation angles of the joints can be determined based on the video frame data and the pose estimation model. It is understood that after acquiring video frame data, it is necessary to perform pose estimation on the character included in the video frame data. This can be done using a pose estimation model to determine the three-dimensional spatial coordinates of the character's joints and the rotation angles of the joints, in order to construct a driver file that can accurately drive the virtual human.

[0070] In one possible implementation, the pose estimation model can include a ROMP model and a Mediapipe model. Video frame data can be input into the ROMP model to obtain the rotation angles of the torso joints and the local rotation angles of the root joints, where the torso joints are all joints except the root joint. Video frame data can be input into the Mediapipe model to obtain the initial 3D spatial coordinates of the character's joints. The initial 3D spatial coordinates are repositioned to obtain the target 3D spatial coordinates of the character's joints. A target coordinate system with the root joint as the origin is constructed based on the target 3D spatial coordinates. A rotation matrix for the root joint is constructed based on the local rotation angles of the root joint and the target coordinate system. The rotation matrix is ​​converted into Euler angles for the root joint to obtain the global rotation angle of the root joint, which is then used as the rotation angle of the root joint. The 3D spatial coordinates of the character's joints are calculated based on the global rotation angles and the original 3D spatial coordinates of the character's joints, where the original 3D spatial coordinates are the 3D spatial coordinates of the character when in a tethered pose.

[0071] Understandably, the ROMP (Regression of Multiple 3D People) model is a single-stage network used to regress multiple 3D human body meshes from pixel level. It can directly estimate multiple different maps from a single image, and through these maps, it can resolve the 3D human body meshes of all people. The Mediapipe model is an open-source project from Google that provides open-source, cross-platform solutions for commonly used machine learning. The Mediapipe model is actually an integrated library of machine learning vision algorithms, including models for face detection, facial landmarks, gesture recognition, avatar segmentation, and pose recognition. Video frame data can be input into the ROMP model to obtain the local rotation angles of the person's joints, including the local rotation angles of the torso joints and the root joint (the hip joint). The local rotation angles can be represented using three-dimensional Euler angles. It can be observed that the local rotation angles of the torso joints determined by the ROMP model are quite accurate, and these local rotation angles can be directly used as the rotation angles of the torso joints when building the driver file. However, the local rotation angle of the root joint estimated by the ROMP model differs significantly from the actual rotation angle of the character. If the local rotation angle of the root joint is directly determined as the rotation angle of the root joint when building the driver file, it will cause the rotation of the virtual character's root joint to be unstable.

[0072] Therefore, to address the issue of root joint rotation angles, video frame data can be input into the Mediapipe model to obtain the initial 3D spatial coordinates of the character's joints. The Mediapipe model uses a skeletal node template such as... Figure 2 As shown, the Mediapipe model can estimate the initial 3D spatial coordinates of the character's 32 joints. However, to calculate the rotation angle of the root joint, the initial 3D spatial coordinates of the character's joints estimated by the Mediapipe model can be repositioned to obtain the target 3D spatial coordinates of the character's joints, as shown in Figure 3. The correspondence between some of the target 3D spatial coordinates and the initial 3D spatial coordinates is as follows:

[0073] Hips=(left_hip+right_hip) / 2;

[0074] Neck=(left_shoulder+right_shoulder) / 2;

[0075] Spine = Hips + (Neck - Hips) / 3;

[0076] LeftUpLeg = left_hip;

[0077] RightUpLeg = right_hip;

[0078] Among them, Hips, Neck, Spine, LeftUpLeg, and RightUpLeg are Figure 3 The target 3D space coordinates of some joints of the character, left_hip, right_hip, left_shoulder, and right_shoulder are... Figure 2 The initial three-dimensional spatial coordinates of some joints of the character.

[0079] It should be noted that the target 3D space coordinates of the character's joints may be opposite to those in the virtual engine used to drive the virtual human. If the rotation angle of the root joint is calculated directly using the target 3D space coordinates, the virtual human may end up moving upside down. Therefore, all coordinate points can be rotated 180 degrees around the X-axis to be consistent with the Y-axis in the virtual engine.

[0080] After obtaining the target 3D spatial coordinates of the character's joints, a target coordinate system with the root joint as the origin can be constructed based on the target 3D spatial coordinates, as shown below:

[0081] X_dir=keypoints3d[LeftUpLeg]-keypoints3d[RightUpLeg];

[0082] Y_dir=keypoints3d[Spine]-keypoints3d[Hips];

[0083] Z_dir = X_dir * Y_dir;

[0084] Where X_dir is the X-axis of the target coordinate system, Y_dir is the Y-axis of the target coordinate system, Z_dir is the Z-axis of the target coordinate system, * is the cross product symbol, keypoints3d[LeftUpLeg] is the target 3D spatial coordinate of the LeftUpLeg joint, keypoints3d[RightUpLeg] is the target 3D spatial coordinate of the RightUpLeg joint, keypoints3d[Spine] is the target 3D spatial coordinate of the Spine joint, and keypoints3d[Hips] is the target 3D spatial coordinate of the Hips joint.

[0085] After constructing the target coordinate system with the root joint as the origin, the three coordinate axes can be normalized. A rotation matrix can be constructed according to the xyz column order and the local rotation angle of the root joint. Then, the rotation matrix can be converted into a quaternion, as shown below:

[0086] Quaternion q = w + xi + yj + zk, ||q|| = x 2 +y 2 +z 2 +w 2 = 1, where w is the real part of the quaternion and xi+yj+zk is the imaginary part of the quaternion. The rotation matrix constructed using this quaternion can be expressed as:

[0087]

[0088] Therefore, the formula for converting a rotation matrix to a quaternion is as follows:

[0089]

[0090]

[0091]

[0092]

[0093] After converting the rotation matrix into a quaternion, the quaternion can be further converted into Euler angles (φ,θ,ψ), as shown below:

[0094]

[0095] The Euler angles that can be calculated at this point are the global rotation angles of the root joint, and these global rotation angles can be used as the rotation angles of the root joint when constructing the driver file.

[0096] After obtaining the global rotation angle of the root joint, the three-dimensional spatial coordinates of the character's joints can be calculated based on the global rotation angle and the original three-dimensional spatial coordinates of the character's joints. The original three-dimensional spatial coordinates are the three-dimensional spatial coordinates of the character when in a tethered pose, i.e., Tpose, as shown below:

[0097] First, convert the Euler angles into a rotation matrix R, as shown below:

[0098]

[0099] Where α is the angle of rotation of the root joint around the z-axis, β is the angle of rotation of the root joint around the y-axis, and γ is the angle of rotation of the root joint around the x-axis.

[0100] Then, based on the rotation matrix and the original 3D spatial coordinates of the character's joints, the 3D spatial coordinates of the character's joints are calculated, as shown below:

[0101] ′

[0102] p = Rp;

[0103] Where p′ is the three-dimensional spatial coordinate of the character's joints, R is the rotation matrix, and p is the original three-dimensional spatial coordinate of the character's joints, that is, the three-dimensional spatial coordinate of the character when it is in the bound pose, i.e., Tpose.

[0104] S103. Construct a driving file based on the three-dimensional spatial coordinates and the rotation angle.

[0105] In this embodiment, a driving file can be constructed based on the three-dimensional spatial coordinates of the character's joints and the rotation angles of the character's joints. It can be understood that after obtaining the three-dimensional spatial coordinates of the character's joints and the rotation angles of the character's joints, a driving file can be constructed to enable the virtual human to be driven based on the driving file.

[0106] In one possible implementation, an initial driver file can be created, which includes the original 3D spatial coordinates and original rotation angles of the character's joints. The original 3D spatial coordinates are the 3D spatial coordinates of the character when in a tethered pose, and the original rotation angle is zero. The original 3D spatial coordinates are then replaced with the 3D spatial coordinates of the character's joints. The rotation angles of the character's joints are then mirrored to obtain the mirrored rotation angles of the character's joints. The original rotation angles are then replaced with the mirrored rotation angles. Finally, a scaling value is set to obtain the driver file.

[0107] Understandably, the first step is to create a raw driver file, which could be the FBX file corresponding to the Tpose virtual human. This file is the one whose data needs to be completed. It provides the original 3D spatial coordinates and original rotation angles of the human's joints before rotation. If the blank driver file is directly completed using the obtained 3D spatial coordinates and rotation angles of the human's joints, the resulting driver file will have mirrored data. For example, the movement of the human's left hand in a video frame will be displayed on the virtual human's right hand. Therefore, the rotation angles of the human's joints need to be mirrored to obtain the mirrored rotation angles. Finally, the original rotation angles are replaced with the mirrored rotation angles, and a scaling value is set (the default scaling value can be 1), thus constructing the driver file.

[0108] In one possible implementation, before constructing the driver file based on the three-dimensional spatial coordinates and the rotation angle, the three-dimensional spatial coordinates and the rotation angle can be filtered to obtain the filtered three-dimensional spatial coordinates and the filtered rotation angle of the character's joints; then, constructing the driver file based on the three-dimensional spatial coordinates and the rotation angle specifically involves constructing the driver file based on the filtered three-dimensional spatial coordinates and the filtered rotation angle of the character's joints.

[0109] Understandably, if the driver file is directly constructed based on the obtained 3D spatial coordinates and rotation angles of the character's joints, and the virtual human is driven based on this driver file, the virtual human may experience motion jitter. Therefore, it is necessary to first filter the 3D spatial coordinates and rotation angles of the character's joints to obtain the filtered 3D spatial coordinates and rotation angles of the character's joints, and then construct the driver file based on the filtered 3D spatial coordinates and rotation angles.

[0110] In one possible implementation, a one-euro filter can be used to perform initial filtering on the three-dimensional spatial coordinates and rotation angles to obtain the initially filtered three-dimensional spatial coordinates and rotation angles; a weighted mean filter can then be used to further filter the initially filtered three-dimensional spatial coordinates and rotation angles to obtain the filtered three-dimensional spatial coordinates and rotation angles of the character's joints.

[0111] Understandably, filters can be constructed for the three-dimensional spatial coordinates and rotation angles separately. A one-euro filter can be used first to filter out small jitters, and then a weighted average filter can be used to filter out larger disturbances. The one-euro filter is an adaptive cutoff frequency low-pass filter proposed by GeryCasiez et al. in 2012. It has only two configurable parameters and can filter noise signals in real time.

[0112] The formula for the weighted mean filter is:

[0113]

[0114] Where y(n) is the filtered value, x(n) is the value of the current frame, x(n-1) is the value of the previous frame, and x(n-2) is the value of the two frames before the current frame. That is, for each input frame, the values ​​of the current frame and the two frames before it are weighted and summed to obtain the three-dimensional spatial coordinates and rotation angles of the character's joints.

[0115] S104. Drive the virtual human based on the driver file.

[0116] In this embodiment, after constructing the driver file, the virtual human can be driven based on the driver file, enabling the virtual human to be driven solely by video. Specifically, the driver file can be encapsulated in JSON format and sent to the Unreal Engine driving the virtual human via a socket interface. The Unreal Engine can extend the LiveLink plugin in the UE to process the received data. First, a FLiveLinkStaticDataStruct is constructed, and bone nodes are added based on poseNames and parents; then, the 3D spatial coordinates, rotation angle, and scaling value of each bone are added to the FLiveLinkFrameDataStruct frame by frame. It is worth noting that the definition of rotation in the UE is FRotator(Pitch, Yaw, Roll), where Pitch represents the angle of rotation around the y-axis, Yaw represents the angle of rotation around the z-axis, and Roll represents the angle of rotation around the x-axis. In addition to extending the plugin, an animation blueprint needs to be created for the virtual human skeleton to be driven in the UE, so that the data received by LiveLink can flow to that skeleton.

[0117] This application provides a virtual human driving method, comprising: acquiring video frame data, the video frame data including at least one human figure; determining the three-dimensional spatial coordinates of the joints of the human figure and the rotation angle of the joints of the human figure based on the video frame data and a pose estimation model; constructing a driving file based on the three-dimensional spatial coordinates and the rotation angle; and driving the virtual human figure based on the driving file. It is evident that this application can directly construct a driving file for driving a virtual human figure from video frame data, enabling the virtual human figure to be driven solely by video. Compared to the prior art method of first recording actions using a motion capture device and then using FBX motion files to drive the virtual human figure, this significantly reduces the cost of use and improves the interactivity of the virtual human figure.

[0118] Please see Figure 4 This application provides a virtual human driving device, the device comprising:

[0119] Acquisition unit 401 is used to acquire video frame data, wherein the video frame data includes at least one person;

[0120] The determining unit 402 is used to determine the three-dimensional spatial coordinates of the joints of the character and the rotation angle of the joints of the character based on the video frame data and the pose estimation model.

[0121] Construction unit 403 is used to construct a driving file based on the three-dimensional spatial coordinates and the rotation angle;

[0122] The driving unit 404 is used to drive the virtual human based on the driving file.

[0123] Optionally, the attitude estimation model includes the ROMP model and the Mediapipe model; the determining unit 402 is specifically used for:

[0124] The video frame data is input into the ROMP model to obtain the rotation angle of the torso joints and the local rotation angle of the root joint. The torso joints are the joints of the character other than the root joint.

[0125] The video frame data is input into the Mediapipe model to obtain the initial three-dimensional spatial coordinates of the character's joints;

[0126] The initial three-dimensional spatial coordinates are repositioned to obtain the target three-dimensional spatial coordinates of the character's joints;

[0127] Construct a target coordinate system with the root joint point as the origin based on the target's three-dimensional spatial coordinates;

[0128] Construct the rotation matrix of the root joint based on the local rotation angle of the root joint and the target coordinate system;

[0129] The rotation matrix is ​​converted into Euler angles of the root joint to obtain the global rotation angle of the root joint, and the global rotation angle of the root joint is used as the rotation angle of the root joint.

[0130] The three-dimensional spatial coordinates of the character's joints are calculated based on the global rotation angle and the original three-dimensional spatial coordinates of the character's joints. The original three-dimensional spatial coordinates are the three-dimensional spatial coordinates of the character when it is in a bound posture.

[0131] Optionally, the building unit 403 is specifically used for:

[0132] Create an initial driver file, which includes the original three-dimensional spatial coordinates and original rotation angles of the character's joints. The original three-dimensional spatial coordinates are the three-dimensional spatial coordinates of the character when it is in a tethered pose, and the original rotation angle is zero.

[0133] Replace the original three-dimensional spatial coordinates with the three-dimensional spatial coordinates of the character's joints;

[0134] The rotation angles of the character's joints are mirrored to obtain the mirrored rotation angles of the character's joints.

[0135] Replace the original rotation angle with the mirror rotation angle;

[0136] Set the scaling value to obtain the driver file.

[0137] Optionally, the device further includes:

[0138] The filtering unit filters the three-dimensional spatial coordinates and the rotation angle respectively to obtain the filtered three-dimensional spatial coordinates of the character's joints and the filtered rotation angle of the character's joints.

[0139] Therefore, the building unit 403 is specifically used for:

[0140] A driver file is constructed based on the filtered three-dimensional spatial coordinates of the character's joints and the filtered rotation angles of the character's joints.

[0141] Optionally, the filtering unit is specifically used for:

[0142] The three-dimensional spatial coordinates and the rotation angle are initially filtered by a one-euro filter to obtain the initially filtered three-dimensional spatial coordinates and the initially filtered rotation angle.

[0143] The three-dimensional spatial coordinates and rotation angles obtained from the initial filtering are further filtered using a weighted mean filter to obtain the filtered three-dimensional spatial coordinates and rotation angles of the character's joints.

[0144] This application also provides a computer device, including: a memory, a processor, and a bus system;

[0145] The memory is used to store programs;

[0146] The processor is used to execute the program in the memory to implement any of the virtual human driving methods described above;

[0147] The bus system is used to connect the memory and the processor to enable communication between the memory and the processor.

[0148] This application also provides a computer-readable storage medium storing instructions that, when run on a computer, cause the computer to execute any of the virtual human driving methods described above.

[0149] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0150] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A virtual human driving method, characterized in that, The method includes: Acquire video frame data, wherein the video frame data includes at least one person; The three-dimensional spatial coordinates of the joints of the character and the rotation angle of the joints of the character are determined based on the video frame data and the pose estimation model. Construct a driver file based on the three-dimensional spatial coordinates and the rotation angle; The virtual human is driven based on the aforementioned driver file; The step of determining the three-dimensional spatial coordinates of the character's joints and the rotation angle of the character's joints based on the video frame data and the pose estimation model includes: First, the video frame data is input into the first model of the pose estimation model to obtain the rotation angle of the torso joints and the local rotation angle of the root joints of the person. The torso joints are the joints of the person other than the root joints. The video frame data is input into the second model of the pose estimation model to obtain the initial three-dimensional spatial coordinates of the character's joints; The three-dimensional spatial coordinates of the character's joints are determined based on the local rotation angle of the root joint and the initial three-dimensional spatial coordinates.

2. The method according to claim 1, characterized in that, The pose estimation model includes the ROMP model and the Mediapipe model; determining the three-dimensional spatial coordinates of the character's joints and the rotation angle of the character's joints based on the video frame data and the pose estimation model includes: The video frame data is input into the ROMP model to obtain the rotation angle of the torso joints and the local rotation angle of the root joint of the character. The torso joints are the joints of the character other than the root joint. The video frame data is input into the Mediapipe model to obtain the initial three-dimensional spatial coordinates of the character's joints; The initial three-dimensional spatial coordinates are repositioned to obtain the target three-dimensional spatial coordinates of the character's joints; Construct a target coordinate system with the root joint point as the origin based on the target's three-dimensional spatial coordinates; Construct the rotation matrix of the root joint based on the local rotation angle of the root joint and the target coordinate system; The rotation matrix is ​​converted into Euler angles of the root joint to obtain the global rotation angle of the root joint, and the global rotation angle of the root joint is used as the rotation angle of the root joint. The three-dimensional spatial coordinates of the character's joints are calculated based on the global rotation angle and the original three-dimensional spatial coordinates of the character's joints. The original three-dimensional spatial coordinates are the three-dimensional spatial coordinates of the character when it is in a bound posture.

3. The method according to claim 1, characterized in that, The driving file is constructed based on the three-dimensional spatial coordinates and the rotation angle, including: Create an initial driver file, which includes the original three-dimensional spatial coordinates and original rotation angles of the character's joints. The original three-dimensional spatial coordinates are the three-dimensional spatial coordinates of the character when it is in a bound pose, and the original rotation angle is zero. Replace the original three-dimensional spatial coordinates with the three-dimensional spatial coordinates of the character's joints; The rotation angles of the character's joints are mirrored to obtain the mirrored rotation angles of the character's joints. Replace the original rotation angle with the mirror rotation angle; Set the scaling value to obtain the driver file.

4. The method according to claim 1, characterized in that, Before constructing the driver file based on the three-dimensional spatial coordinates and the rotation angle, the method further includes: The three-dimensional spatial coordinates and the rotation angle are filtered respectively to obtain the filtered three-dimensional spatial coordinates of the character's joints and the filtered rotation angle of the character's joints. Therefore, the step of constructing the driving file based on the three-dimensional spatial coordinates and the rotation angle specifically involves: A driver file is constructed based on the filtered three-dimensional spatial coordinates of the character's joints and the filtered rotation angles of the character's joints.

5. The method according to claim 4, characterized in that, The step of filtering the three-dimensional spatial coordinates and the rotation angle to obtain the filtered three-dimensional spatial coordinates and the filtered rotation angle of the character's joints includes: The three-dimensional spatial coordinates and the rotation angle are initially filtered by a one-euro filter to obtain the initially filtered three-dimensional spatial coordinates and the initially filtered rotation angle. The three-dimensional spatial coordinates and rotation angles obtained from the initial filtering are further filtered using a weighted mean filter to obtain the filtered three-dimensional spatial coordinates and rotation angles of the character's joints.

6. A virtual human driving device, characterized in that, The device includes: An acquisition unit is used to acquire video frame data, wherein the video frame data includes at least one person; The determining unit is used to determine the three-dimensional spatial coordinates of the joints of the character and the rotation angle of the joints of the character based on the video frame data and the pose estimation model. A construction unit is used to construct a driver file based on the three-dimensional spatial coordinates and the rotation angle; A driving unit is used to drive the virtual human based on the driving file; The determining unit is specifically used for: First, the video frame data is input into the first model of the pose estimation model to obtain the rotation angle of the torso joints and the local rotation angle of the root joints of the person. The torso joints are the joints of the person other than the root joints. The video frame data is input into the second model of the pose estimation model to obtain the initial three-dimensional spatial coordinates of the character's joints; The three-dimensional spatial coordinates of the character's joints are determined based on the local rotation angle of the root joint and the initial three-dimensional spatial coordinates.

7. The apparatus according to claim 6, characterized in that, The attitude estimation model includes the ROMP model and the Mediapipe model; the determination unit is specifically used for: The video frame data is input into the ROMP model to obtain the rotation angle of the torso joints and the local rotation angle of the root joint of the character. The torso joints are the joints of the character other than the root joint. The video frame data is input into the Mediapipe model to obtain the initial three-dimensional spatial coordinates of the character's joints; The initial three-dimensional spatial coordinates are repositioned to obtain the target three-dimensional spatial coordinates of the character's joints; Construct a target coordinate system with the root joint point as the origin based on the target's three-dimensional spatial coordinates; Construct the rotation matrix of the root joint based on the local rotation angle of the root joint and the target coordinate system; The rotation matrix is ​​converted into Euler angles of the root joint to obtain the global rotation angle of the root joint, and the global rotation angle of the root joint is used as the rotation angle of the root joint. The three-dimensional spatial coordinates of the character's joints are calculated based on the global rotation angle and the original three-dimensional spatial coordinates of the character's joints. The original three-dimensional spatial coordinates are the three-dimensional spatial coordinates of the character when it is in a bound posture.

8. The apparatus according to claim 6, characterized in that, The building unit is specifically used for: Create an initial driver file, which includes the original three-dimensional spatial coordinates and original rotation angles of the character's joints. The original three-dimensional spatial coordinates are the three-dimensional spatial coordinates of the character when it is in a tethered pose, and the original rotation angle is zero. Replace the original three-dimensional spatial coordinates with the three-dimensional spatial coordinates of the character's joints; The rotation angles of the character's joints are mirrored to obtain the mirrored rotation angles of the character's joints. Replace the original rotation angle with the mirror rotation angle; Set the scaling value to obtain the driver file.

9. A computer device, characterized in that, include: Memory, processor, and bus system; The memory is used to store programs; The processor is used to execute the program in the memory to implement the method of any one of claims 1 to 5; The bus system is used to connect the memory and the processor to enable communication between the memory and the processor.

10. A computer-readable storage medium, characterized in that, The device stores instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1 to 5.

Images