Universal 3D digital human real-time motion capture method and system based on monocular camera

Abstract

The invention provides a universal 3D digital human real-time motion capture method and system based on a monocular camera, and the method comprises the steps: S1, detecting 3D key points based on the monocular camera, aligning the key points to a unified coordinate system, detecting face 3D landmark points from a video, and outputting blendshape for driving the face of a digital human; s2, converting the key points into universal driving data, and converting 3D key point coordinates of the whole body skeleton into BVH rotation of the corresponding skeleton; s3, enabling the low-delay smooth data to drive the 3D digital human, and transmitting the facial blendshape parameters and the bone BVH to a rendering platform in a rotating manner; the skeleton rotating coordinate system is converted into a coordinate system of a rendering platform, and the 3D digital human is driven to act; and correspondingly controlling the digital human Morph Target by using a blendshape parameter, and driving the facial expression of the digital human. According to the method, the ROI regions of the trunk, the hands and the face are independently detected, so that the detection precision of the key points is improved; and the control data is a whole, so that the integrity of digital human control is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer graphics, and in particular to a method and system for capturing real-time motion of a universal 3D digital human based on a monocular camera. Background Art

[0002] Digital human technology combines advanced technologies such as artificial intelligence and computer graphics to create virtual characters that can simulate human interaction. This technology not only improves user experience and reduces corporate operating costs, but also enhances brand image. In the absence of face-to-face services, digital humans ensure the sustainability of services. With the continuous development of technology, the application potential of digital humans in data processing, decision support, and emotional interaction is gradually emerging, and their role in the future society will become increasingly important.

[0003] Real-time driving of digital humans, also known as motion capture technology, is a key component for digital humans to simulate natural human behavior. It captures the motion data of real humans, converts it into digital signals, and then drives virtual characters to perform the same or similar movements, making the behavior of digital humans more realistic and smooth. However, most high-precision motion capture technologies require professional motion capture equipment and a spacious enough venue to house these devices and capture the movements of actors. These devices usually include wearable sensors, optical cameras, and supporting software systems, which work together to accurately record and analyze human movements.

[0004] Because the above equipment and venues require high initial investment and operation and maintenance costs, a method of using a low-cost monocular camera to capture motion and drive 3D digital humans in real time is proposed. Artificial intelligence technology is introduced to detect key points of the human body, realizing low-cost and low-restriction real-time digital human driving.

[0005] Patent document A sensor-based digital motion capture system (application number: CN202310971385.5) discloses a sensor-based digital motion capture system, which relates to the field of motion capture technology. The method includes: a human skeleton model construction module, used to construct a human skeleton model; a sensor group configuration module, used to configure M inertial sensor groups based on the human skeleton model; a sensor data acquisition module, used to obtain M motion sensor data; a first verification module, used to obtain a first verification result; a second verification module, used to obtain a second verification result; a data compensation module, used to obtain M compensated sensor data; a motion capture module, used to obtain a motion capture result, which solves the technical problems in the prior art that the reliability of the basic data is insufficient due to insufficient credibility analysis of the sensor's collected data, and the reliability of the basic data is insufficient, and the verification compensation of the sensor's data collection results is realized, so as to reduce the error of the basic data, thereby ensuring the accuracy and reliability of the motion capture results. Technical effect.

[0006] This patent uses sensors as driving data for digital human motion capture to create a human skeleton model. Through the data verification module and the data compensation module, the reliability of the sensor captured data is increased, and the accuracy and reliability of the motion capture results are improved. However, because the overall components of the sensor cannot be changed at will, the human skeleton model is basically fixed, and the sensor cost is relatively high.

[0007] This patent does not use a monocular camera to capture the motion of digital humans, and cannot drive the motion of digital humans in real time at a low cost. This patent does not capture the model of 3D key points with high accuracy, and does not convert it into BVH data format, and cannot achieve universal control of digital humans, that is, universal motion capture.

[0008] Patent document Motion capture method and digital animation production method based on visual images (application number: CN202011043833.8) discloses a motion capture method and digital animation production method based on visual images. The purpose of the invention is to provide a motion capture method and digital animation production method based on visual images. The technical solution of the invention is: a motion capture method based on visual images, characterized in that: human body motion images synchronously acquired by multi-angle cameras are obtained; human body motion images at different angles are analyzed to obtain the coordinates of human body joint points on the image; and human body joint coordinates at different angles are fused and calculated to obtain 3D joint point coordinates. The invention is applicable to the field of computer image processing.

[0009] This patent uses multi-angle cameras to detect key points of the human body at the same time, and fuses them into 3D joint point coordinates. Because of the multi-angle cameras, the detected 3D joint point coordinates are more accurate. However, to fuse the data obtained by the multi-angle cameras, it is necessary to know the relative position of the cameras, which means that the cameras basically need to be fixed. In addition, this patent directly uses 3D joint point coordinates for motion capture, and cannot universally control various models. However, this invention does not use a monocular camera for key point detection, and cannot always keep the camera position unchanged; and this patent does not convert the skeletal joint rotation data into the BVH format, and cannot more universally perform motion capture to drive the movement of digital humans. Summary of the invention

[0010] In view of the defects in the prior art, the object of the present invention is to provide a universal 3D digital human real-time motion capture method and system based on a monocular camera.

[0011] A general 3D digital human real-time motion capture method based on a monocular camera provided by the present invention comprises:

[0012] Step S1: Detect 3D key points based on a monocular camera, align the key points to a unified coordinate system, detect facial 3D landmark points from the video, and output blendshape to drive the digital human face;

[0013] Step S2: convert the key points into general driving data, and convert the 3D key point coordinates of the whole body skeleton into the BVH rotation of the corresponding skeleton;

[0014] Step S3: Use low-latency smooth data to drive the 3D digital human, transmit the facial blendshape parameters and bone BVH rotation to the rendering platform; convert the bone rotation coordinate system to the coordinate system of the rendering platform to drive the 3D digital human's movements; use the blendshape parameters to control the digital human Morph Target and drive the digital human's facial expressions.

[0015] Preferably, in step S1:

[0016] Detect the 3D key points of the human skeleton from the monocular camera video image frame or the video image frame, including the key points of the torso skeleton and the key points of the hands, and align the key points to a unified coordinate system as the data basis for subsequent driving of the digital human body movement;

[0017] The mediapipe face landmark algorithm is used to detect facial expressions. The face is recognized from the input image and cropped out. The coordinates of the facial landmark points are detected and the blendshape parameters corresponding to the landmark points are output. Blendshape is a morphological target animation that smoothly transitions between different facial expressions or shapes through linear interpolation. The facial expressions of digital humans are controlled through blendshape parameters.

[0018] The human body ROI area and the hand ROI area are cropped out, and the mediapipe pose landmark detection algorithm and mediapipe hand landmark are used to obtain the 3D coordinates and pixel coordinates of the body key points and hand key points; the body 3D key points take the center of the two hip joints as the origin, and the hand 3D key points coordinate system takes the wrist of each hand as the origin;

[0019] After obtaining the 3D coordinates of the body and hand relative to their respective coordinate systems, align the coordinate systems to the same coordinate system; the body coordinates and hand coordinates have the same wrist key point, and with the help of the coordinates of the wrist key point in two different coordinate systems and their respective pixel coordinates, align the coordinates of the finger key points in the hand key points to the body coordinate system;

[0020] The pixel coordinates of the finger key point are finger pixel, the 3D coordinate with the wrist as the origin is finger real ; The pixel coordinate of the midpoint of the hip joint is middle pixel ; The pixel coordinates of the wrist key point are wrist pixel , the 3D coordinates with the wrist as the origin are wrist real , the 3D coordinates of the specific finger key points in the coordinate system with the midpoint of the hip joint as the origin are:

[0021]

[0022] finger z =wrist real .z*+finger real .z(3)

[0023] Among them, finger x is the x-coordinate value of the finger key point in the coordinate system with the midpoint of the hip joint as the origin, finger y is the y coordinate value in this coordinate system, finger z is the z coordinate value in this coordinate system, wrist real .x and other.x refers to wrist real The x coordinate value in 3D coordinates, and the other .yz is the corresponding yz coordinate value;

[0024] The coordinates of the key points of the fingers are aligned to the coordinate system with the midpoint of the hip joint as the origin, which is used for the subsequent overall driving of the digital human.

[0025] Preferably, in step S2:

[0026] Based on the parent-child relationship of bones, the coordinates of the 3D key points of the whole body skeleton are converted into the BVH rotation of the corresponding bones, in preparation for the subsequent driving of the 3D digital human;

[0027] The movement of the digital human skeleton starts from the root bone. Other bones continue to move in layers based on the parent bone until the end bone. The movement of each bone will affect the state of its child bones. Before the child bone moves, its coordinate axis is the same as that of the parent bone. Each movement of the child bone is to continue to rotate based on the rotation of the parent bone and affect the bones of the subsequent layers.

[0028] After obtaining the 3D coordinates of the key points of the whole body, the key point coordinates are converted into the rotation of each key point relative to the parent bone using the BVH format as the standard; the BVH format is used to store human motion capture data, with nodes as the core elements, recording the motion information of the human skeleton in consecutive frames. The BVH file contains two main parts: the hierarchical structure and motion information. The default state of the BVH format is that the bones present Tpose, and the initial coordinate system of each bone of the body is the same, consistent with the root bone.

[0029] Preferably, the steps for calculating the rotation of the target bone relative to the parent bone are as follows:

[0030] Starting from the root bone, calculate the cosine matrix of the bones relative to the initial state in order according to the hierarchy: In bone rotation, each bone will have an axis that always points to its child bones, and the two axes will always be in the same plane. Through the 3D coordinates of the whole body bones, get the direction vector i, j, k of each axis in the initial coordinate system, and calculate the cosine matrix R of the coordinate axis of the target bone relative to the initial coordinate axis:

[0031]

[0032] Among them, i x is the x component of the direction vector i in the initial coordinate system, j x is the x component of the direction vector j in the initial coordinate system, k x is the x component of the direction vector k in the initial coordinate system; i y is the y component of the direction vector i in the initial coordinate system, j y is the y component of the direction vector j in the initial coordinate system, k y is the y component of the direction vector k in the initial coordinate system; i z is the z component of the direction vector i in the initial coordinate system, j z is the z component of the direction vector j in the initial coordinate system, k z is the z component of the direction vector k in the initial coordinate system;

[0033] Convert the direction cosine matrix to quaternion: Convert the direction cosine matrix to quaternion for subsequent calculations to obtain the rotation quaternion quat of the bone relative to the initial coordinate axis. target :

[0034] quat target =(w,x,y,z)(5)

[0035] Among them, w represents the degree of rotation, and its value is related to the rotation angle; xyz is a three-dimensional vector, which is a direction unit vector and defines the direction of the rotation axis;

[0036] Get the rotation relative to the parent bone: Get the rotation quaternion of the target bone relative to the initial coordinate axis, and the rotation quaternion of the parent bone relative to the initial coordinate axis. According to the characteristics of the quaternion, calculate the rotation quaternion Local relative to the parent bone. quat :

[0037] Local quat =quat target -quat father (6)

[0038] Among them, quat father The rotation quaternion of the parent bone, quat target The rotation quaternion of the current bone;

[0039] Get BVH format rotation according to quaternion: According to the rotation order of the set BVH format, Local quat Convert to Euler angle to represent rotation; the rotation order represents the order of Euler angle rotation of a certain axis; get the rotation of the bone corresponding to each key point relative to its parent bone for subsequent driving of the 3D digital human.

[0040] Preferably, in step S3:

[0041] Transmit facial blendshape parameters and bone BVH rotation to the rendering platform through the socket; convert the bone rotation coordinate system to the rendering platform coordinate system to drive the 3D digital human action;

[0042] After the driving data is obtained, it is transmitted to the rendering platform and drives the 3D digital human. After the human body's bone rotation and facial blendshape parameters are obtained, the data is smoothed and transmitted to the rendering platform through socket communication.

[0043] The smoothing process uses a sliding weighted average filter to retain the most recent N frames of data. When setting the weight, the closer the data is to the latest frame, the higher the weight is. The smoothed data is obtained using the following formula:

[0044]

[0045] w i =decadeRate N-i ,(0 <decadeRate<1)

[0046]

[0047] Where y(t) is the latest data value after smoothing, N is the number of the most recently preset multi-frame data recorded, n is the sequence number of a certain data from the beginning to the end of the recorded data, decadeRate is the rate of change of the weight, and w is the weight value corresponding to each data calculated by decadeRate. i w j Represents the weight corresponding to the ijth value, and the subscript represents the corresponding serial number. is the weight value that needs to be multiplied by the value corresponding to the normalized sequence number n, and x(n) is the nth value recorded;

[0048] Obtain facial blendshape parameters in the rendering platform, and control facial expressions based on each parameter corresponding to the model's morphtarget; generate expressions based on the weighted superposition of each expression parameter;

[0049] For the acquired bone rotation, quaternion is used as an intermediate variable to convert the bone rotation to the rotation order corresponding to the rendering platform. According to the bone rotation, starting from the root bone, the rotation of the digital human skeleton is controlled in sequence according to the hierarchical order to achieve the purpose of controlling the movement of the digital human.

[0050] According to the present invention, a universal 3D digital human real-time motion capture system based on a monocular camera is provided, comprising:

[0051] Module M1: Detect 3D key points based on a monocular camera, align the key points to a unified coordinate system, detect facial 3D landmark points from the video, and output blendshape to drive the digital human face;

[0052] Module M2: Convert key points into general driving data, convert the 3D key point coordinates of the whole body skeleton into the BVH rotation of the corresponding bones;

[0053] Module M3: Use low-latency smooth data to drive the 3D digital human, transmit the facial blendshape parameters and bone BVH rotation to the rendering platform; convert the bone rotation coordinate system to the rendering platform coordinate system to drive the 3D digital human's movements; use the blendshape parameters to control the digital human Morph Target and drive the digital human's facial expressions.

[0054] Preferably, in the module M1:

[0055] Detect the 3D key points of the human skeleton from the monocular camera video image frame or the video image frame, including the key points of the torso skeleton and the key points of the hands, and align the key points to a unified coordinate system as the data basis for subsequent driving of the digital human body movement;

[0056] The mediapipe face landmark algorithm is used to detect facial expressions. The face is recognized from the input image and cropped out. The coordinates of the facial landmark points are detected and the blendshape parameters corresponding to the landmark points are output. Blendshape is a morphological target animation that smoothly transitions between different facial expressions or shapes through linear interpolation. The facial expressions of digital humans are controlled through blendshape parameters.

[0057] The human body ROI area and the hand ROI area are cropped out, and the mediapipe pose landmark detection algorithm and mediapipe hand landmark are used to obtain the 3D coordinates and pixel coordinates of the body key points and hand key points; the body 3D key points take the center of the two hip joints as the origin, and the hand 3D key points coordinate system takes the wrist of each hand as the origin;

[0058] After obtaining the 3D coordinates of the body and hand relative to their respective coordinate systems, align the coordinate systems to the same coordinate system; the body coordinates and hand coordinates have the same wrist key point, and with the help of the coordinates of the wrist key point in two different coordinate systems and their respective pixel coordinates, align the coordinates of the finger key points in the hand key points to the body coordinate system;

[0059] The pixel coordinates of the finger key point are finger pixel , the 3D coordinate with the wrist as the origin is finger real ; The pixel coordinate of the midpoint of the hip joint is middle pixel ; The pixel coordinates of the wrist key point are wrist pixel , the 3D coordinates with the wrist as the origin are wrist real , the 3D coordinates of the specific finger key points in the coordinate system with the midpoint of the hip joint as the origin are:

[0060]

[0061] finger z =wrist real .z*+finger real .z(3)

[0062] Among them, finger x is the x-coordinate value of the finger key point in the coordinate system with the midpoint of the hip joint as the origin, finger y is the y coordinate value in this coordinate system, finger z is the z coordinate value in this coordinate system, wrist real .x and other.x refers to wrist realThe x coordinate value in 3D coordinates, and the other .yz is the corresponding yz coordinate value;

[0063] The coordinates of the key points of the fingers are aligned to the coordinate system with the midpoint of the hip joint as the origin, which is used for the subsequent overall driving of the digital human.

[0064] Preferably, in the module M2:

[0065] Based on the parent-child relationship of bones, the coordinates of the 3D key points of the whole body skeleton are converted into the BVH rotation of the corresponding bones, in preparation for the subsequent driving of the 3D digital human;

[0066] The movement of the digital human skeleton starts from the root bone. Other bones continue to move in layers based on the parent bone until the end bone. The movement of each bone will affect the state of its child bones. Before the child bone moves, its coordinate axis is the same as that of the parent bone. Each movement of the child bone is to continue to rotate based on the rotation of the parent bone and affect the bones of the subsequent layers.

[0067] After obtaining the 3D coordinates of the key points of the whole body, the key point coordinates are converted into the rotation of each key point relative to the parent bone using the BVH format as the standard; the BVH format is used to store human motion capture data, with nodes as the core elements, recording the motion information of the human skeleton in consecutive frames. The BVH file contains two main parts: the hierarchical structure and motion information. The default state of the BVH format is that the bones present Tpose, and the initial coordinate system of each bone of the body is the same, consistent with the root bone.

[0068] Preferably, the steps for calculating the rotation of the target bone relative to the parent bone are as follows:

[0069] Starting from the root bone, calculate the cosine matrix of the bones relative to the initial state in order according to the hierarchy: In bone rotation, each bone will have an axis that always points to its child bones, and the two axes will always be in the same plane. Through the 3D coordinates of the whole body bones, get the direction vector i, j, k of each axis in the initial coordinate system, and calculate the cosine matrix R of the coordinate axis of the target bone relative to the initial coordinate axis:

[0070]

[0071] Among them, i x is the x component of the direction vector i in the initial coordinate system, j x is the x component of the direction vector j in the initial coordinate system, k x is the x component of the direction vector k in the initial coordinate system; i y is the y component of the direction vector i in the initial coordinate system, j y is the y component of the direction vector j in the initial coordinate system, k yis the y component of the direction vector k in the initial coordinate system; i z is the z component of the direction vector i in the initial coordinate system, j z is the z component of the direction vector j in the initial coordinate system, k z is the z component of the direction vector k in the initial coordinate system;

[0072] Convert the direction cosine matrix to quaternion: Convert the direction cosine matrix to quaternion for subsequent calculations to obtain the rotation quaternion quat of the bone relative to the initial coordinate axis. target :

[0073] quat target =(w,x,y,z)(5)

[0074] Among them, w represents the degree of rotation, and its value is related to the rotation angle; xyz is a three-dimensional vector, which is a direction unit vector and defines the direction of the rotation axis;

[0075] Get the rotation relative to the parent bone: Get the rotation quaternion of the target bone relative to the initial coordinate axis, and the rotation quaternion of the parent bone relative to the initial coordinate axis. According to the characteristics of the quaternion, calculate the rotation quaternion Local relative to the parent bone. quat :

[0076] Local quat =quat target -quat father (6)

[0077] Among them, quat father The rotation quaternion of the parent bone, quat target The rotation quaternion of the current bone;

[0078] Get BVH format rotation according to quaternion: According to the rotation order of the set BVH format, Local quat Convert to Euler angle to represent rotation; the rotation order represents the order of Euler angle rotation of a certain axis; get the rotation of the bone corresponding to each key point relative to its parent bone for subsequent driving of the 3D digital human.

[0079] Preferably, in the module M3:

[0080] Transmit facial blendshape parameters and bone BVH rotation to the rendering platform through the socket; convert the bone rotation coordinate system to the rendering platform coordinate system to drive the 3D digital human action;

[0081] After the driving data is obtained, it is transmitted to the rendering platform and drives the 3D digital human. After the human body's bone rotation and facial blendshape parameters are obtained, the data is smoothed and transmitted to the rendering platform through socket communication.

[0082] The smoothing process uses a sliding weighted average filter to retain the most recent N frames of data. When setting the weight, the closer the data is to the latest frame, the higher the weight is. The smoothed data is obtained using the following formula:

[0083]

[0084] w i =decadeRate N-i ,(0 <decadeRate<1)

[0085]

[0086] Where y(t) is the latest data value after smoothing, N is the number of the most recently preset multi-frame data recorded, n is the sequence number of a certain data from the beginning to the end of the recorded data, decadeRate is the rate of change of the weight, and w is the weight value corresponding to each data calculated by decadeRate. i w j Represents the weight corresponding to the ijth value, and the subscript represents the corresponding serial number. is the weight value that needs to be multiplied by the value corresponding to the normalized sequence number n, and x(n) is the nth value recorded;

[0087] Obtain facial blendshape parameters in the rendering platform, and control facial expressions based on each parameter corresponding to the morphtarget of the model; generate expressions based on the weighted superposition of each expression parameter;

[0088] For the acquired bone rotation, quaternion is used as an intermediate variable to convert the bone rotation to the rotation order corresponding to the rendering platform. According to the bone rotation, starting from the root bone, the rotation of the digital human skeleton is controlled in sequence according to the hierarchical order to achieve the purpose of controlling the movement of the digital human.

[0089] Compared with the prior art, the present invention has the following beneficial effects:

[0090] 1. The present invention achieves overall control of the digital human by acquiring key points of the torso, hands, and face and aligning them to the same coordinate system;

[0091] 2. The present invention improves the detection accuracy of key points by separately detecting the ROI areas of the torso, hands and face; and the control data is a whole, which improves the completeness of digital human control;

[0092] 3. The present invention is a method for controlling a universal 3D digital human. In terms of facial expressions, blendshape data is generated to control the facial expressions of the digital human. In terms of body control, the movement of the digital human is controlled by converting the data into universal BVH format data. The present invention can universally control various digital humans and has a very high value. BRIEF DESCRIPTION OF THE DRAWINGS

[0093] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0094] Figure 1 This is a flow chart for monocular motion capture;

[0095] Figure 2 This is a schematic diagram of a facial landmark image;

[0096] Figure 3 Schematic diagram of key points and coordinate system of human body;

[0097] Figure 4 This is a schematic diagram of the bone rotation coordinate axis. DETAILED DESCRIPTION

[0098] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0099] Embodiment 1:

[0100] The present invention belongs to the field of computer vision. The technical problems to be solved by the present invention are embodied in the following aspects:

[0101] (1) The present invention proposes a low-cost method for capturing full-body 3D key points based on a monocular camera, which obtains the coordinates of the 3D key points of the human body, the 3D key points of the hands, and the 3D key points of the face from the RGB image and aligns them in the same coordinate system.

[0102] (2) The present invention proposes a method for driving 3D digital humans with universal key points, which converts the 3D key points of the whole body into BVH format data for universal driving of 3D digital human motion and facial blendshape; and performs low-latency smoothing processing on the data to achieve the purpose of real-time control of the digital human. This method can be used to drive a humanoid digital human, and can still be driven in real time after replacing a digital human with different skeletons.

[0103] (3) The method of the present invention can obtain the universal BVH format data driving the movement of digital humans and facial blendshape in real time through a monocular RGB camera, thereby realizing low-cost universal digital human motion capture.

[0104] A general 3D digital human real-time motion capture method based on a monocular camera provided by the present invention comprises:

[0105] Step S1: Detect 3D key points based on a monocular camera, align the key points to a unified coordinate system, detect facial 3D landmark points from the video, and output blendshape to drive the digital human face;

[0106] Specifically, in step S1:

[0107] Detect the 3D key points of the human skeleton from the monocular camera video image frame or the video image frame, including the key points of the torso skeleton and the key points of the hands, and align the key points to a unified coordinate system as the data basis for subsequent driving of the digital human body movement;

[0108] The mediapipe face landmark algorithm is used to detect facial expressions. The face is recognized from the input image and cropped out. The coordinates of the facial landmark points are detected and the blendshape parameters corresponding to the landmark points are output. Blendshape is a morphological target animation that smoothly transitions between different facial expressions or shapes through linear interpolation. The facial expressions of digital humans are controlled through blendshape parameters.

[0109] The human body ROI area and the hand ROI area are cropped out, and the mediapipe pose landmark detection algorithm and mediapipe hand landmark are used to obtain the 3D coordinates and pixel coordinates of the body key points and hand key points; the body 3D key points take the center of the two hip joints as the origin, and the hand 3D key points coordinate system takes the wrist of each hand as the origin;

[0110] After obtaining the 3D coordinates of the body and hand relative to their respective coordinate systems, align the coordinate systems to the same coordinate system; the body coordinates and hand coordinates have the same wrist key point, and with the help of the coordinates of the wrist key point in two different coordinate systems and their respective pixel coordinates, align the coordinates of the finger key points in the hand key points to the body coordinate system;

[0111] The pixel coordinates of the finger key point are finger pixel , the 3D coordinate with the wrist as the origin is finger real ; The pixel coordinate of the midpoint of the hip joint is middle pixel ; The pixel coordinates of the wrist key point are wrist pixel , the 3D coordinates with the wrist as the origin are wrist real , the 3D coordinates of the specific finger key points in the coordinate system with the midpoint of the hip joint as the origin are:

[0112]

[0113] finger z =wrist real .z*+finger real .z(3)

[0114] Among them, finger x is the x-coordinate value of the finger key point in the coordinate system with the midpoint of the hip joint as the origin, finger y is the y coordinate value in this coordinate system, finger z is the z coordinate value in this coordinate system, wrist real .x and other.x refers to wrist real The x coordinate value in 3D coordinates, and the other .yz is the corresponding yz coordinate value;

[0115] The coordinates of the key points of the fingers are aligned to the coordinate system with the midpoint of the hip joint as the origin, which is used for the subsequent overall driving of the digital human.

[0116] Step S2: convert the key points into general driving data, and convert the 3D key point coordinates of the whole body skeleton into the BVH rotation of the corresponding skeleton;

[0117] Specifically, in step S2:

[0118] Based on the parent-child relationship of bones, the coordinates of the 3D key points of the whole body skeleton are converted into the BVH rotation of the corresponding bones, in preparation for the subsequent driving of the 3D digital human;

[0119] The movement of the digital human skeleton starts from the root bone. Other bones continue to move in layers based on the parent bone until the end bone. The movement of each bone will affect the state of its child bones. Before the child bone moves, its coordinate axis is the same as that of the parent bone. Each movement of the child bone is to continue to rotate based on the rotation of the parent bone and affect the bones of the subsequent layers.

[0120] After obtaining the 3D coordinates of the key points of the whole body, the key point coordinates are converted into the rotation of each key point relative to the parent bone using the BVH format as the standard; the BVH format is used to store human motion capture data, with nodes as the core elements, recording the motion information of the human skeleton in consecutive frames. The BVH file contains two main parts: the hierarchical structure and motion information. The default state of the BVH format is that the bones present Tpose, and the initial coordinate system of each bone of the body is the same, consistent with the root bone.

[0121] Specifically, the steps to calculate the rotation of the target bone relative to the parent bone are as follows:

[0122] Starting from the root bone, calculate the cosine matrix of the bones relative to the initial state in order according to the hierarchy: In bone rotation, each bone will have an axis that always points to its child bones, and the two axes will always be in the same plane. Through the 3D coordinates of the whole body bones, get the direction vector i, j, k of each axis in the initial coordinate system, and calculate the cosine matrix R of the coordinate axis of the target bone relative to the initial coordinate axis:

[0123]

[0124] Among them, i x is the x component of the direction vector i in the initial coordinate system, j x is the x component of the direction vector j in the initial coordinate system, k x is the x component of the direction vector k in the initial coordinate system; i y is the y component of the direction vector i in the initial coordinate system, j y is the y component of the direction vector j in the initial coordinate system, k y is the y component of the direction vector k in the initial coordinate system; i z is the z component of the direction vector i in the initial coordinate system, j z is the z component of the direction vector j in the initial coordinate system, k z is the z component of the direction vector k in the initial coordinate system;

[0125] Convert the direction cosine matrix to quaternion: Convert the direction cosine matrix to quaternion for subsequent calculations to obtain the rotation quaternion quat of the bone relative to the initial coordinate axis. target :

[0126] quat target =(w,x,y,z)(5)

[0127] Among them, w represents the degree of rotation, and its value is related to the rotation angle; xyz is a three-dimensional vector, which is a direction unit vector and defines the direction of the rotation axis;

[0128] Get the rotation relative to the parent bone: Get the rotation quaternion of the target bone relative to the initial coordinate axis, and the rotation quaternion of the parent bone relative to the initial coordinate axis. According to the characteristics of the quaternion, calculate the rotation quaternion Local relative to the parent bone. quat :

[0129] Local quat =quat target -quat father (6)

[0130] Among them, quat father The rotation quaternion of the parent bone, quat target The rotation quaternion of the current bone;

[0131] Get BVH format rotation according to quaternion: According to the rotation order of the set BVH format, Local quat Convert to Euler angle to represent rotation; the rotation order represents the order of Euler angle rotation of a certain axis; get the rotation of the bone corresponding to each key point relative to its parent bone for subsequent driving of the 3D digital human.

[0132] Step S3: Use low-latency smooth data to drive the 3D digital human, transmit the facial blendshape parameters and bone BVH rotation to the rendering platform; convert the bone rotation coordinate system to the coordinate system of the rendering platform to drive the 3D digital human's movements; use the blendshape parameters to control the digital human Morph Target and drive the digital human's facial expressions.

[0133] Specifically, in step S3:

[0134] Transmit facial blendshape parameters and bone BVH rotation to the rendering platform through the socket; convert the bone rotation coordinate system to the rendering platform coordinate system to drive the 3D digital human action;

[0135] After the driving data is obtained, it is transmitted to the rendering platform and drives the 3D digital human. After the human body's bone rotation and facial blendshape parameters are obtained, the data is smoothed and transmitted to the rendering platform through socket communication.

[0136] The smoothing process uses a sliding weighted average filter to retain the most recent N frames of data. When setting the weight, the closer the data is to the latest frame, the higher the weight is. The smoothed data is obtained using the following formula:

[0137]

[0138] w i =decadeRate N-i ,(0 <decadeRate<1)

[0139]

[0140] Where y(t) is the latest data value after smoothing, N is the number of the most recently preset multi-frame data recorded, n is the sequence number of a certain data from the beginning to the end of the recorded data, decadeRate is the rate of change of the weight, and w is the weight value corresponding to each data calculated by decadeRate. i w j Represents the weight corresponding to the ijth value, and the subscript represents the corresponding serial number. is the weight value that needs to be multiplied by the value corresponding to the normalized sequence number n, and x(n) is the nth value recorded;

[0141] Obtain facial blendshape parameters in the rendering platform, and control facial expressions based on each parameter corresponding to the morphtarget of the model; generate expressions based on the weighted superposition of each expression parameter;

[0142] For the acquired bone rotation, quaternion is used as an intermediate variable to convert the bone rotation to the rotation order corresponding to the rendering platform. According to the bone rotation, starting from the root bone, the rotation of the digital human skeleton is controlled in sequence according to the hierarchical order to achieve the purpose of controlling the movement of the digital human.

[0143] Embodiment 2:

[0144] Embodiment 2 is a preferred example of Embodiment 1, and is used to illustrate the present invention in more detail.

[0145] The present invention also provides a universal 3D digital human real-time motion capture system based on a monocular camera. The universal 3D digital human real-time motion capture system based on a monocular camera can be implemented by executing the process steps of the universal 3D digital human real-time motion capture method based on a monocular camera, that is, those skilled in the art can understand the universal 3D digital human real-time motion capture method based on a monocular camera as a preferred implementation of the universal 3D digital human real-time motion capture system based on a monocular camera.

[0146] According to the present invention, a universal 3D digital human real-time motion capture system based on a monocular camera is provided, comprising:

[0147] Module M1: Detect 3D key points based on a monocular camera, align the key points to a unified coordinate system, detect facial 3D landmark points from the video, and output blendshape to drive the digital human face;

[0148] Specifically, in the module M1:

[0149] Detect the 3D key points of the human skeleton from the monocular camera video image frame or the video image frame, including the key points of the torso skeleton and the key points of the hands, and align the key points to a unified coordinate system as the data basis for subsequent driving of the digital human body movement;

[0150] The mediapipe face landmark algorithm is used to detect facial expressions. The face is recognized from the input image and cropped out. The coordinates of the facial landmark points are detected and the blendshape parameters corresponding to the landmark points are output. Blendshape is a morphological target animation that smoothly transitions between different facial expressions or shapes through linear interpolation. The facial expressions of digital humans are controlled through blendshape parameters.

[0151] The human body ROI area and the hand ROI area are cropped out, and the mediapipe pose landmark detection algorithm and mediapipe hand landmark are used to obtain the 3D coordinates and pixel coordinates of the body key points and hand key points; the body 3D key points take the center of the two hip joints as the origin, and the hand 3D key points coordinate system takes the wrist of each hand as the origin;

[0152] After obtaining the 3D coordinates of the body and hand relative to their respective coordinate systems, align the coordinate systems to the same coordinate system; the body coordinates and hand coordinates have the same wrist key point, and with the help of the coordinates of the wrist key point in two different coordinate systems and their respective pixel coordinates, align the coordinates of the finger key points in the hand key points to the body coordinate system;

[0153] The pixel coordinates of the finger key point are finger pixel , the 3D coordinate with the wrist as the origin is finger real ; The pixel coordinate of the midpoint of the hip joint is middle pixel ; The pixel coordinates of the wrist key point are wrist pixel , the 3D coordinates with the wrist as the origin are wrist real , the 3D coordinates of the specific finger key points in the coordinate system with the midpoint of the hip joint as the origin are:

[0154]

[0155] finger z =wrist real .z*+finger real .z(3)

[0156] Among them, finger x is the x-coordinate value of the finger key point in the coordinate system with the midpoint of the hip joint as the origin, finger y is the y coordinate value in this coordinate system, finger z is the z coordinate value in this coordinate system, wrist real .x and other.x refers to wrist real The x coordinate value in 3D coordinates, and the other .yz is the corresponding yz coordinate value;

[0157] The coordinates of the key points of the fingers are aligned to the coordinate system with the midpoint of the hip joint as the origin, which is used for the subsequent overall driving of the digital human.

[0158] Module M2: Convert key points into general driving data, convert the 3D key point coordinates of the whole body skeleton into the BVH rotation of the corresponding bones;

[0159] Specifically, in the module M2:

[0160] Based on the parent-child relationship of bones, the coordinates of the 3D key points of the whole body skeleton are converted into the BVH rotation of the corresponding bones, in preparation for the subsequent driving of the 3D digital human;

[0161] The movement of the digital human skeleton starts from the root bone. Other bones continue to move in layers based on the parent bone until the end bone. The movement of each bone will affect the state of its child bones. Before the child bone moves, its coordinate axis is the same as that of the parent bone. Each movement of the child bone is to continue to rotate based on the rotation of the parent bone and affect the bones of the subsequent layers.

[0162] After obtaining the 3D coordinates of the key points of the whole body, the key point coordinates are converted into the rotation of each key point relative to the parent bone using the BVH format as the standard; the BVH format is used to store human motion capture data, with nodes as the core elements, recording the motion information of the human skeleton in consecutive frames. The BVH file contains two main parts: the hierarchical structure and motion information. The default state of the BVH format is that the bones present Tpose, and the initial coordinate system of each bone of the body is the same, consistent with the root bone.

[0163] Specifically, the steps to calculate the rotation of the target bone relative to the parent bone are as follows:

[0164] Starting from the root bone, calculate the cosine matrix of the bones relative to the initial state in order according to the hierarchy: In bone rotation, each bone will have an axis that always points to its child bones, and the two axes will always be in the same plane. Through the 3D coordinates of the whole body bones, get the direction vector i, j, k of each axis in the initial coordinate system, and calculate the cosine matrix R of the coordinate axis of the target bone relative to the initial coordinate axis:

[0165]

[0166] Among them, i x is the x component of the direction vector i in the initial coordinate system, j x is the x component of the direction vector j in the initial coordinate system, k x is the x component of the direction vector k in the initial coordinate system; i y is the y component of the direction vector i in the initial coordinate system, j y is the y component of the direction vector j in the initial coordinate system, k y is the y component of the direction vector k in the initial coordinate system; i z is the z component of the direction vector i in the initial coordinate system, j z is the z component of the direction vector j in the initial coordinate system, k z is the z component of the direction vector k in the initial coordinate system;

[0167] Convert the direction cosine matrix to quaternion: Convert the direction cosine matrix to quaternion for subsequent calculations to obtain the rotation quaternion quat of the bone relative to the initial coordinate axis. target :

[0168] quat target =(w,x,y,z)(5)

[0169] Among them, w represents the degree of rotation, and its value is related to the rotation angle; xyz is a three-dimensional vector, which is a direction unit vector and defines the direction of the rotation axis;

[0170] Get the rotation relative to the parent bone: Get the rotation quaternion of the target bone relative to the initial coordinate axis, and the rotation quaternion of the parent bone relative to the initial coordinate axis. According to the characteristics of the quaternion, calculate the rotation quaternion Local relative to the parent bone. quat :

[0171] Local quat =quat target -quat father (6)

[0172] Among them, quat father The rotation quaternion of the parent bone, quat targetThe rotation quaternion of the current bone;

[0173] Get BVH format rotation according to quaternion: According to the rotation order of the set BVH format, Local quat Convert to Euler angle to represent rotation; the rotation order represents the order of Euler angle rotation of a certain axis; get the rotation of the bone corresponding to each key point relative to its parent bone for subsequent driving of the 3D digital human.

[0174] Module M3: Use low-latency smooth data to drive the 3D digital human, transmit the facial blendshape parameters and bone BVH rotation to the rendering platform; convert the bone rotation coordinate system to the rendering platform coordinate system to drive the 3D digital human's movements; use the blendshape parameters to control the digital human Morph Target and drive the digital human's facial expressions.

[0175] Specifically, in the module M3:

[0176] Transmit facial blendshape parameters and bone BVH rotation to the rendering platform through the socket; convert the bone rotation coordinate system to the rendering platform coordinate system to drive the 3D digital human action;

[0177] After the driving data is obtained, it is transmitted to the rendering platform and drives the 3D digital human. After the human body's bone rotation and facial blendshape parameters are obtained, the data is smoothed and transmitted to the rendering platform through socket communication.

[0178] The smoothing process uses a sliding weighted average filter to retain the most recent N frames of data. When setting the weight, the closer the data is to the latest frame, the higher the weight is. The smoothed data is obtained using the following formula:

[0179]

[0180] w i =decadeRate N-i ,(0 <decadeRate<1)

[0181]

[0182] Where y(t) is the latest data value after smoothing, N is the number of the most recently preset multi-frame data recorded, n is the sequence number of a certain data from the beginning to the end of the recorded data, decadeRate is the rate of change of the weight, and w is the weight value corresponding to each data calculated by decadeRate. i w j Represents the weight corresponding to the ijth value, and the subscript represents the corresponding serial number. is the weight value that needs to be multiplied by the value corresponding to the normalized sequence number n, and x(n) is the nth value recorded;

[0183] Obtain facial blendshape parameters in the rendering platform, and control facial expressions based on each parameter corresponding to the morphtarget of the model; generate expressions based on the weighted superposition of each expression parameter;

[0184] For the acquired bone rotation, quaternion is used as an intermediate variable to convert the bone rotation to the rotation order corresponding to the rendering platform. According to the bone rotation, starting from the root bone, the rotation of the digital human skeleton is controlled in sequence according to the hierarchical order to achieve the purpose of controlling the movement of the digital human.

[0185] Embodiment 3:

[0186] Embodiment 3 is a preferred example of Embodiment 1, and is used to illustrate the present invention in more detail.

[0187] This method consists of three parts: detecting 3D key points based on a monocular camera, converting key points into general driving data, and driving 3D digital humans with low-latency smooth data.

[0188] 1) Overall architecture:

[0189] This paper proposes a universal 3D digital human real-time motion capture method based on a monocular RGB camera, which is used for low-cost real-time 3D digital human motion capture and driving, and can be applied to different digital human models. The overall process of this method is shown in Figure 1 ,The basic description of each part is as follows.

[0190] Detect 3D key points based on a monocular camera: Detect the 3D key points of the human skeleton, including the torso skeleton key points and the hand key points, from the monocular camera video image frame or the video image frame, and align the key points to a unified coordinate system as the data basis for subsequent driving of the digital human body movement; detect the facial 3D landmark points from the video, and output the blendshape to drive the digital human face;

[0191] Key point conversion general driving data: Based on the parent-child relationship of bones, the 3D key point coordinates of the whole body skeleton are converted into the BVH rotation of the corresponding bones, preparing for the subsequent driving of 3D digital humans;

[0192] Low-latency smooth data drives 3D digital humans: transmit facial blendshape parameters and bone BVH rotation to the rendering platform through the socket; convert the bone rotation coordinate system to the rendering platform's coordinate system, and then drive the 3D digital human's movements; use the blendshape parameters to control the digital human Morph Target and drive the digital human's facial expressions.

[0193] 2) Detect 3D key points based on a monocular camera:

[0194] 3D key points are the basic data for driving the movement of digital humans, and the overall driving of digital humans requires the control of body key points, finger key points, and facial expression data. Common methods use inertial motion capture equipment and multi-angle optical motion capture equipment, but the cost is relatively high. This method uses a low-cost monocular camera to detect 3D key points of the human body in the image, including the body, fingers, and face.

[0195] First, the mediapipe face landmark algorithm is used to detect facial expressions. The face is first identified from the input image, cropped, and then input into the algorithm. The detection results are as follows: Figure 2 The coordinates of the 478 landmark points of the face shown in the figure are obtained, and the blendshape parameters corresponding to the landmark points are output. Blendshape is morph target animation, which is a 3D graphics technology used to smoothly transition between different facial expressions or shapes through linear interpolation. With this data, most digital human facial expressions can be controlled.

[0196] Then, the human body ROI area and the hand ROI area are cropped out, and the mediapipe pose landmark detection algorithm and mediapipe hand landmark are used to obtain the 3D coordinates and pixel coordinates of the body key points and hand key points. The coordinate system of the body 3D key points is as follows Figure 3 As shown, the coordinate system of the 3D key points of the hand takes the center of the two hip joints as the origin, and the wrist of each hand as the origin.

[0197] After obtaining the 3D coordinates of the body and hand relative to their respective coordinate systems, the coordinate systems need to be aligned to the same coordinate system. The body coordinates and hand coordinates have the same key point, which is the wrist key point. With the help of the coordinates of the wrist key point in two different coordinate systems and their respective pixel coordinates, the coordinates of the 20 key points of the fingers in the hand key point are aligned to the body coordinate system.

[0198] Taking a key point of a finger as an example, its pixel coordinate is finger_pixel, and its 3D coordinate with the wrist as the origin is finger_real; the pixel coordinate of the midpoint of the hip joint is middle_pixel; the pixel coordinate of the wrist key point is wrist_pixel, and its 3D coordinate with the wrist as the origin is wrist_real. The specific 3D coordinate of the finger key point with the midpoint of the hip joint as the origin is:

[0199]

[0200] finger z =wrist real .z*+finger real .z#(3)

[0201] Finally, the coordinates of the finger key points are aligned to the coordinate system with the midpoint of the hip joint as the origin, which can be used for subsequent overall driving of the digital human.

[0202] 3) Key point conversion general drive data:

[0203] The movement of the digital human skeleton starts from the root bone, and other bones continue to move in layers based on the parent bone until the end bone. The movement of each bone will affect the state of its child bones, that is, before the child bone moves, its coordinate axis should be the same as the parent bone. Because the length of the bone is fixed, each movement of the child bone continues to rotate based on the rotation of the parent bone and affects the bones of the subsequent layers.

[0204] After obtaining the key points, the rotation of the model skeleton is generally calculated for the target model. However, this method cannot quickly drive other related parts. Therefore, after obtaining the 3D coordinates of the key points of the whole body, in order to drive the movement of most digital humans, this method uses the BVH format as the standard to convert the key point coordinates into the rotation of each key point relative to the parent bone. The BVH format, namely the BioVision Hierarchy format, is a file format for storing human motion capture data. Files in this format use nodes as core elements to record the motion information of the human skeleton within several consecutive frames. BVH files usually contain two main parts: a hierarchical structure (Hierarchy) and motion information (Motion). The default state of the BVH format is that the skeleton presents Tpose, and the initial coordinate system of each bone of the body is the same, which is consistent with the root bone. The coordinate axes of bone rotation are as follows: Figure 4 shown.

[0205] The steps to calculate the rotation of the target bone relative to the parent bone are as follows:

[0206] 1. Starting from the root bone, calculate the cosine matrix of the bones relative to the initial state in order: In bone rotation, each bone will have an axis that always points to its child bones, and the two axes will always be in the same plane. Therefore, through the 3D coordinates of the whole body bones, the direction vector i, j, k of each axis in the initial coordinate system can be obtained, so that the cosine matrix of the coordinate axis of the target bone relative to the initial coordinate axis can be calculated.

[0207]

[0208] 2. Convert the direction cosine matrix to quaternion: In order to avoid deadlock in rotation, convert the direction cosine matrix to quaternion for subsequent calculations to obtain the rotation quaternion quat of the bone relative to the initial coordinate axis. target .

[0209] quat target =(w,x,y,z)#(5)

[0210] 3. Get the rotation relative to the parent bone: According to the above steps, the rotation quaternion of the target bone relative to the initial coordinate axis and the rotation quaternion of the parent bone relative to the initial coordinate axis are obtained. According to the characteristics of the quaternion, the rotation quaternion relative to the parent bone is:

[0211] Local quat =quat target -quat father #(6)

[0212] 4. Obtain BVH format rotation based on quaternion: According to the rotation order of the set BVH format, Local quat Convert to Euler angles to represent rotation. The rotation order represents the order in which the Euler angles rotate a certain axis. For example, xyz means that the rotation represented by the Euler angles is first around the x-axis, then the y-axis, and finally the z-axis.

[0213] At this point, according to the above steps, the rotation of the bone corresponding to each 3D key point relative to its parent bone is obtained for subsequent driving of the 3D digital human.

[0214] 4) Low-latency smooth data drives 3D digital humans:

[0215] After the driving data is obtained, it will be transmitted to the rendering platform and drive the 3D digital human. However, direct data driving will produce large jitter. Therefore, in this method, after obtaining the human body's bone rotation and facial blendshape parameters, the data needs to be smoothed first, and then transmitted to the rendering platform through socket communication.

[0216] Smoothing uses sliding weighted average filtering, which can smooth data with low latency. This method retains the most recent N frames of data, and then when setting weights, the closer the data is to the latest frame, the higher the weight is, and finally the smoothed data is obtained. The formula is as follows:

[0217]

[0218] w i =decadeRate N-i ,(0 <decadeRate<1)

[0219]

[0220] The facial blendshape parameters are obtained in the rendering platform, and facial expressions are controlled based on each parameter corresponding to the morph target of the model. Complex expressions are generated by superimposing the weights of each expression parameter.

[0221] For the acquired bone rotation, quaternion is needed as an intermediate variable to avoid deadlock and convert the bone rotation to the rotation order corresponding to the rendering platform. For example, the bone rotation in the xyz rotation order is first converted to quaternion, and then converted to the bone rotation in the yxz rotation order.

[0222] Finally, according to the rotation of the bones, starting from the root bone, the rotation of the digital human bones is controlled in sequence according to the hierarchy to achieve the purpose of controlling the movement of the digital human.

[0223] Those skilled in the art know that, in addition to realizing the system and its various devices, modules, and units provided by the present invention in a purely computer-readable program code, it is entirely possible to realize the same functions in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered as a hardware component, and the devices, modules, and units included therein for realizing various functions can also be regarded as structures within the hardware component; the devices, modules, and units for realizing various functions can also be regarded as both software modules for realizing the method and structures within the hardware component.

[0224] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A universal 3D digital human real-time motion capture method based on a monocular camera, characterized in that: include: Step S1: Detect 3D key points based on a monocular camera, align the 3D key points to a unified coordinate system, detect facial 3D landmark points from the video, and output facial blendshape parameters for driving the digital human face; Step S2: convert the 3D key points into general driving data, and convert the 3D key point coordinates of the whole body skeleton into the corresponding bone BVH rotation; Step S3: Transmit the facial blendshape parameters and the bone BVH rotation to the rendering platform; convert the bone rotation coordinate system to the rendering platform coordinate system to drive the 3D digital human movement; use the facial blendshape parameters to control the digital human Morph Target and drive the digital human facial expression.

2. The method for real-time motion capture of a universal 3D digital human based on a monocular camera according to claim 1, characterized in that: In step S1: Detect the 3D key points of the human skeleton from the monocular camera video image frame or the video image frame, including the 3D key points of the torso and the 3D key points of the hands, and align the 3D key points to a unified coordinate system as the data basis for subsequent driving of the digital human body movement; The mediapipe face landmark algorithm is used to detect facial expressions. The face is recognized from the input image and cropped out. The coordinates of the facial landmark points are detected and input into the algorithm. The facial blendshape parameters corresponding to the landmark points are output. Blendshape is a morphological target animation that smoothly transitions between different facial expressions or shapes through linear interpolation. The facial expressions of digital humans are controlled through blendshape parameters. The human body ROI area and the hand ROI area are cropped out, and the mediapipe pose landmark detection algorithm and mediapipe hand landmark are used to obtain the 3D coordinates and pixel coordinates of the body key points and hand key points; the body 3D key points take the center of the two hip joints as the origin, and the hand 3D key points coordinate system takes the wrist of each hand as the origin; After obtaining the 3D coordinates of the body and hand relative to their respective coordinate systems, align the coordinate systems to the same coordinate system; the body coordinates and hand coordinates have the same wrist key point, and with the help of the coordinates of the wrist key point in two different coordinate systems and their respective pixel coordinates, align the coordinates of the finger key points in the hand 3D key points to the body coordinate system; The pixel coordinates of the finger key point are finger pixel , the 3D coordinate with the wrist as the origin is finger real ; The pixel coordinate of the midpoint of the hip joint is middle pixel ; The pixel coordinates of the wrist key point are wrist pixel , the 3D coordinate with the wrist as the origin is wrist real , the 3D coordinates of the specific finger key points in the coordinate system with the midpoint of the hip joint as the origin are: finger z =wrist real .z*+finger real .z(3) Among them, finger x is the x-coordinate value of the finger key point in the coordinate system with the midpoint of the hip joint as the origin, finger y is the y coordinate value in this coordinate system, finger z is the z coordinate value in this coordinate system, wrist real .x and other.x refers to wrist real The x coordinate value in 3D coordinates, and the other .yz is the corresponding yz coordinate value; The coordinates of the key points of the fingers are aligned to the coordinate system with the midpoint of the hip joint as the origin, which is used for the subsequent overall driving of the digital human.

3. The method for real-time motion capture of a universal 3D digital human based on a monocular camera according to claim 1, characterized in that: In step S2: Based on the parent-child relationship of bones, the coordinates of the 3D key points of the whole body are converted into the BVH rotation of the corresponding bones, in preparation for the subsequent driving of the 3D digital human; The movement of the digital human skeleton starts from the root bone. Other bones continue to move in layers based on the parent bone until the end bone. The movement of each bone will affect the state of its child bones. Before the child bone moves, its coordinate axis is the same as that of the parent bone. Each movement of the child bone is to continue to rotate based on the rotation of the parent bone and affect the bones of the subsequent layers. After obtaining the 3D coordinates of the key points of the whole body, the 3D key point coordinates are converted into the rotation of each 3D key point relative to the parent bone using the BVH format as the standard; the BVH format is used to store human motion capture data, with nodes as the core elements, recording the motion information of the human skeleton in consecutive frames. The BVH file contains two main parts: the hierarchical structure and motion information. The default state of the BVH format is that the bones present Tpose, and the initial coordinate system of each bone of the body is the same, consistent with the root bone.

4. The method for real-time motion capture of a universal 3D digital human based on a monocular camera according to claim 3, characterized in that: The steps to calculate the rotation of the target bone relative to the parent bone are as follows: Starting from the root bone, calculate the cosine matrix of the bones relative to the initial state in order according to the hierarchy: In bone rotation, each bone will have an axis that always points to its child bones, and the two axes will always be in the same plane. Through the 3D coordinates of the whole body bones, get the direction vector i, j, k of each axis in the initial coordinate system, and calculate the cosine matrix R of the coordinate axis of the target bone relative to the initial coordinate axis: Among them, i x is the x component of the direction vector i in the initial coordinate system, j x is the x component of the direction vector j in the initial coordinate system, k x is the x component of the direction vector k in the initial coordinate system; i y is the y component of the direction vector i in the initial coordinate system, j y is the y component of the direction vector j in the initial coordinate system, k y is the y component of the direction vector k in the initial coordinate system; i z is the z component of the direction vector i in the initial coordinate system, j z is the z component of the direction vector j in the initial coordinate system, k z is the z component of the direction vector k in the initial coordinate system; Convert the direction cosine matrix to quaternion: Convert the direction cosine matrix to quaternion for subsequent calculations to obtain the rotation quaternion quat of the bone relative to the initial coordinate axis. target : quat target =(w,x,y,z)(5) Among them, w represents the degree of rotation, and its value is related to the rotation angle; xyz is a three-dimensional vector, which is a direction unit vector and defines the direction of the rotation axis; Get the rotation relative to the parent bone: Get the rotation quaternion of the target bone relative to the initial coordinate axis, and the rotation quaternion of the parent bone relative to the initial coordinate axis. According to the characteristics of the quaternion, calculate the rotation quaternion Local relative to the parent bone. quat : Local quat =four target -four father (6) Among them, quat father The rotation quaternion of the parent bone, quat target The rotation quaternion of the current bone; Get BVH format rotation according to quaternion: According to the rotation order of the set BVH format, Local quat Convert to Euler angle to represent rotation; the rotation order represents the order of Euler angle rotation of a certain axis; get the rotation of the bone corresponding to each 3D key point relative to its parent bone for subsequent driving of the 3D digital human.

5. The method for real-time motion capture of a universal 3D digital human based on a monocular camera according to claim 1, characterized in that: In step S3: Transmit facial blendshape parameters and bone BVH rotation to the rendering platform through the socket; convert the bone rotation coordinate system to the rendering platform coordinate system to drive the 3D digital human action; After the driving data is obtained, it is transmitted to the rendering platform and drives the 3D digital human. After the human body's bone rotation and facial blendshape parameters are obtained, the data is smoothed and transmitted to the rendering platform through socket communication. The smoothing process uses a sliding weighted average filter to retain the most recent N frames of data. When setting the weight, the closer the data is to the latest frame, the higher the weight is. The smoothed data is obtained using the following formula: w i =decadeRate N-i ,(0<decadeRate<1) Where y(t) is the latest data value after smoothing, N is the number of the most recently preset multi-frame data recorded, n is the sequence number of a certain data from the beginning to the end of the recorded data, decadeRate is the rate of change of the weight, and w is the weight value corresponding to each data calculated by decadeRate. i w j Represents the weight corresponding to the ijth value, and the subscript represents the corresponding serial number. is the weight value that needs to be multiplied by the value corresponding to the normalized sequence number n, and x(n) is the nth value recorded; Obtain facial blendshape parameters in the rendering platform, and control facial expressions based on each parameter corresponding to the morphtarget of the model; generate expressions based on the weighted superposition of each expression parameter; For the acquired bone rotation, quaternion is used as an intermediate variable to convert the bone rotation to the rotation order corresponding to the rendering platform. According to the bone rotation, starting from the root bone, the rotation of the digital human skeleton is controlled in sequence according to the hierarchical order to achieve the purpose of controlling the movement of the digital human.

6. A universal 3D digital human real-time motion capture system based on a monocular camera, characterized in that: include: Module M1: detects 3D key points based on a monocular camera, aligns the 3D key points to a unified coordinate system, detects facial 3D landmark points from the video, and outputs facial blendshape parameters to drive the digital human face; Module M2: Convert 3D key points into general driving data, convert the 3D key point coordinates of the whole body skeleton into the corresponding bone BVH rotation; Module M3: Transmit facial blendshape parameters and bone BVH rotation to the rendering platform; convert the bone rotation coordinate system to the rendering platform coordinate system to drive the 3D digital human movement; use the facial blendshape parameters to control the digital human Morph Target and drive the digital human facial expression.

7. The universal 3D digital human real-time motion capture system based on a monocular camera according to claim 6, characterized in that: In the module M1: Detect the 3D key points of the human skeleton from the monocular camera video image frame or the video image frame, including the 3D key points of the torso and the 3D key points of the hands, and align the 3D key points to a unified coordinate system as the data basis for subsequent driving of the digital human body movement; The mediapipe face landmark algorithm is used to detect facial expressions. The face is recognized from the input image and cropped out. The coordinates of the facial landmark points are detected and input into the algorithm. The facial blendshape parameters corresponding to the landmark points are output. Blendshape is a morphological target animation that smoothly transitions between different facial expressions or shapes through linear interpolation. The facial expressions of digital humans are controlled through blendshape parameters. The human body ROI area and the hand ROI area are cropped out, and the mediapipe pose landmark detection algorithm and mediapipe hand landmark are used to obtain the 3D coordinates and pixel coordinates of the body key points and hand key points; the body 3D key points take the center of the two hip joints as the origin, and the hand 3D key points coordinate system takes the wrist of each hand as the origin; After obtaining the 3D coordinates of the body and hand relative to their respective coordinate systems, align the coordinate systems to the same coordinate system; the body coordinates and hand coordinates have the same wrist key point, and with the help of the coordinates of the wrist key point in two different coordinate systems and their respective pixel coordinates, align the coordinates of the finger key points in the hand 3D key points to the body coordinate system; The pixel coordinates of the finger key point are finger pixel , the 3D coordinate with the wrist as the origin is finger real ; The pixel coordinate of the midpoint of the hip joint is middle pixel ; The pixel coordinates of the wrist key point are wrist pixel , the 3D coordinate with the wrist as the origin is wrist real , the 3D coordinates of the specific finger key points in the coordinate system with the midpoint of the hip joint as the origin are: finger z =wrist real .z*+finger real .z(3) Among them, finger x is the x-coordinate value of the finger key point in the coordinate system with the midpoint of the hip joint as the origin, finger y is the y coordinate value in this coordinate system, finger z is the z coordinate value in this coordinate system, wrist real .x and other.x refers to wrist real The x coordinate value in 3D coordinates, and the other .yz is the corresponding yz coordinate value; The coordinates of the key points of the fingers are aligned to the coordinate system with the midpoint of the hip joint as the origin, which is used for the subsequent overall driving of the digital human.

8. The universal 3D digital human real-time motion capture system based on a monocular camera according to claim 6, characterized in that: In the module M2: Based on the parent-child relationship of bones, the coordinates of the 3D key points of the whole body are converted into the BVH rotation of the corresponding bones, in preparation for the subsequent driving of the 3D digital human; The movement of the digital human skeleton starts from the root bone. Other bones continue to move in layers based on the parent bone until the end bone. The movement of each bone will affect the state of its child bones. Before the child bone moves, its coordinate axis is the same as that of the parent bone. Each movement of the child bone is to continue to rotate based on the rotation of the parent bone and affect the bones of the subsequent layers. After obtaining the 3D coordinates of the key points of the whole body, the 3D key point coordinates are converted into the rotation of each 3D key point relative to the parent bone using the BVH format as the standard; the BVH format is used to store human motion capture data, with nodes as the core elements, recording the motion information of the human skeleton in consecutive frames. The BVH file contains two main parts: the hierarchical structure and motion information. The default state of the BVH format is that the bones present Tpose, and the initial coordinate system of each bone of the body is the same, consistent with the root bone.

9. The universal 3D digital human real-time motion capture system based on a monocular camera according to claim 8, characterized in that: The steps to calculate the rotation of the target bone relative to the parent bone are as follows: Starting from the root bone, calculate the cosine matrix of the bones relative to the initial state in order according to the hierarchy: In bone rotation, each bone will have an axis that always points to its child bones, and the two axes will always be in the same plane. Through the 3D coordinates of the whole body bones, get the direction vector i, j, k of each axis in the initial coordinate system, and calculate the cosine matrix R of the coordinate axis of the target bone relative to the initial coordinate axis: Among them, i x is the x component of the direction vector i in the initial coordinate system, j x is the x component of the direction vector j in the initial coordinate system, k x is the x component of the direction vector k in the initial coordinate system; i y is the y component of the direction vector i in the initial coordinate system, j y is the y component of the direction vector j in the initial coordinate system, k y is the y component of the direction vector k in the initial coordinate system; i z is the z component of the direction vector i in the initial coordinate system, j z is the z component of the direction vector j in the initial coordinate system, k z is the z component of the direction vector k in the initial coordinate system; Convert the direction cosine matrix to quaternion: Convert the direction cosine matrix to quaternion for subsequent calculations to obtain the rotation quaternion quat of the bone relative to the initial coordinate axis. target : quat target =(w,x,y,z)(5) Among them, w represents the degree of rotation, and its value is related to the rotation angle; xyz is a three-dimensional vector, which is a direction unit vector and defines the direction of the rotation axis; Get the rotation relative to the parent bone: Get the rotation quaternion of the target bone relative to the initial coordinate axis, and the rotation quaternion of the parent bone relative to the initial coordinate axis. According to the characteristics of the quaternion, calculate the rotation quaternion Local relative to the parent bone. quat : Local quat =four target -four father (6) Among them, quat father The rotation quaternion of the parent bone, quat target The rotation quaternion of the current bone; Get BVH format rotation according to quaternion: According to the rotation order of the set BVH format, Local quat Convert to Euler angle to represent rotation; the rotation order represents the order of Euler angle rotation of a certain axis; get the rotation of the bone corresponding to each 3D key point relative to its parent bone for subsequent driving of the 3D digital human.

10. The universal 3D digital human real-time motion capture system based on a monocular camera according to claim 6, characterized in that: In the module M3: Transmit facial blendshape parameters and bone BVH rotation to the rendering platform through the socket; convert the bone rotation coordinate system to the rendering platform coordinate system to drive the 3D digital human action; After the driving data is obtained, it is transmitted to the rendering platform and drives the 3D digital human. After the human body's bone rotation and facial blendshape parameters are obtained, the data is smoothed and transmitted to the rendering platform through socket communication. The smoothing process uses a sliding weighted average filter to retain the most recent N frames of data. When setting the weight, the closer the data is to the latest frame, the higher the weight is. The smoothed data is obtained using the following formula: w i =decadeRate N-i ,(0<decadeRate<1) Where y(t) is the latest data value after smoothing, N is the number of the most recently preset multi-frame data recorded, n is the sequence number of a certain data from the beginning to the end of the recorded data, decadeRate is the rate of change of the weight, and w is the weight value corresponding to each data calculated by decadeRate. i w j Represents the weight corresponding to the ijth value, and the subscript represents the corresponding serial number. is the weight value that needs to be multiplied by the value corresponding to the normalized sequence number n, and x(n) is the nth value recorded; Obtain facial blendshape parameters in the rendering platform, and control facial expressions based on each parameter corresponding to the morphtarget of the model; generate expressions based on the weighted superposition of each expression parameter; For the acquired bone rotation, quaternion is used as an intermediate variable to convert the bone rotation to the rotation order corresponding to the rendering platform. According to the bone rotation, starting from the root bone, the rotation of the digital human skeleton is controlled in sequence according to the hierarchical order to achieve the purpose of controlling the movement of the digital human.

Images