Training methods, devices, electronic equipment, and storage media for keypoint rotation models

Abstract

This application discloses a training method, apparatus, electronic device, and storage medium for a keypoint rotation model, relating to the fields of computer and image processing technology. It enables the generation of a keypoint rotation model capable of determining rotation values ​​using the three-dimensional coordinates of keypoints. The specific scheme includes: acquiring at least one set of sample data; the sample data includes sample coordinates and sample rotation values ​​of keypoints of a sample person; wherein the sample coordinates are the three-dimensional coordinates of the keypoints of the sample person in real three-dimensional space; processing the sample coordinates of the keypoints of the sample person based on an initial keypoint rotation model to obtain predicted rotation values; iteratively updating the initial keypoint rotation model based on the predicted rotation values ​​and sample rotation values ​​to obtain a keypoint rotation model; the keypoint rotation model has the function of obtaining the rotation value of a person based on the three-dimensional coordinates of the keypoints of that person.

Description

Technical Field

[0001] The present application relates to the field of computer and image processing, and in particular to a training method and device, electronic device and storage medium for a key point rotation model. Background Art

[0002] Digital humans have become an indispensable tool in the fields of games, animation, and film. How to drive digital humans efficiently and accurately is a technical issue that is widely concerned.

[0003] In related technologies, motion capture equipment can be used to obtain all the motion information of real people, and directly drive the digital human in the virtual space to move according to the motion information of real people. However, the cost and complexity of motion capture equipment are extremely high.

[0004] In other related technologies, real-person video data is collected through a monocular camera, and the positions of key points of the human body are advanced in the video data to obtain the motion information of each key point, so as to realize the driving of the digital human through the motion information of each key point. However, this method only uses a monocular camera, and the video data obtained has a single perspective. In actual application, the key points located in the blind spot of the camera cannot be captured, resulting in the inconsistency of the motion information of the key points located in the blind spot of the camera with other key points during the driving process of the digital human, which is visually manifested as the shaking of the digital human, the dislocation of joint rotation, and the loss of some positions. Summary of the invention

[0005] The embodiment of the present application provides a training method for a key point rotation model, which can solve the problems of obvious jitter, joint rotation dislocation and easy loss of digital humans during the driving process.

[0006] In order to achieve the above purpose, the embodiment of the present application adopts the following technical solution:

[0007] In a first aspect, a digital human driving method is provided, the method comprising: collecting video data of multiple perspectives in a real three-dimensional space through multiple video acquisition devices; wherein the video data includes a target person; determining a first coordinate of a key point of the target person; wherein the first coordinate is a two-dimensional coordinate of the key point in a video frame in the video data; determining a mapping relationship according to the first coordinate; wherein the mapping relationship is a corresponding relationship between the key point and a virtual key point in a virtual three-dimensional space; calculating a second coordinate according to the mapping relationship and the first coordinate; wherein the second coordinate is a three-dimensional coordinate of the virtual key point in the virtual three-dimensional space; processing the second coordinate based on a key point rotation model to obtain a rotation value of the virtual key point in the virtual three-dimensional space; and driving the digital human to move based on the rotation value of the virtual key point in the virtual three-dimensional space.

[0008] In combination with the first aspect, in another possible implementation, after collecting video data of multiple perspectives in a real three-dimensional space through multiple video acquisition devices, the method also includes: detecting a target person in the video data; when the target person is detected, determining key points of the target person; when the target person cannot be detected, the digital human maintains a standby posture.

[0009] In combination with the first aspect, in another possible implementation, determining a mapping relationship based on the first coordinates includes: determining a first parameter matrix based on the first coordinates of the key point; wherein the first parameter matrix is ​​used to indicate a transformation relationship of a position point in a real three-dimensional space from a first coordinate system to a second coordinate system, the first coordinate system being a three-dimensional coordinate system in the real three-dimensional space with the center of the target person as the origin, and the second coordinate system being a three-dimensional coordinate system in the real three-dimensional space with the optical center of a video acquisition device as the origin; obtaining a second parameter matrix; wherein the second parameter matrix is ​​used to indicate a transformation relationship of a position point in the real three-dimensional space from the second coordinate system to a third coordinate system, the three-coordinate system being a coordinate system with the image center of a video frame of the video data as the origin; and taking a product matrix of the first parameter matrix and the second parameter matrix as the mapping relationship.

[0010] In combination with the first aspect, in another possible implementation method, the first parameter matrix is ​​calculated according to the first coordinates of the key point, including: determining the initial first parameter matrix according to the first coordinates of the key point; obtaining the third coordinates of the key point; wherein the third coordinates are the three-dimensional coordinates of the key point in the real three-dimensional space; calculating the first coordinate prediction value according to the third coordinates and the initial first parameter matrix; determining the first loss value based on the first coordinate prediction value and the first coordinate; iteratively updating the initial first parameter matrix based on the first loss value until the first loss value is less than a first preset threshold, and determining the latest updated initial first parameter matrix as the first parameter matrix.

[0011] In combination with the first aspect, in another possible implementation, before processing the second coordinate based on the key point rotation model to obtain the rotation value of the virtual key point in the virtual three-dimensional space, the method includes: obtaining at least one set of sample data; the sample data includes sample coordinates and sample rotation values ​​of the key points of the sample character in the sample video; the sample coordinates are the two-dimensional coordinates of the key points of the sample character in the video frame of the sample video; processing the sample coordinates of the key points of the sample character based on the initial key point rotation model to obtain a predicted rotation value; determining a second loss value based on the predicted rotation value and the sample rotation value; iteratively updating the initial key point rotation model based on the second loss value until the second loss value is less than a second preset threshold, and determining the latest updated initial key point rotation model as the key point rotation model.

[0012] In combination with the first aspect, in another possible implementation manner, the data format of the sample coordinates and sample rotation values ​​of the key points of the sample person is a quaternion format.

[0013] In combination with the first aspect, in another possible implementation, after processing the second coordinate based on the key point rotation model to obtain the rotation value of the key point in the virtual three-dimensional space, and before driving the digital human to move based on the rotation value of the virtual key point in the virtual three-dimensional space, the method also includes: smoothing the rotation value of the virtual key point in the virtual three-dimensional space.

[0014] In a second aspect, a digital human driving device is provided, which has the function of implementing the method of the first aspect. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions.

[0015] In a third aspect, a digital human driving device is provided, the digital human driving device comprising a video acquisition module, which is used to acquire video data of multiple perspectives in a real three-dimensional space through multiple video acquisition devices; wherein the video data includes a target person; a first coordinate determination module, which is used to determine the first coordinate of a key point of the target person in the video data acquired by the video acquisition module; wherein the first coordinate is a two-dimensional coordinate of the key point in a video frame in the video data; a mapping relationship determination module, which is used to determine a mapping relationship according to the first coordinate determined by the first coordinate determination module; wherein the mapping relationship is a corresponding relationship between the key point and the virtual key point in the virtual three-dimensional space; a second coordinate calculation module, which is used to calculate according to the mapping relationship determined by the mapping relationship determination module and the first coordinate determined by the first coordinate module; wherein the second coordinate is the three-dimensional coordinate of the virtual key point in the virtual three-dimensional space; a second coordinate processing module, which is used to process the second coordinate calculated by the second coordinate calculation module based on the key point rotation model to obtain a rotation value of the virtual key point in the virtual three-dimensional space; and a digital human driving module, which is used to drive the digital human to move based on the rotation value in the virtual three-dimensional space obtained by the second coordinate processing module.

[0016] In a fourth aspect, a digital human driving device is provided, comprising: a processor and a memory; the memory is used to store computer execution instructions, and when the first device is running, the processor executes the computer execution instructions stored in the memory to enable the first device to perform a digital human driving method provided in any one of the above-mentioned first aspects.

[0017] In a fifth aspect, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium, and when the computer-readable storage medium is run on a computer, the computer can execute the digital human driving method provided in any one of the first aspects above.

[0018] In a sixth aspect, a computer program product containing instructions is provided, which, when executed on a digital human driving device, enables the digital human driving device to execute any digital human driving method provided in the first aspect.

[0019] In a seventh aspect, a device (for example, the device may be a chip system) is provided, the device including a processor for supporting a digital human driving device to implement the functions involved in the first aspect. In a possible design, the device also includes a memory for storing program instructions and data necessary for the digital human driving device. When the device is a chip system, it may be composed of a chip, or may include a chip and other discrete devices.

[0020] Based on the technical solution provided by the embodiment of the present application, firstly, video data of multiple perspectives of the target person are collected in the real three-dimensional space through multiple video acquisition devices, so as to solve the problem that when the video data perspective is single, some key points of the target person cannot be detected, resulting in obvious jitter, joint dislocation and easy loss of the digital human during the driving process; secondly, the first coordinates of the key points of the target person are determined, and the mapping relationship is determined according to the first coordinates. The mapping relationship is the correspondence between the key points and the virtual key points in the virtual three-dimensional space, that is, the process of determining the mapping relationship is the calibration position process of the video acquisition device. It can be seen that the technical solution completes the calibration through the first coordinates of the key points of the target person in the multi-perspective video data without a calibration plate, reducing the laying cost of the calibration plate; finally, the second coordinate is calculated according to the mapping relationship and the first coordinate, and the second coordinate is processed based on the key point rotation model to obtain the rotation value of the virtual key point in the virtual three-dimensional space. The digital human is driven to move based on the rotation value of the virtual key point in the virtual three-dimensional space, and the rotation values ​​of all key points are obtained at the same time by processing the second coordinate by the key point rotation model, thereby improving the driving efficiency of the digital human.

[0021] In summary, on the one hand, the technical solution provided by the present application can solve the problem that when the video data has a single perspective, some key points of the target person cannot be detected, resulting in obvious jitter, joint dislocation and easy loss of the digital human during the driving process; on the other hand, the technical solution provided by the present application can complete the calibration through the first coordinates of the key points of the target person in the multi-perspective video data without a calibration plate, thereby reducing the cost of laying the calibration plate; on the other hand, the technical solution provided by the present application can improve the efficiency of digital human driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 Schematic diagram of the positional relationship among the world coordinate system, camera coordinate system and image coordinate system;

[0023] Figure 2A schematic diagram of the software architecture of a digital human driving device provided in an embodiment of the present application;

[0024] Figure 3 A schematic diagram of a digital human driving method provided in an embodiment of the present application Figure 1 ;

[0025] Figure 4 A schematic diagram of a digital human driving method provided in an embodiment of the present application Figure 2 ;

[0026] Figure 5 A schematic diagram of a digital human driving method provided in an embodiment of the present application Figure 3 ;

[0027] Figure 6 A schematic diagram of a digital human driving method provided in an embodiment of the present application Figure 4 ;

[0028] Figure 7 A schematic diagram of a digital human driving method provided in an embodiment of the present application Figure 5 ;

[0029] Figure 8 A schematic diagram of a digital human driving method provided in an embodiment of the present application Figure 6 ;

[0030] Fig. 9 A schematic diagram of a digital human driving method provided in an embodiment of the present application Figure 7 ;

[0031] Fig.10 A schematic diagram of another example of a digital human driving method provided in an embodiment of the present application;

[0032] Fig.11 A schematic diagram of a passive calibration process of another digital human driving method provided in an embodiment of the present application;

[0033] Fig.12 A schematic diagram of the IKNet training process of another digital human driving method provided in an embodiment of the present application;

[0034] Fig.13 A schematic diagram of the structure of another digital human driving device provided in an embodiment of the present application;

[0035] Fig.14 A schematic diagram of the structure of another digital human driving device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0036] In order to make the purpose and implementation method of the present application clearer, the exemplary implementation method of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments.

[0037] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.

[0038] The terms "first", "second", "third", etc. in the specification and claims of this application and the above drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise noted. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances.

[0039] The terms "including" and "having" and any variations thereof in this application are intended to cover but not exclude inclusion. For example, a product or device comprising a series of components is not necessarily limited to all the components explicitly listed, but may include other components not explicitly listed or inherent to these products or devices.

[0040] The term "and / or" in this application is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this application generally indicates that the associated objects before and after are in an "or" relationship.

[0041] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the above-mentioned module, program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0042] Based on the embodiments described in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the claims attached to this application. In addition, although the disclosure in this application is introduced according to one or several exemplary examples, it should be understood that each aspect of these disclosures can also constitute a complete implementation method separately. It should be noted that the brief description of the terms in this application is only for the convenience of understanding the implementation methods described below, and is not intended to limit the implementation methods of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.

[0043] First, the terms involved in this application are explained as follows:

[0044] Key points: refers to specific points on the human body, usually expressed as coordinates on a two-dimensional image or coordinates in a three-dimensional space. It can be divided into face key points such as eyes, nose, mouth and other specific parts of facial features; body posture key points such as neck, shoulders, elbows, wrists, waist, knees and ankles; hand key points such as fingers, palms and wrists. Key points in the following embodiments generally refer to body posture key points unless otherwise specified.

[0045] Multi-camera: A camera with more than one lens.

[0046] Camera calibration: Camera calibration refers to the process of determining the camera's intrinsic and extrinsic parameters. Through camera calibration, the image coordinates can be mapped to the three-dimensional world coordinate system.

[0047] World coordinate system: A three-dimensional rectangular coordinate system that can be used as a reference to describe the relative spatial position of the camera and the object to be measured. Figure 1 As shown, the world coordinate system O W -X W Y W Z W Usually the center of the object to be measured is taken as the origin, and the world coordinate system can be freely determined according to the actual situation. Once the world coordinate system is determined, its position and angle cannot be changed in the subsequent process.

[0048] Camera coordinate system: A three-dimensional rectangular coordinate system that can be used as a reference to describe the relative spatial position of the camera and the object to be measured. Figure 1 As shown, the camera coordinate system O C -X C Y C Z C Usually the optical center of the camera lens is taken as the origin, and the position and angle of the camera coordinate system change with the position and posture of the camera lens.

[0049] Image coordinate system: A two-dimensional rectangular coordinate system, refer to Figure 1As shown, the image coordinate system O-XY usually takes the intersection of the optical axis and the image plane as its origin, that is, the center point of the image as its origin, and the X-axis and the Y-axis are parallel to the two sides of the image plane respectively.

[0050] IK: Inverse Kinematics, refers to a physical movement method that uses the displacement and movement direction of the parent joint to inherit the information to its child joints. That is, by positioning the lower bones in the skeleton chain and rotating the higher bones, the joint posture is set, and the rotation of each parent joint is calculated according to the position movement of the terminal child joint.

[0051] FK: Forward Kinematics, refers to a physical movement method in which the parent drives the child to achieve movement, that is, the position of each child joint is calculated based on the rotation of the parent joint.

[0052] IKNet: Inverse Kinematics, a network that inputs child joint positions and outputs parent joint rotations.

[0053] Savgol algorithm: Savitzky-Golay algorithm is a commonly used algorithm for smoothing data in digital signal processing. Specifically, the Savgol algorithm can perform polynomial fitting on the original data and smooth the fitting curve, thereby eliminating noise and oscillation in the original data.

[0054] HRNet: High-Resolution Net, a neural network model based on deep learning, can be used to process image tasks. Specifically, HRNet uses information at multiple resolutions to extract features, and generates more expressive feature representations by combining this information. A large number of experiments have shown that HRNet has achieved state-of-the-art results in tasks such as key point detection, posture estimation, and human segmentation. At the same time, HRNet can also be applied to other image processing tasks that require high-resolution spatial feature details.

[0055] PnP algorithm: Perspective-n-Point algorithm, is a posture estimation algorithm in computer vision. This algorithm can solve the camera's posture (position and direction) and the spatial relationship between the target object and the camera by knowing the coordinate points and corresponding pixel points of the target object in three-dimensional space.

[0056] LM algorithm: Levenberg-Marquardt algorithm is an optimization algorithm used for nonlinear least squares problems. It combines the advantages of Gauss-Newton method and gradient descent method, and can effectively solve nonlinear parameter estimation problems in numerical calculations.

[0057] Camera extrinsics: Camera extrinsics refer to parameters that describe the position and orientation of a camera in three-dimensional space, usually including the camera's rotation matrix and translation vector.

[0058] Camera intrinsics: Camera intrinsics refer to the internal parameters of the camera, which are used to describe the basic configuration and characteristics of the camera optical system. Camera intrinsics are usually stored in the camera's metadata and can be accessed and used by applications or computer vision algorithms.

[0059] Bundle adjust optimization method: that is, bundle adjustment optimization method. This method uses the least squares method to adjust the camera pose and 3D point cloud coordinates so that the error between the points reprojected onto the image plane and the corresponding points in the original image is minimized, thereby obtaining more accurate camera pose and 3D point cloud coordinates.

[0060] Euclidean distance: Euclidean distance, also known as Euclidean distance, is the distance between two points in n-dimensional space. Generally, in n-dimensional space, the Euclidean distance d between point A (x1, x2, ... xn) and point B (y1, y2, ... yn) can be calculated by the following formula:

[0061]

[0062] Among them, xi is the coordinate value of point A in the i-th dimension in n-dimensional space, yi is the coordinate value of point B in the i-th dimension in n-dimensional space, and d is the Euclidean distance between points A and B.

[0063] Adam optimization algorithm: Adaptive Moment Estimation algorithm, is a commonly used adaptive learning rate optimization method for efficiently training deep neural networks. It is an optimization method that adjusts the learning rate of each parameter by calculating the first-order moment (mean) and second-order moment (unbiased estimate of variance) of the gradient and combining the adaptive learning rate method of the previous parameter update amount.

[0064] As people's demand for digital experience grows, the trend of using digital characters (such as virtual assistants) is also accelerating. In this digital age, digital humans have always been a compelling topic. Digital humans can be virtual characters, humanoid robots, or any digital entity with voice, vision, touch, and action capabilities. They are widely used in many fields such as healthcare, financial services, industrial automation, customer service, etc.

[0065] In related technologies, driving the synchronous movement of digital humans and real people is achieved by collecting the movement information of the key points of the real human body. Specifically, the posture data of the real human body is first collected through a monocular camera, and then the key point detection is performed to determine the position of the key point, and then the whole body movement information is obtained through the key point, and finally the digital human is driven in real time according to the whole body movement information.

[0066] However, the above solution only uses a monocular camera, and the video data obtained has a single perspective. In actual application, the key points in the camera blind spot cannot be captured, resulting in the inconsistency between the action information of the key points in the camera blind spot and other key points during the digital human driving process, which is visually manifested as the digital human shaking, joint rotation dislocation, and partial position loss. At the same time, due to the single perspective of the monocular camera, the digital human driving device will still regard the back of the real person as the front when the real person in the three-dimensional space is facing away from the camera, resulting in the inconsistency between the movement direction of the digital human and the real person. Therefore, the real person in the real three-dimensional space must always face the lens of the monocular camera, resulting in limited range of movement and movement.

[0067] If multi-cameras are used, the complex calibration problem of the multi-cameras will greatly increase the labor cost. At the same time, the calibration of the multi-cameras requires laying out calibration plates, which brings unnecessary economic costs.

[0068] In other related technologies, real people in real three-dimensional space can be directly equipped with motion capture equipment, and the digital human driving device can directly drive the digital human to move according to the motion information obtained in real time by the motion capture equipment. However, the cost and complexity of motion capture equipment are extremely high.

[0069] In view of the above problems, the embodiment of the present application proposes a digital human driving method. First, multiple video acquisition devices are used to collect video data of multiple perspectives of a target person in a real three-dimensional space to solve the problem that when the video data has a single perspective, some key points of the target person cannot be detected, resulting in obvious jitter, joint dislocation and easy loss of the digital person during the driving process; secondly, the first coordinates of the key points of the target person are determined, and a mapping relationship is determined according to the first coordinates. The mapping relationship is the correspondence between the key points and the virtual key points in the virtual three-dimensional space, that is, the process of determining the mapping relationship is the calibration position process of the video acquisition device. It can be seen that the technical solution completes the calibration through the first coordinates of the key points of the target person in the multi-perspective video data without a calibration plate, reducing the laying cost of the calibration plate; finally, the second coordinate is calculated according to the mapping relationship and the first coordinate, and the second coordinate is processed based on the key point rotation model to obtain the rotation value of the virtual key point in the virtual three-dimensional space. The digital human is driven to move based on the rotation value of the virtual key point in the virtual three-dimensional space, and the rotation values ​​of all key points are obtained at the same time by processing the second coordinate by the key point rotation model, thereby improving the driving efficiency of the digital human.

[0070] Figure 2 A schematic diagram of the system architecture of an exemplary application space of a digital human driving method and device to which the embodiments of the present application can be applied is shown.

[0071] like Figure 2 As shown, the system architecture 200 applied by the technical solution provided in this application may include at least one terminal device, the Internet 204 and a server 205. Figure 2 The terminal devices 201, 202, and 203 are used as examples for illustration. In practice, there may be more or fewer terminal devices. The Internet 204 is used as a medium for providing communication links between the terminal devices 201, 202, and 203 and the server 205. The Internet 204 may include various connection types, such as wired or wireless communication links or optical fiber cables, etc. The terminal devices 201, 202, and 203 may be various electronic devices with display screens, including but not limited to desktop computers, portable computers, smart phones, and tablet computers, etc. It should be understood that Figure 2 The number of terminal devices, networks and servers in the embodiment is only for illustration. According to the implementation requirements, there may be any number of terminal devices, networks and servers. For example, the server 205 may be a server cluster composed of multiple servers.

[0072] The digital human driving method provided in the embodiment of the present application can be executed on the server 205. For example, the server 205 can collect video data of multiple perspectives in the real three-dimensional space through multiple video acquisition devices, determine the first coordinates of the key points of the target person, determine the mapping relationship according to the first coordinates, calculate the second coordinates according to the mapping relationship and the first coordinates, process the second coordinates based on the key point rotation model to obtain the rotation value of the virtual key point in the virtual three-dimensional space, and drive the digital human to move based on the rotation value of the virtual key point in the virtual three-dimensional space. The digital human driving method provided in the embodiment of the present application can also be executed by a terminal device, and the digital human driving method provided in the embodiment of the present application can also be executed jointly by the terminal device and the server 205, which is not specifically limited in this embodiment.

[0073] The video data involved in this application may be data authorized by the user or fully authorized by all parties.

[0074] The methods in the following embodiments can be implemented in a digital human driving device having the above hardware structure and software structure. The digital human driving method provided in the embodiments of the present application is described below.

[0075] Reference Figure 3As shown, the embodiment of the present application provides a digital human driving method, which is applied to a digital human driving device. The digital human driving device has the functions of receiving video data and image processing. The digital human driving device can be the server 105 in the above embodiment or a part thereof. The method can include S310-S360:

[0076] Before executing S310 , the digital human driving device first generates a digital human in a virtual three-dimensional space.

[0077] S310, collecting video data of multiple perspectives in a real three-dimensional space through multiple video collection devices.

[0078] The video data includes a target person, and the video acquisition device may be a monocular camera. Multiple video acquisition devices are multiple monocular cameras.

[0079] In some examples, the target person is located in a collection area in a real three-dimensional space, and the video collection device can collect video data containing the target person from various perspectives within the collection area.

[0080] In some examples, the video acquisition device may also be a multi-eye camera. As long as it can acquire video data of the target person from various perspectives, this application does not specifically limit the type and quantity of the video acquisition device.

[0081] In some examples, after the video acquisition device starts to collect video data, the target person may leave the collection area of ​​the video acquisition device during movement, or the target person may not enter the collection area after the video acquisition device starts to collect video data. In this case, the posture position of the digital human needs to be initialized. Figure 3 , reference Figure 4 As shown, after S310 , S410 to S430 are executed.

[0082] S410: Detect a target person in the video data.

[0083] In some examples, the digital human driving device can obtain video data captured by the video capture device through network transmission or USB transmission, and the video data can have color, video resolution of 720P, 1080P, 2K or 4K, video frame rate of 30 frames, video audio bit rate of 128kb / s, audio sampling rate of 44100Hz and other attributes.

[0084] In some examples, after acquiring the video data collected by the video acquisition device, the digital human driving device uses an image segmentation algorithm to detect whether a target person exists in the video data. Specifically, first, the digital human driving device obtains the image of the video frame of the video data as the original image based on the video data collected by the video acquisition device, and converts the original image into a black and white grayscale image to more easily separate the target person and the background in the original image; secondly, an edge detection algorithm is used to identify edge features in the black and white grayscale image; thirdly, the edge features are binarized to obtain a binarized image; then, a contour discovery algorithm is used to extract the target person contour and other contours from the binarized image; finally, other contours are filtered to obtain the target person contour.

[0085] S420: When the target person is detected, determine key points of the target person.

[0086] In some examples, when the digital human driving device detects the target person in the video data, HRNet is used to detect the key points of the target person. The key points are 21 key points such as some major joints of the target person, such as neck, shoulder, elbow, wrist, waist, knee and ankle. The specific process of HRNet detecting the key points of the target person includes: first, the image of the video frame of the video data in which the target person is detected is input into HRNet for feature extraction, and feature maps of different resolutions and scales are obtained in different feature layers; second, the feature maps extracted from multiple feature layers and the original image are formed into a pyramid, and more comprehensive, more representative and diverse feature maps are obtained through the interaction and information sharing between feature maps of multiple scales; then, the probability matrix of the key point coordinates is output in each feature map using the above extracted feature maps; finally, the predicted key point coordinates are output to the image of the video frame of the original video data, and the connection relationship between the key points can be represented by lines or circles.

[0087] It is understandable that when the digital human driving device detects the key points of the target person, in addition to HRNet, other key point detection algorithms can be used, such as the affinity field of key points (OpenPose) algorithm or the fast human pose estimation (FastPose) algorithm. This application does not make specific restrictions on the algorithm used by the digital human driving device to detect the key points of the target person.

[0088] S430: When the target person cannot be detected, the digital human maintains a standby posture.

[0089] In some examples, when the digital human driving device cannot detect the target person, that is, cannot detect the key point positions of the target person and subsequent key point motion information, it is necessary to initialize the digital human's posture to a standby posture, such as a vertical standing posture.

[0090] It is understandable that the standby posture may also be other postures besides vertical standing. The user may customize the standby posture according to actual needs, and this application does not impose any specific restrictions on the standby posture.

[0091] In the technical solution provided in S410 to S430 above, after the digital human driving device generates the digital human in the virtual three-dimensional space, it obtains the video data collected by the video acquisition device and detects whether there is a target person in the video data. If the target person is detected, it continues to detect the key point position of the target person. If the target person cannot be detected, the digital human driving device keeps the digital human in the virtual three-dimensional space in a standby posture, thereby avoiding the problem of jitter caused by other factors when the digital human driving device cannot detect the target person and the digital human does not receive action information. It can be seen that this technical solution can improve the stability of the digital human's posture.

[0092] S320: Determine the first coordinates of the key points of the target person.

[0093] The first coordinate is the two-dimensional coordinate of the key point in the video frame in the video data, and the specific first coordinate may be the two-dimensional coordinate of the key point in the image coordinate system.

[0094] In some examples, an image coordinate system is determined, with the image center of the video frame in the video data as the origin, the positive direction of the x-axis to the right, and the positive direction of the y-axis downward as the image coordinate system, and the coordinates of each key point in the image coordinate system are determined as the first coordinates.

[0095] S330. Determine a mapping relationship according to the first coordinate.

[0096] Among them, the mapping relationship is the correspondence between the first coordinate of the key point and the three-dimensional coordinate of the virtual key point in the virtual three-dimensional space. The virtual key points of the digital person in the virtual three-dimensional space correspond one-to-one to the key points of the target person in the real three-dimensional space. For example, there is a key point on the waist of the target person in the real three-dimensional space, and there is also a virtual key point on the waist of the digital person in the corresponding virtual three-dimensional space.

[0097] In some examples, taking the video acquisition device as a camera, the mapping relationship can be determined by the camera extrinsic parameters. Specifically, since the camera extrinsic parameters are used to indicate the correspondence between the first coordinate of the key point and the three-dimensional coordinate of the key point in the real three-dimensional space, if the aforementioned correspondence is used in the virtual three-dimensional space, the camera extrinsic parameters are used to indicate the correspondence between the first coordinate of the key point and the three-dimensional coordinate of the virtual key point in the virtual three-dimensional space. That is, at this time, the mapping relationship can be determined by the camera extrinsic parameters. In this case, the process of determining the mapping relationship is the process of camera calibration. Based on this, combined with Figure 3 , reference Figure 4 As shown, S330 includes S510 to S530.

[0098] S510: Determine a first parameter matrix according to first coordinates of the key point.

[0099] Among them, the first parameter matrix is ​​used to indicate the transformation relationship of the position point in the real three-dimensional space from the first coordinate system to the second coordinate system. The first coordinate system is a three-dimensional coordinate system in the real three-dimensional space with the center of the target person as the origin, that is, the world coordinate system, and the second coordinate system is a three-dimensional coordinate system in the real three-dimensional space with the optical center of the video acquisition device as the origin, that is, the camera coordinate system.

[0100] In some examples, the video acquisition device is taken as an example of a camera, the first coordinates are taken as the first coordinates of the key points at the four positions of the head, spine, and feet, and the process of determining the mapping relationship is taken as an example of the camera calibration process. The three-dimensional coordinates of the head, spine, and feet of the target person in the real three-dimensional space are obtained, and the rotation and translation vectors of the camera in the world coordinate system, i.e., the camera extrinsic parameters, are calculated as the first parameter matrix according to the first coordinates of the four key points and the three-dimensional coordinates of the four key points in the real three-dimensional space through the LM algorithm.

[0101] Since the process of determining the first parameter matrix in S510 only involves four key points: head, spine, and feet, and the digital human driving device can determine 21 key points through HRNet in S420, it can be seen that S510 may not be comprehensive enough to cause the result of the first parameter matrix to be inaccurate. Figure 5 , reference Figure 6 As shown, in order to obtain a more accurate first parameter matrix, S510 may include S610 to S650.

[0102] S610: Determine an initial first parameter matrix according to the first coordinates of the key points.

[0103] In some examples, the first parameter matrix determined in S510 according to the first coordinates of the key points at the four positions of the head, spine, and feet can be used as the initial first parameter matrix. The initial first parameter matrix needs to be further optimized because the key points involved are not comprehensive enough.

[0104] S620: Obtain the third coordinate of the key point.

[0105] Among them, the third coordinate is the three-dimensional coordinate of the key point in the real three-dimensional space.

[0106] In some examples, a world coordinate system is established with the center of the target person in the real three-dimensional space as the origin, and the three-dimensional coordinates of each key point of the target person are determined in the world coordinate system as the third coordinate.

[0107] S630, calculating a first coordinate prediction value according to the third coordinate and the initial first parameter matrix.

[0108] In some examples, the first parameter matrix is ​​taken as the camera extrinsic parameter, the initial first parameter matrix is ​​taken as the initial camera extrinsic parameter, and the third coordinate is taken as the three-dimensional coordinate of the key point in the real three-dimensional space. Since the initial camera extrinsic parameter is used to indicate the transformation relationship of the position point in the real three-dimensional space from the world coordinate system to the camera coordinate system, the digital human driving device can first obtain the camera intrinsic parameter and distortion parameter, and then calculate the predicted value of the key point at the first coordinate through the camera extrinsic parameter, distortion parameter, and camera intrinsic parameter in the third coordinate. The specific calculation reference formula is as follows:

[0109]

[0110] Among them, u and v are the first coordinate prediction values, A is the camera internal parameter matrix, Π is the distortion parameter matrix, T w is the camera extrinsic matrix, X W , Y W , Z W is the coordinate value of the third coordinate.

[0111] S640: Determine a first loss value based on the first coordinate prediction value and the first coordinate.

[0112] In some examples, the Euclidean distance between the first coordinate prediction value and the first coordinate is calculated as the first loss value.

[0113] S650, iteratively update the initial first parameter matrix based on the first loss value until the first loss value is less than a first preset threshold, and determine the latest updated initial first parameter matrix as the first parameter matrix.

[0114] In some examples, taking the video data duration of 3s, the frame rate of 30 frames, the number of key points of 21 as an example, and taking the first preset threshold of 20mm as an example, the digital human driving device can obtain 3×30×21=1890 groups of key point coordinate arrays, and use the bundle adjust algorithm to iteratively update the initial first parameter matrix with the first loss value. The optimization function of bundle adjust is as follows:

[0115]

[0116] Among them, u is the first coordinate of the key point, C is the camera extrinsic parameter, and X is the third coordinate in the world coordinate system. The optimization direction is to minimize the optimization function, that is, the 3D coordinate in the world coordinate system is transformed by the reprojection of the extrinsic parameter, and the error with the 2D key point in the image coordinate system is minimized. After N rounds of optimization, the 3D coordinates of the 2D key points of a single frame image are obtained using the camera extrinsic parameters. After reprojecting the 3D coordinates, the Euclidean distance with the 2D key points is calculated. The optimization is considered to be completed if the distance is less than 20mm. The initial first parameter matrix after the optimization is completed is used as the first parameter matrix.

[0117] In the technical solutions provided by S610 to S650 above, the initial first parameter matrix is ​​determined according to the first coordinates of the key points, the third coordinates of the key points are obtained, the first coordinate prediction value is calculated according to the third coordinates and the initial first parameter matrix, the first loss value is determined based on the first coordinate prediction value and the first coordinate, the initial first parameter matrix is ​​iteratively updated based on the first loss value until the first loss value is less than the first preset threshold value, and the latest updated initial first parameter matrix is ​​determined as the first parameter matrix. It can be seen that the technical solution can make the first parameter matrix more accurate by iteratively correcting the initial first parameter matrix multiple times, and solves the problem of low accuracy of the initial first parameter matrix due to insufficient key points involved.

[0118] S520: Obtain a second parameter matrix.

[0119] The second parameter matrix is ​​used to indicate the transformation relationship of the position point in the real three-dimensional space from the second coordinate system to the third coordinate system, and the three-coordinate system is a coordinate system with the image center of the video frame of the video data as the origin.

[0120] In some examples, the second parameter matrix includes an intrinsic parameter matrix and a distortion parameter matrix of a video acquisition device. The digital human driving device can obtain the intrinsic parameter matrix and the distortion parameter matrix of the video acquisition device as the second parameter matrix in a wired or wireless manner.

[0121] It is understandable that there is no logical order between S510 and S520. The digital human driving device can execute S510 first and then S520, or execute S520 first and then S510, or execute S510 and S520 at the same time. The embodiment of the present application does not impose any specific restrictions on the execution order of S510 and S520.

[0122] S530: Use the product matrix of the first parameter matrix and the second parameter matrix as a mapping relationship.

[0123] In the technical solution provided in S510 to S530, the first parameter matrix, i.e., the external parameter matrix, is determined according to the first coordinates of the key points, the second parameter matrix is ​​obtained, and the product matrix of the first parameter matrix and the second parameter matrix is ​​used as the mapping relationship. It can be seen that the technical solution can complete the calibration through the first coordinates of the key points without the calibration plate, reducing the laying cost of the calibration plate.

[0124] S340, calculating the second coordinate according to the mapping relationship and the first coordinate.

[0125] The second coordinate is the three-dimensional coordinate of the virtual key point in the virtual three-dimensional space.

[0126] In some examples, the second coordinate is calculated based on the mapping relationship and the first coordinate, and the specific formula is as follows:

[0127]

[0128] Among them, u and v are the coordinate values ​​of the first coordinate, T is the mapping relationship matrix, X V , Y V , Z V is the coordinate value of the second coordinate.

[0129] S350: Process the second coordinate based on the key point rotation model to obtain a rotation value of the virtual key point in the virtual three-dimensional space.

[0130] In some examples, a trained key point rotation model is used to process the second coordinate to obtain the rotation value of the virtual key point in the virtual three-dimensional space. The key point rotation model can be an IKNet. The specific process includes: determining the hierarchy of all key points, first moving the most child key point to the position of the second coordinate of the most child key point, and then moving all key points to the position of the second coordinate corresponding to the key point in order from low to high in the hierarchy, and calculating the rotation values ​​of all key points compared to the initial state.

[0131] Combination Figure 3 , reference Figure 7 As shown, before S350, the digital human driving device executes S710 to S740.

[0132] S710: Obtain at least one set of sample data.

[0133] The sample coordinates are the two-dimensional coordinates of the key points of the sample person in the video frame of the sample video.

[0134] In some examples, a sample image coordinate system is determined, with the image center of the video frame in the sample video data as the origin, the positive x-axis to the right, and the positive y-axis downward as the image coordinate system, and the coordinates of each key point of the sample character in the sample image coordinate system are determined as sample coordinates.

[0135] S720: Process the sample coordinates of the key points of the sample person based on the initial key point rotation model to obtain predicted rotation values.

[0136] In some examples, the initial key point rotation model can be an IKNet, and the specific process includes: determining the hierarchical structure of all sample key points, first moving the most child sample key point to the position of the sample coordinates of the most child sample key point, and then moving all sample key points to the positions of the sample coordinates corresponding to the sample key points in order from low to high in the hierarchy, and calculating the predicted rotation values ​​of all sample key points compared to the initial state.

[0137] In some examples, the data format of the sample coordinates and sample rotation values ​​of the key points of the sample person is in quaternion format. Figure 7 , reference Figure 8 As shown, S810 to S830 are executed after S720.

[0138] S810: Normalize the predicted rotation value.

[0139] In some examples, the predicted rotation values ​​are normalized by the min-max scaling method, which can be specifically referred to the following formula:

[0140]

[0141] Among them, x is the predicted rotation value before normalization, x new is the predicted rotation value after normalization, x min is the minimum value among the predicted rotation values, x max is the maximum value among the predicted rotation values.

[0142] It is understandable that the normalization processing method is not limited to the minimum-maximum scaling method, but can also be a standard score method or other methods. The present application does not make any specific restrictions on the method used for normalization processing.

[0143] S820: Encode the normalized predicted rotation value into a vector.

[0144] In some examples, the normalized predicted rotation value is input into the IKNet convolutional network encoder and encoded into a 1x512 dimensional variable. It is understood that the present application does not impose any specific restrictions on the dimension of the encoded vector here.

[0145] S830: Decode the vector into a predicted rotation value in a quaternion format.

[0146] In some examples, a 1x512-dimensional variable is input as an input to an IKNet convolutional network decoder for decoding. The IKNet convolutional network decoder converts the 1x512-dimensional vector into a multi-channel feature map. Then, based on these feature maps, a series of convolution and pooling operations are performed to obtain multiple predicted rotation values ​​in the form of quaternions.

[0147] In the technical solution provided by S810 to S830 above, the predicted rotation value is normalized, the predicted rotation value after normalization is encoded into a vector, and the vector is decoded into a predicted rotation value in quaternion format. It can be seen that the technical solution uses quaternions instead of matrix or Euler angle format to describe the rotation value. Compared with matrices, quaternions are more convenient to store and call, because quaternions only need 4 values ​​to describe a rotation value, while the matrix needs 9 values ​​to describe the same rotation value. Therefore, using quaternions to describe rotation values ​​can reduce the storage pressure of digital human driving devices.

[0148] S730. Determine a second loss value based on the predicted rotation value and the sample rotation value.

[0149] In some examples, the digital human driving device may use the quaternion cosin distance between the predicted rotation value and the sample rotation value as the second loss value. The specific formula of the quaternion cosin distance is as follows:

[0150] L cos =1-cos(Q GT *Q -1 )

[0151] Among them, L cos The quaternion cosine distance between the predicted rotation value and the sample rotation value, Q GT is the sample rotation value, and Q is the predicted rotation value.

[0152] In other examples, the digital human driving device may use the quaternion Euclidean distance between the predicted rotation value and the sample rotation value as the second loss value.

[0153] In some other examples, the digital human driving device may also use the quaternion normalized loss value of the predicted rotation value and the sample rotation value as the second loss value. It is understandable that the present application does not impose any specific limitation on the method of determining the second loss value.

[0154] S740, iteratively updating the initial key point rotation model based on the second loss value until the second loss value is less than a second preset threshold, and determining the most recently updated initial key point rotation model as the key point rotation model.

[0155] In some examples, the second loss value is back-propagated, one or more of the loss functions such as quaternion cosin distance, quaternion L2 distance, quaternion normalized loss and 3D coordinate L2 distance are back-propagated, the gradients of the parameters of each component in the model are calculated, and the parameters of each component in the model are updated using the Adam optimizer.

[0156] It is understandable that after the key point rotation model is obtained through S710 to S740, the digital human can be driven to move according to the video data of one frame, that is, the first coordinates of the key points on one video frame image.

[0157] In the technical solution provided by S710 to S740 above, at least one set of sample data is obtained, and the sample coordinates of the key points of the sample character are processed based on the initial key point rotation model to obtain a predicted rotation value, and a second loss value is determined based on the predicted rotation value and the sample rotation value, and the initial key point rotation model is iteratively updated based on the second loss value until the second loss value is less than the second preset threshold value, and the latest updated initial key point rotation model is determined as the key point rotation model. It can be seen that this technical solution can improve the accuracy of the key point rotation model and ensure the accuracy of the digital human and the target character when they move synchronously.

[0158] Combination Figure 3 , reference Fig. 9 As shown, before executing S360, S910 is executed.

[0159] S910: Smoothing the rotation value of the virtual key point in the virtual three-dimensional space.

[0160] In some examples, the Savgol algorithm is used to smooth the rotation values. The specific process includes: first, the rotation values ​​are represented as a time series, and each time point corresponds to a rotation value; second, the parameters of the Savgol filter are determined according to actual needs: window size, polynomial order, and derivative order; then, the rotation values ​​are filtered to obtain a smoothed time series; finally, the smoothed time series is recombined into rotation values ​​to obtain smoothed rotation values.

[0161] In the technical solution provided by S910 above, the rotation value of the virtual key point in the virtual three-dimensional space is smoothed, which can reduce noise, eliminate data spikes, and improve the movement stability of the digital human when the digital human driving device drives the digital human.

[0162] S360: driving the digital human to move based on the rotation value of the virtual key point in the virtual three-dimensional space.

[0163] In the technical solutions provided in S310 to S360 above, firstly, video data of multiple perspectives of the target person are collected in the real three-dimensional space by multiple video acquisition devices, so as to solve the problem that when the video data perspective is single, some key points of the target person cannot be detected, resulting in obvious jitter, joint dislocation and easy loss of the digital human in the driving process; secondly, the first coordinates of the key points of the target person are determined, and the mapping relationship is determined according to the first coordinates. The mapping relationship is the corresponding relationship between the key points and the virtual key points in the virtual three-dimensional space, that is, the process of determining the mapping relationship is the calibration position process of the video acquisition device. It can be seen that this technical solution completes the calibration through the first coordinates of the key points of the target person in the multi-perspective video data without a calibration plate, reducing the laying cost of the calibration plate; finally, the second coordinate is calculated according to the mapping relationship and the first coordinate, and the second coordinate is processed based on the key point rotation model to obtain the rotation value of the virtual key point in the virtual three-dimensional space. The digital human is driven to move based on the rotation value of the virtual key point in the virtual three-dimensional space, and the rotation values ​​of all key points are obtained at the same time by processing the second coordinate by the key point rotation model, thereby improving the driving efficiency of the digital human.

[0164] It can be seen that, on the one hand, the technical solution provided by the present application can solve the problem that when the video data has a single perspective, some key points of the target person cannot be detected, resulting in obvious jitter, joint dislocation and easy loss of the digital human during the driving process; on the other hand, the technical solution provided by the present application can complete the calibration through the first coordinates of the key points of the target person in the multi-perspective video data without a calibration plate, thereby reducing the cost of laying the calibration plate; on the other hand, the technical solution provided by the present application can improve the efficiency of digital human driving.

[0165] In order to more clearly illustrate the technical solution provided by the embodiment of the present application, taking the video acquisition device as a plurality of monocular cameras as an example, taking the first parameter matrix as the camera external parameter matrix, the second parameter matrix as the camera internal parameter as an example, taking the key point selection model as IKNet as an example, the digital human driving method provided by the present application may include the following: Fig.10 The process shown may specifically include:

[0166] 1. Multiple monocular cameras collect video streams.

[0167] For example, multiple monocular cameras are set up, N is the number of cameras, N is greater than 2, and the target person is within the acquisition range of multiple monocular cameras, and first stands vertically (standby posture) to wait for camera calibration (obtaining the first parameter). After the target person stands vertically, multiple monocular cameras start to collect video streams. Each monocular camera has a different viewing angle and outputs the single-person action video data of the target person within the acquisition range. The audio bit rate of the video data is 128kb / s, and the audio sampling rate is 44100Hz.

[0168] 2. Human body detection.

[0169] For example, first, the digital human driving device obtains the image of the video frame of the video data as the original image based on the video data collected by multiple monocular cameras, and converts the original image into a black and white grayscale image to more easily separate the target person and the background in the original image; secondly, the edge detection algorithm is used to identify the edge features in the black and white grayscale image; thirdly, the edge features are binarized to obtain a binary image; then, the contour discovery algorithm is used to extract the target person contour and other contours from the binary image; finally, other contours are filtered to obtain the target person contour.

[0170] 3. When the target person is detected, key point detection is performed.

[0171] For example, for each video frame of the video data collected by each monocular camera, the HRNet algorithm is used to calculate 21 2D key points of the human body, and the output is the 2D coordinates (first coordinates) of the 2D key points in the shape of Nx21x2.

[0172] 4. When the target person cannot be detected, all joint rotation angles are output as 0, making the digital human still.

[0173] For example, if the outline of the target person is not detected in process 3, the digital human driving device outputs all joint rotation angles as 0, making the digital human still.

[0174] 5. When calibration information exists for multiple monocular cameras, directly read the internal and external parameters of multiple monocular cameras.

[0175] For example, when calibration information (a first parameter matrix and a second parameter matrix) exists for multiple monocular cameras, the digital human driving device directly accesses the storage module of each monocular camera through a wired or wireless means to obtain the calibration information.

[0176] 6. When there is no calibration information for multiple monocular cameras, perform passive calibration.

[0177] For example, the passive calibration process refers to Fig.11As shown, specifically, first, the digital human driving device extracts the 2D coordinates (first coordinates) of the 2D key points from one of the above frames, and according to the factory preset intrinsic parameters and distortion coefficients (second parameter matrix) of the camera, the shape of the intrinsic parameter data is 3x3, and the shape of the distortion coefficient is 1x5. Based on the solvePnP algorithm, the LM optimization solution is used to obtain the rough position and rotation (initial first parameter matrix) of each camera, where the position vector shape is 1×3 and the rotation vector shape is 1×3; secondly, more frames in the video stream are obtained, about 3s, 90 frames, and the 2D coordinates of 21 joints of the human body in each frame are extracted to start the precise parameter optimization of the camera. The data shape is N×90×21×2, which is transformed into N×1809×2, with a total of 1809 sets of data; finally, bundle is used The adjust algorithm optimizes the rough position and rotation of the camera extracted above. The LM optimization algorithm is used to randomly select 500 sets of data each time, and iterates 20 times in total. It iterates until the reprojection error (first loss value) is less than 20mm (the first preset threshold) to obtain accurate camera intrinsic parameters (second parameter matrix) and extrinsic parameters (first parameter matrix) to complete passive calibration.

[0178] 7. The triangulation algorithm obtains the three-dimensional coordinates of the key points.

[0179] For example, the 2D key point coordinates (first coordinates) of 21 joints are extracted through HRNet, and the data shape is Nx21x2. The 3D coordinates (third coordinates) of 21 joints of the human body are obtained through the triangulation algorithm, and the data shape is Nx21x3.

[0180] 8. Input the 3D coordinates of the key points into IKNet to obtain the joint rotation angle in quaternion format.

[0181] For example, refer to Fig.12 As shown in the figure, the specific process of inputting the 3D coordinates of the key points into IKNet and obtaining the joint rotation angle in quaternion format includes:

[0182] (1) Pairs of human 3D key points (sample coordinates) and joint rotation angle data (sample rotation values) collected by motion capture equipment, where the data shape of each frame of human 3D key points is 1x21x3, and the data is normalized to between 0 and 1.

[0183] (2) The data is input into the IKNet convolutional network encoder and encoded into a 1x512 dimensional variable.

[0184] (3) Use the above 1x512 dimensional variable to input the IKNet convolutional network decoder, and output the joint rotation angles of 21 joints in quaternion format with an output data shape of 1x21x4.

[0185] (4) Compare the quaternion rotation value (predicted rotation value) output by the model with the quaternion (sample rotation value) collected by the motion capture device, and calculate three loss functions (second loss value) including quaternion cosine distance, quaternion L2 distance, and quaternion normalized loss.

[0186] (5) Through the FK forward dynamics algorithm, the 21 joint point quaternions output by the model are converted into 3D key points with a shape of 1x21x3, and the L2 distance of the 3D coordinates of the input 3D key points is calculated.

[0187] (6) Perform the back-propagation process, back-propagate four loss functions including quaternion cosine distance, quaternion L2 distance, quaternion normalized loss, and 3D coordinate L2 distance, calculate the gradient of the parameters of each component in the model, and use the Adam optimizer to update the parameters of each component in the model.

[0188] (7) The model training process includes several rounds, and each round executes the above model training process until the training termination condition is met and the model training is completed.

[0189] 9. Use the savgol smoothing algorithm to smooth the joint rotation angle.

[0190] For example, first, the rotation value is represented as a time series, and each time point corresponds to a rotation value; secondly, the parameters of the Savgol filter are determined according to actual needs: window size, polynomial order, and derivative order; then, the rotation value is filtered and a smoothed time series is obtained; finally, the smoothed time series is recombined into the rotation value to obtain the smoothed rotation value.

[0191] 10. The digital human driving device drives the digital human to move.

[0192] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of the method. In order to realize the above functions, it includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0193] The embodiment of the present application can divide the digital human driving device into functional modules according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0194] Reference Fig.13 As shown, an embodiment of the present application provides a digital human driving device, which may include a video acquisition module 131, a first coordinate determination module 132, a mapping relationship determination module 133, a second coordinate calculation module 134, a second coordinate processing module 135 and a digital human driving module 136.

[0195] Specifically, the video acquisition module 131 is used to acquire video data of multiple perspectives in a real three-dimensional space through multiple video acquisition devices; wherein the video data includes a target person; the first coordinate determination module 132 is used to determine the first coordinate of the key point of the target person in the video data acquired by the video acquisition module; wherein the first coordinate is the two-dimensional coordinate of the key point in the video frame in the video data; the mapping relationship determination module 133 is used to determine the mapping relationship according to the first coordinate determined by the first coordinate determination module; wherein the mapping relationship is the corresponding relationship between the key point and the virtual key point in the virtual three-dimensional space; the second coordinate calculation module 134 is used to calculate according to the mapping relationship determined by the mapping relationship determination module and the first coordinate determined by the first coordinate module; wherein the second coordinate is the three-dimensional coordinate of the virtual key point in the virtual three-dimensional space; the second coordinate processing module 135 is used to process the second coordinate calculated by the second coordinate calculation module based on the key point rotation model to obtain the rotation value of the virtual key point in the virtual three-dimensional space; the digital human driving module 136 is used to drive the digital human to move based on the rotation value in the virtual three-dimensional space obtained by the second coordinate processing module.

[0196] In a possible implementation, after collecting video data of multiple perspectives in a real three-dimensional space through multiple video acquisition devices, the digital human driving device also includes: detecting a target person in the video data; when the target person is detected, determining key points of the target person; when the target person cannot be detected, the digital human maintains a standby posture.

[0197] In a possible implementation, a mapping relationship is determined according to the first coordinates, and the mapping relationship determination module 133 is specifically used to: determine a first parameter matrix according to the first coordinates of the key point; wherein the first parameter matrix is ​​used to indicate the transformation relationship of the position point in the real three-dimensional space from the first coordinate system to the second coordinate system, the first coordinate system is a three-dimensional coordinate system in the real three-dimensional space with the center of the target person as the origin, and the second coordinate system is a three-dimensional coordinate system in the real three-dimensional space with the optical center of the video acquisition device as the origin; obtain a second parameter matrix; wherein the second parameter matrix is ​​used to indicate the transformation relationship of the position point in the real three-dimensional space from the second coordinate system to the third coordinate system, the three-coordinate system is a coordinate system with the image center of the video frame of the video data as the origin; and use the product matrix of the first parameter matrix and the second parameter matrix as the mapping relationship.

[0198] In a possible implementation, the first parameter matrix is ​​calculated according to the first coordinates of the key point, and the mapping relationship determination module 133 is specifically used to: determine the initial first parameter matrix according to the first coordinates of the key point; obtain the third coordinates of the key point; wherein the third coordinates are the three-dimensional coordinates of the key point in the real three-dimensional space; calculate the first coordinate prediction value according to the third coordinates and the initial first parameter matrix; determine the first loss value based on the first coordinate prediction value and the first coordinate; iteratively update the initial first parameter matrix based on the first loss value until the first loss value is less than a first preset threshold value, and determine the latest updated initial first parameter matrix as the first parameter matrix.

[0199] In a possible implementation, before the key point rotation model processes the second coordinate to obtain the rotation value of the virtual key point in the virtual three-dimensional space, the digital human driving device also includes: acquiring at least one set of sample data; the sample data includes sample coordinates and sample rotation values ​​of the key points of the sample character in the sample video; the sample coordinates are the two-dimensional coordinates of the key points of the sample character in the video frame of the sample video; based on the initial key point rotation model, the sample coordinates of the key points of the sample character are processed to obtain a predicted rotation value; based on the predicted rotation value and the sample rotation value, a second loss value is determined; based on the second loss value, the initial key point rotation model is iteratively updated until the second loss value is less than a second preset threshold, and the latest updated initial key point rotation model is determined as the key point rotation model.

[0200] In a possible implementation, the data format of the sample coordinates and sample rotation values ​​of the key points of the sample character is a quaternion format.

[0201] In one possible implementation, after processing the second coordinate based on the key point rotation model to obtain the rotation value of the virtual key point in the virtual three-dimensional space, before driving the digital human to move based on the rotation value of the virtual key point in the virtual three-dimensional space, the digital human driving device also includes: smoothing the rotation value of the virtual key point in the virtual three-dimensional space.

[0202] It should be understood that the division of units or modules (hereinafter referred to as units) in the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or physically separated. And the units in the device can all be implemented in the form of software calling through processing elements; or all in the form of hardware; or some units can be implemented in the form of software calling through processing elements, and some units can be implemented in the form of hardware.

[0203] For example, each unit can be a separately established processing element, or it can be integrated in a certain chip of the device. In addition, it can be stored in a memory in the form of a program, and called and executed by a certain processing element of the device. In addition, all or part of these units can be integrated together, or they can be implemented independently. The processing element here can also be called a processor, which can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each unit above can be implemented by an integrated logic circuit of hardware in the processor element or in the form of software called by a processing element.

[0204] In one example, the unit in the above apparatus may be one or more integrated circuits configured to implement the above method, such as one or more ASICs, or one or more DSPs, or one or more FPGAs, or a combination of at least two of these integrated circuit forms.

[0205] For another example, when the units in the device can be implemented in the form of a processing element scheduling program, the processing element can be a general-purpose processor, such as a CPU or other processor that can call a program. For another example, these units can be integrated together and implemented in the form of a system on a chip SOC.

[0206] In one implementation, the units implementing the corresponding steps in the above method can be implemented in the form of a processing element scheduling program. For example, the device may include a processing element and a storage element, and the processing element calls the program stored in the storage element to execute the digital human driving method of the above method embodiment. The storage element can be a storage element on the same chip as the processing element, that is, an on-chip storage element.

[0207] In another implementation, the program for executing the above method may be in a storage element on a different chip from the processing element, i.e., an off-chip storage element. In this case, the processing element calls or loads the program from the off-chip storage element to the on-chip storage element to call and execute the digital human driving method of the above method embodiment.

[0208] Reference Fig.14 As shown, the embodiment of the present application also provides a digital human driving device, including a communicator 141, which is configured to collect video data of multiple perspectives in a real three-dimensional space through multiple video acquisition devices; wherein the video data includes a target person; a processor 142, coupled to the communicator, is configured to determine a first coordinate of a key point of the target person; wherein the first coordinate is a two-dimensional coordinate of the key point in a video frame in the video data; a mapping relationship is determined according to the first coordinate; wherein the mapping relationship is a corresponding relationship between the key point and the virtual key point in the virtual three-dimensional space; a second coordinate is calculated according to the mapping relationship and the first coordinate; wherein the second coordinate is a three-dimensional coordinate of the virtual key point in the virtual three-dimensional space; the second coordinate is processed based on the key point rotation model to obtain a rotation value of the virtual key point in the virtual three-dimensional space; and the digital human is driven to move based on the rotation value of the virtual key point in the virtual three-dimensional space.

[0209] In a possible implementation, after collecting video data of multiple perspectives in a real three-dimensional space through multiple video acquisition devices, the processor 142 is configured to detect a target person in the video data; when the target person is detected, determine the key points of the target person; when the target person cannot be detected, the digital human maintains a standby posture.

[0210] In one possible implementation, a mapping relationship is determined based on the first coordinates, and the processor 142 is configured to determine a first parameter matrix based on the first coordinates of the key point; wherein the first parameter matrix is ​​used to indicate a transformation relationship of a position point in a real three-dimensional space from a first coordinate system to a second coordinate system, the first coordinate system is a three-dimensional coordinate system in the real three-dimensional space with the center of the target person as the origin, and the second coordinate system is a three-dimensional coordinate system in the real three-dimensional space with the optical center of the video acquisition device as the origin; obtain a second parameter matrix; wherein the second parameter matrix is ​​used to indicate a transformation relationship of a position point in the real three-dimensional space from the second coordinate system to a third coordinate system, the three-coordinate system is a coordinate system with the image center of a video frame of the video data as the origin; and a product matrix of the first parameter matrix and the second parameter matrix is ​​used as a mapping relationship.

[0211] In one possible implementation, a first parameter matrix is ​​calculated based on the first coordinates of the key point, and the processor 142 is configured to determine an initial first parameter matrix based on the first coordinates of the key point; obtain a third coordinate of the key point; wherein the third coordinate is the three-dimensional coordinate of the key point in the real three-dimensional space; calculate a first coordinate prediction value based on the third coordinate and the initial first parameter matrix; determine a first loss value based on the first coordinate prediction value and the first coordinate; iteratively update the initial first parameter matrix based on the first loss value until the first loss value is less than a first preset threshold value, and determine the most recently updated initial first parameter matrix as the first parameter matrix.

[0212] In one possible implementation, before processing the second coordinate based on the key point rotation model to obtain the rotation value of the virtual key point in the virtual three-dimensional space, the processor 142 is configured to obtain at least one set of sample data; the sample data includes sample coordinates and sample rotation values ​​of the key points of the sample character in the sample video; the sample coordinates are the two-dimensional coordinates of the key points of the sample character in the video frame of the sample video; the sample coordinates of the key points of the sample character are processed based on the initial key point rotation model to obtain a predicted rotation value; based on the predicted rotation value and the sample rotation value, a second loss value is determined; based on the second loss value, the initial key point rotation model is iteratively updated until the second loss value is less than a second preset threshold, and the latest updated initial key point rotation model is determined as the key point rotation model.

[0213] In a possible implementation, the data format of the sample coordinates and sample rotation values ​​of the key points of the sample character is a quaternion format.

[0214] In one possible implementation, after the second coordinate is input into the key point rotation model to output the rotation value of each virtual key point in the virtual three-dimensional space, and before the digital human is driven to move based on the rotation value of the virtual key point in the virtual three-dimensional space, the processor 142 is configured to smooth the rotation value of the virtual key point in the virtual three-dimensional space.

[0215] The embodiment of the present application also provides a digital human driving device, which may include: a display screen, a memory, and one or more processors. The display screen, the memory, and the processor are coupled. The memory is used to store computer program code, and the computer program code includes computer instructions. When the processor executes the computer instructions, the digital human driving device can perform various functions or steps performed by the digital human driving device in the above method embodiment.

[0216] For example, the present application also provides a chip, which can be applied to the above-mentioned digital human driving device or server. The chip includes one or more interface circuits and one or more processors; the interface circuit and the processor are interconnected through a line; the processor receives and executes computer instructions from the memory of the digital human driving device through the interface circuit to implement the method in the above method embodiment.

[0217] The present application also provides a computer-readable storage medium on which computer program instructions (or instructions) are stored. When the computer program instructions are executed by a digital human driving device, the digital human driving device can implement the digital human driving method as described above.

[0218] The embodiment of the present application also provides a computer program product, including computer instructions for running the digital human driving device as described above. When the computer program product runs in the digital human driving device, the digital human driving device can implement the digital human driving method as described above.

[0219] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0220] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0221] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0222] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0223] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, such as a program. The software product is stored in a program product, such as a computer-readable storage medium, including a number of instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks, or optical disks.

[0224] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto, and any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A training method for a key point rotation model, characterized in that: The method comprises: Acquire at least one set of sample data; the sample data includes sample coordinates and sample rotation values ​​of key points of the sample person; wherein the sample coordinates are three-dimensional coordinates of the key points of the sample person in a real three-dimensional space; Processing the sample coordinates of the key points of the sample person based on the initial key point rotation model to obtain a predicted rotation value; the predicted rotation value is the rotation value of the key points of the sample person compared to the initial state when the key points of the sample person are moved to the positions of the sample coordinates corresponding to the key points of the sample person; Based on the predicted rotation value and the sample rotation value, the initial key point rotation model is iteratively updated to obtain a key point rotation model; the key point rotation model has the function of obtaining the rotation value of the character based on the three-dimensional coordinates of the key points of the character, and the rotation value of the character obtained by the key point rotation model is used to drive the digital human to move; The iterative updating of the initial key point rotation model based on the predicted rotation value and the sample rotation value to obtain the key point rotation model includes: Determining a second loss value based on the predicted rotation value and the sample rotation value; Iteratively updating the initial key point rotation model based on the second loss value until the second loss value is less than a second preset threshold, and determining the most recently updated initial key point rotation model as the key point rotation model; The method of using the rotation value of the character obtained by the key point rotation model to drive the digital human to move includes: Acquire video data of multiple viewing angles in a real three-dimensional space; the video data includes a target person; Determine the first coordinates of the key points of the target person; Determine a first parameter matrix according to the first coordinates of the key point; wherein the first parameter matrix is ​​used to indicate the transformation relationship of the position point in the real three-dimensional space from the first coordinate system to the second coordinate system, the first coordinate system is a three-dimensional coordinate system in the real three-dimensional space with the center of the target person as the origin, and the second coordinate system is a three-dimensional coordinate system in the real three-dimensional space with the optical center of the video acquisition device as the origin; Acquire a second parameter matrix; wherein the second parameter matrix is ​​used to indicate a transformation relationship of a position point in the real three-dimensional space from a second coordinate system to a third coordinate system, wherein the third coordinate system is a coordinate system with an image center of a video frame of the video data as an origin; determine a product matrix of the first parameter matrix and the second parameter matrix as a mapping relationship between the key point and the virtual key point in the virtual three-dimensional space; Calculate a second coordinate according to the mapping relationship and the first coordinate; Processing the second coordinate based on the key point rotation model to obtain a rotation value of the virtual key point in the virtual three-dimensional space; The digital human is driven to move based on the rotation value of the virtual key point in the virtual three-dimensional space.

2. The method according to claim 1, characterized in that The data format of the sample coordinates and sample rotation values ​​of the key points of the sample person is a quaternion format; after processing the sample coordinates of the key points of the sample person based on the initial key point rotation model to obtain the predicted rotation value, before iteratively updating the initial key point rotation model based on the predicted rotation value and the sample rotation value to obtain the key point rotation model, the method further includes: Normalizing the predicted rotation value; Encoding the normalized predicted rotation value into a vector; The vector obtained by encoding the predicted rotation value is decoded into a predicted rotation value in quaternion format.

3. The method according to claim 2, characterized in that The determining a second loss value based on the predicted rotation value and the sample rotation value comprises: The quaternion cosine distance between the predicted rotation value and the sample rotation value is used as the second loss value; and / or, using the quaternion Euclidean distance between the predicted rotation value and the sample rotation value as the second loss value; And / or, a quaternion normalized loss value of the predicted rotation value and the sample rotation value is used as the second loss value.

4. The method according to any one of claims 1 to 3, characterized in that: The initial key point rotation model is IKNet.

5. A training device for a key point rotation model, characterized in that: The device comprises: An acquisition module, used to acquire at least one set of sample data; the sample data includes sample coordinates and sample rotation values ​​of key points of the sample person; wherein the sample coordinates are three-dimensional coordinates of the key points of the sample person in a real three-dimensional space; A training module, for processing the sample coordinates of the key points of the sample person acquired by the acquisition module based on an initial key point rotation model to obtain a predicted rotation value; the predicted rotation value is a rotation value of the key points of the sample person compared to an initial state when the key points of the sample person are moved to the positions of the sample coordinates corresponding to the key points of the sample person; The training module is further used to determine a second loss value based on the predicted rotation value and the sample rotation value; iteratively update the initial key point rotation model based on the second loss value until the second loss value is less than a second preset threshold, and determine the latest updated initial key point rotation model as the key point rotation model; the key point rotation model has the function of obtaining the rotation value of the character based on the three-dimensional coordinates of the key points of the character, and the rotation value of the character obtained by the key point rotation model is used to drive the digital human to move; The method of using the rotation value of the character obtained by the key point rotation model to drive the digital human to move includes: Acquire video data of multiple viewing angles in a real three-dimensional space; the video data includes a target person; Determine the first coordinates of the key points of the target person; Determine a first parameter matrix according to the first coordinates of the key point; wherein the first parameter matrix is ​​used to indicate the transformation relationship of the position point in the real three-dimensional space from the first coordinate system to the second coordinate system, the first coordinate system is a three-dimensional coordinate system in the real three-dimensional space with the center of the target person as the origin, and the second coordinate system is a three-dimensional coordinate system in the real three-dimensional space with the optical center of the video acquisition device as the origin; Acquire a second parameter matrix; wherein the second parameter matrix is ​​used to indicate a transformation relationship of a position point in the real three-dimensional space from a second coordinate system to a third coordinate system, wherein the third coordinate system is a coordinate system with an image center of a video frame of the video data as an origin; determine a product matrix of the first parameter matrix and the second parameter matrix as a mapping relationship between the key point and the virtual key point in the virtual three-dimensional space; Calculate a second coordinate according to the mapping relationship and the first coordinate; Processing the second coordinate based on the key point rotation model to obtain a rotation value of the virtual key point in the virtual three-dimensional space; The digital human is driven to move based on the rotation value of the virtual key point in the virtual three-dimensional space.

6. An electronic device, characterized in that: include: processor; A memory, configured to store executable instructions of the processor; Wherein, the processor is configured to execute the key point rotation model training method described in any one of claims 1-4 by executing the executable instructions.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the training method for the key point rotation model as described in any one of claims 1-4.

Images