Object model construction method and device, electronic equipment and storage medium

By combining parametric object models and surface node models to detect pose changes and fusing facial models, the problem of inaccurate high-speed motion modeling is solved, and high-precision 3D reconstruction of human bodies or objects is achieved.

CN118898680APending Publication Date: 2024-11-05TENCENT TECH SHANGHAI
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310499896.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-04
Publication Date
2024-11-05

AI Technical Summary

Technical Problem

Existing technologies are prone to problems such as detection loss and inaccurate facial geometry and expression detection when detecting and reconstructing high-speed moving human bodies or objects, resulting in imprecise modeling.

Method used

By combining the initial parametric object model and the initial surface node model, pose change detection is performed on the target object in the target video, and an initial facial model is obtained. These models are then fused to construct the object model, thereby improving the modeling accuracy.

Benefits of technology

It effectively detects pose changes at both fast and slow speeds, avoids missing detections, accurately reproduces facial geometry and expressions, and improves modeling accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118898680A_ABST
    Figure CN118898680A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of computers, and provides an object model construction method and device, electronic equipment and a storage medium, which are used for improving the modeling accuracy of an object model. The method comprises the following steps: for a target object, acquiring an initial parameterized object model and an initial surface node model corresponding to a current frame in a target video; based on the initial parameterized object model and the initial surface node model, performing pose detection on a target object in the current frame, and constructing an initial object surface model of the target object in the current frame; obtaining an initial face model corresponding to the current frame, and performing face detection on a target object in the current frame based on the initial face model to obtain a target face model; and fusing the initial object surface model and the target face model to obtain an object model of the target object in the current frame. According to the method, the pose detection is performed based on the parameterized object model and the surface node model, and the face is further detected, so that the accuracy of model construction is higher.
Need to check novelty before this filing date? Find Prior Art

Description

Background Art

[0002] With the development of computer technology, people have gradually mastered the three-dimensional graphics (3D) technology. Based on this technology, models with three-dimensional data can be constructed in a virtual three-dimensional space, so it has been widely used in games, movies, animations, product displays, model design, etc. One of the 3D modeling methods can directly build a model based on a real video to restore the characters and scenes in the video, and the dynamic detection of the motion trajectory of the human body and objects and three-dimensional reconstruction are the two most important aspects involved in this method.

[0003] The detection and reconstruction algorithms in related technologies generally include two parts: detection and reconstruction. The detection part reflects the changes of the model in space by obtaining the motion information of each sampling point of the model between each frame of the video; then, the reconstruction part transforms the motion information of the sampling point into a "surface" based on the voxel grid technology, and finally generates a surface model of the human body or object. However, this process only uses sampling points for detection, and can only detect slow-moving objects or human bodies. For high-speed motion, detection will be lost, resulting in inaccurate subsequent modeling; and this process cannot accurately detect and reproduce the geometry and expression of the face, which further causes imprecise modeling.

[0004] In summary, how to improve the modeling accuracy of the object model is an urgent problem to be solved. Summary of the invention

[0005] The embodiments of the present application provide a method, device, electronic device and storage medium for constructing an object model, so as to improve the modeling accuracy of the object model.

[0006] An embodiment of the present application provides a method for constructing an object model, comprising:

[0007] For the target object, an initial parameterized object model and an initial surface node model corresponding to the current frame are obtained in the target video; the initial parameterized object model represents the joint transformation information of each joint of the target object relative to the reference frame, and the initial surface node model represents the node transformation information of multiple surface nodes on the model relative to the reference frame;

[0008] Based on the initial parameterized object model and the initial surface node model, performing posture detection on the target object in the current frame, and constructing an initial object surface model of the target object in the current frame;

[0009] Acquire an initial facial model corresponding to the current frame, and perform facial detection on the target object in the current frame based on the initial facial model to obtain a target facial model; the initial facial model represents transformation information of multiple key points on the model relative to key points of a reference frame;

[0010] The initial object surface model is fused with the target face model to obtain an object model of the target object in the current frame.

[0011] An object model construction device provided in an embodiment of the present application includes:

[0012] An acquisition unit is used to acquire, for a target object, an initial parameterized object model and an initial surface node model corresponding to a current frame in a target video; the initial parameterized object model represents joint transformation information of each joint of the target object relative to a reference frame, and the initial surface node model represents node transformation information of multiple surface nodes on the model relative to the reference frame;

[0013] A construction unit, configured to perform posture detection on the target object in the current frame based on the initial parameterized object model and the initial surface node model, and to construct an initial object surface model of the target object in the current frame;

[0014] A detection unit, configured to obtain an initial facial model corresponding to the current frame, and perform facial detection on the target object in the current frame based on the initial facial model to obtain a target facial model; the initial facial model represents transformation information of multiple key points on the model relative to key points of a reference frame;

[0015] A fusion unit is used to fuse the initial object surface model with the target face model to obtain an object model of the target object in the current frame.

[0016] Optionally, the acquisition unit is further used for:

[0017] Acquire a depth point cloud model of the target object; the depth point cloud model is constructed based on a multi-view depth map corresponding to the current frame;

[0018] The building blocks are specifically used for:

[0019] Using the initial parameterized object model, the initial surface node model, and the depth point cloud model, multiple rounds of posture detection are performed on the target object in the current frame, and the surface node model obtained by the last round of posture detection is used as the target surface node model of the target object in the current frame; wherein the initial parameterized object model and the initial surface node model are alternately updated during each round of posture detection;

[0020] An initial object surface model of the target object in the current frame is constructed according to node transformation information of a plurality of surface nodes in the target surface node model of the current frame.

[0021] Optionally, the construction unit is specifically configured to perform the following process in each round of posture detection:

[0022] Based on the difference between the current latest parameterized object model and the depth point cloud model, and the difference between the current latest parameterized object model and the parameterized object model to be updated, adjusting the parameterized object model to be updated;

[0023] Based on the adjusted parameterized object model, the current latest surface node model is adjusted to obtain an intermediate surface node model; in the first round of posture detection, the current latest parameterized object model is the initial parameterized object model, and the current latest surface node model is the initial surface node model;

[0024] Based on the difference between the intermediate surface node model and the depth point cloud model, and the difference between the intermediate surface node model and the surface node model to be updated, adjusting the surface node model to be updated;

[0025] Based on the adjusted surface node model, the adjusted parameterized object model is adjusted again.

[0026] Optionally, if the current frame is a designated frame, the initial parameterized object model is obtained by adjusting at least once with a preset parameterized object model, and the initial surface node model is obtained by sampling the initial parameterized object model; the preset parameterized object model is constructed based on a reference frame;

[0027] If the current frame is not a designated frame, the initial parameterized object model is obtained by transforming a target surface node model of a previous frame, and the initial surface node model is the target surface node model of the previous frame.

[0028] Optionally, if the current frame is a designated frame, the initial facial model is a preset facial model corresponding to the reference frame; if the current frame is not a designated frame, the initial facial model is a target facial model of a previous frame;

[0029] The detection unit is specifically used for:

[0030] Acquire a depth point cloud model of the target object and an initial facial model corresponding to the current frame; the depth point cloud model is constructed based on a multi-view depth map corresponding to the current frame;

[0031] Based on the difference between the facial model to be updated and the facial part in the depth point cloud model, and the difference between the facial model to be updated and the initial facial model, the facial model to be updated is adjusted to obtain the target facial model.

[0032] Optionally, the fusion unit is specifically used for:

[0033] Acquire a depth point cloud model of the target object; the depth point cloud model is constructed based on a multi-view depth map corresponding to the current frame;

[0034] The depth point cloud model, the initial object surface model and the target face model are fused to obtain an object model of the target object in the current frame.

[0035] Optionally, the construction unit is specifically used for:

[0036] Obtaining a first vertex comparison difference between each vertex in the current latest parameterized object model and a corresponding point in the depth point cloud model;

[0037] Acquire the joint transformation difference between the joint transformation information of the current latest parameterized object model and the parameterized object model to be updated, the joint point position difference between the positions of the joint points, and the first vertex position difference between the positions of the vertices;

[0038] The parameterized object model to be updated is adjusted based on the first vertex comparison difference, the joint transformation difference, the joint point position difference and the first vertex position difference.

[0039] Optionally, the construction unit is specifically used for:

[0040] Obtaining surface vertex comparison differences between each surface vertex in the intermediate surface node model and corresponding points in the depth point cloud model;

[0041] Obtaining node transformation differences between node transformation information in the intermediate surface node model and the surface node model to be updated, and surface node position differences between positions of each surface node;

[0042] The surface node model to be updated is adjusted based on the surface vertex comparison difference, the node transformation difference and the surface node position difference.

[0043] Optionally, the detection unit is specifically used for:

[0044] Obtaining a second vertex comparison difference between each vertex in the facial model to be updated and a corresponding point on the depth point cloud model;

[0045] Obtaining a key point transformation difference between key point transformation information in the face model to be updated and the initial face model;

[0046] Based on the second vertex comparison difference and the key point transformation difference, the facial model to be updated is adjusted to obtain the target facial model.

[0047] Optionally, the key point transformation information includes: facial position transformation information, facial morphology transformation information and facial expression transformation information of the target object in the current frame relative to the reference frame.

[0048] Optionally, the construction unit is specifically used for:

[0049] Acquire the joint transformation information of the latest parameterized object model and the joint posture difference between the prior posture parameters; the prior posture parameters represent the joint transformation information that conforms to the preset joint posture;

[0050] Obtaining the estimated difference between the projection position of each joint point in the parameterized object model to be updated under multiple viewing angles and the projection position obtained based on the posture estimation;

[0051] The parameterized object model to be updated is adjusted based on the first vertex comparison difference, the joint transformation difference, the joint point position difference, the joint posture difference, the joint point estimation difference and the first vertex position difference.

[0052] Optionally, the construction unit is specifically used for:

[0053] Acquire the positional relationship between each surface node and its adjacent surface nodes in the surface node model to be updated;

[0054] The surface node model to be updated is adjusted based on the surface vertex comparison difference, the node transformation difference, the surface node position difference and the position relationship.

[0055] Optionally, the detection unit is specifically used for:

[0056] Acquire the facial motion difference between the key point transformation information of the face model to be updated and the prior key point transformation information; the prior key point transformation information includes prior expression information and prior morphology information;

[0057] Obtaining the estimated differences of facial key points between the projection positions of the facial key points of the facial model to be updated under multiple viewing angles and the projection positions of the facial key points under corresponding viewing angles obtained based on posture estimation;

[0058] Based on the second vertex comparison difference, the key point transformation difference, the facial motion difference and the facial key point estimated difference, the facial model to be updated is adjusted to obtain the target facial model.

[0059] Optionally, the fusion unit is specifically used for:

[0060] Projecting the initial object surface model into a depth space to obtain a truncated directed distance of each voxel in the initial object surface model; the depth space is determined based on the depth point cloud model and is used to represent the depth value of each point in the depth point cloud model;

[0061] Based on the difference between each truncated directed distance and each depth value in the depth space, determining the distance weight corresponding to each truncated directed distance and the depth weight corresponding to each depth value; the difference is negatively correlated with the distance weight and positively correlated with the depth weight;

[0062] Based on the truncated directed distances and their corresponding distance weights, and the depth values ​​and their corresponding depth weights, constructing an intermediate object surface model;

[0063] The intermediate object surface model is fused with the target face model to obtain an object model of the target object in the current frame.

[0064] Optionally, the fusion unit is specifically used for:

[0065] Determine, based on the distance between each voxel in the target facial model and the facial edge, a first facial weight corresponding to the truncated directed distance of each voxel in the target facial model and a second facial weight corresponding to the truncated directed distance of each voxel in the face of the intermediate object surface model;

[0066] Based on the truncated signed distances of each voxel in the target facial model and the first facial weights corresponding to each voxel, and the truncated signed distances of each voxel in the face of the intermediate object surface model and the second facial weights corresponding to each voxel, an object model of the target object in the current frame is obtained.

[0067] Optionally, the device further comprises:

[0068] The combination unit is used to obtain the object model of the target object in each frame of the target video to obtain an object model sequence for describing the position and posture changes of the target object in the target video.

[0069] An electronic device provided by an embodiment of the present application includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of any one of the above-mentioned object model construction methods.

[0070] An embodiment of the present application provides a computer-readable storage medium, which includes a computer program. When the computer program is run on an electronic device, the computer program is used to enable the electronic device to execute the steps of any one of the above-mentioned object model construction methods.

[0071] An embodiment of the present application provides a computer program product, which includes a computer program, and the computer program is stored in a computer-readable storage medium; when a processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, so that the electronic device performs the steps of any one of the above-mentioned object model construction methods.

[0072] The beneficial effects of this application are as follows:

[0073] The embodiment of the present application provides a method, device, electronic device and storage medium for constructing an object model. Since the present application detects the posture change of the target object in the target video based on the initial parameterized object model and the initial surface node model of the current frame at the same time, wherein the parameterized object model represents the joint transformation information of the object joint, and thus can well detect the posture change of the target object when the movement before and after is large and the movement is fast, and the surface node model represents the node transformation information of multiple surface nodes on the surface of the model, which has an advantage in detecting slow movements, the combination of the two enables the present application to take into account the posture change detection of fast and slow movements at the same time, and there will be no detection loss for high-speed movements, thereby improving the accuracy of modeling; in addition, the present application will also obtain the initial facial model of the target object in the current frame, and perform a separate detection on the face of the target object based on the initial facial model, accurately reproduce the geometry and expression of the face, obtain the target facial model, and finally fuse the target facial model with the initial object surface model obtained by the posture detection to obtain the final object model.

[0074] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0076] Figure 1 A schematic diagram of an application scenario of a method for constructing an object model provided in an embodiment of the present application;

[0077] Figure 2 An overall flow chart of a method for constructing an object model provided in an embodiment of the present application;

[0078] Figure 3A A schematic diagram of a parameterized object model corresponding to a certain frame provided in an embodiment of the present application;

[0079] Figure 3B A schematic diagram of a surface node model corresponding to a certain frame provided in an embodiment of the present application;

[0080] Figure 4 An RGB image under multiple viewing angles corresponding to a frame of a target video provided in an embodiment of the present application;

[0081] Figure 5 A schematic diagram of obtaining a depth map from an IR image provided in an embodiment of the present application;

[0082] Figure 6 A depth point cloud model corresponding to a certain frame of a target video provided in an embodiment of the present application;

[0083] Figure 7 A schematic diagram of constructing an initial object surface model of a target object in a current frame provided by an embodiment of the present application;

[0084] Figure 8 A logical schematic diagram of the first round of posture detection of the current frame provided in an embodiment of the present application;

[0085] Fig. 9 A schematic diagram of TSDF value conversion provided in an embodiment of the present application;

[0086] Fig.10 A schematic diagram of a target facial model provided in an embodiment of the present application;

[0087] Fig.11 A schematic diagram of an abnormal facial model provided in an embodiment of the present application;

[0088] Fig.12 A schematic diagram of an intermediate object surface model provided in an embodiment of the present application;

[0089] Fig.13 A geometric diagram of a target facial model provided in an embodiment of the present application;

[0090] Fig.14 A first facial weight schematic diagram provided in an embodiment of the present application;

[0091] Fig.15A A schematic diagram of a preliminary fusion result of an intermediate object surface model and a target face model provided in an embodiment of the present application;

[0092] Fig. 15B A schematic diagram of eliminating a seam between a target facial model and an intermediate object surface model provided in an embodiment of the present application;

[0093] Fig.16AA model construction result of other frames of a target video provided in an embodiment of the present application;

[0094] Fig. 16B A result after rendering of an object model corresponding to a partial frame provided in an embodiment of the present application;

[0095] Fig.17 An overall flow chart of another method for constructing an object model provided in an embodiment of the present application;

[0096] Fig.18 An overall logic diagram of a construction process of an object model provided in an embodiment of the present application;

[0097] Fig.19 A schematic diagram for comparing the model building results of the present application with the model building results of other solutions provided in an embodiment of the present application;

[0098] Fig. 20 A schematic diagram of the structure of a device for constructing an object model provided in an embodiment of the present application;

[0099] Fig.21 A schematic diagram of a hardware structure of an electronic device provided in an embodiment of the present application;

[0100] Fig. 22 A schematic diagram of the hardware structure of another electronic device provided for application of an embodiment of the present application. DETAILED DESCRIPTION

[0101] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the technical solution of the present application, rather than all of the embodiments. Based on the embodiments recorded in the application documents, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the technical solution of the present application.

[0102] The following is an introduction to some concepts involved in the embodiments of the present application.

[0103] Parameterized object model: an object model that can represent the transformation of each joint of an object relative to a reference frame. In this application, it is mainly divided into an initial parameterized object model and a parameterized object model to be updated; wherein, if the current frame is a specified frame, the initial parameterized object model corresponding to the current frame is obtained by adjusting the preset parameterized object model at least once; if the current frame is not a specified frame, the initial parameterized object model is obtained by transforming the target surface node model of the previous frame, and essentially reflects the joint transformation information of each joint in the previous frame relative to the reference frame; the parameterized object model to be updated in the current frame is: a model that needs to be solved based on the objective function of the parameterized object model in the current round of the current frame, which reflects the joint transformation information of each joint in the current frame relative to the reference frame, and will be continuously updated in multiple rounds of posture detection in the current frame.

[0104] Surface node model: an object model that can represent the node transformation information of multiple surface nodes on the model relative to a reference frame. In this application, it is mainly divided into an initial surface node model, an intermediate surface node model, a surface node model to be updated, and a target surface node model; wherein, if the current frame is a specified frame, the initial surface node model corresponding to the current frame is obtained by sampling the initial parameterized object model; if the current frame is not a specified frame, the initial surface node model corresponding to the current frame is the target surface node model of the previous frame, which reflects the node transformation information of each surface node in the previous frame relative to the reference frame; in the current frame, the intermediate surface node model of each round of posture detection is obtained based on the adjusted parameterized object model in this round; the surface node model to be updated in the current frame is a model that needs to be solved based on the objective function of the surface node model in the current round of the current frame; the target surface node model of the current frame is the surface node model obtained in the last round of posture detection of the current frame.

[0105] Facial model: represents the transformation information of multiple key points on the model relative to the key points of the reference frame, which can reflect the transformation of the object's face such as rotation, translation, shape, expression, etc.; in this application, it is mainly divided into an initial facial model, a target facial model and a facial model to be updated; among them, when the current frame is a non-specified frame, the initial facial model is the target facial model of the previous frame; when the current frame is a specified frame, the initial facial model is the preset facial model corresponding to the reference frame; the facial model to be updated is the model that needs to be solved based on the objective function of the facial model in the current frame; the target facial model is the facial model obtained after the facial detection of the current frame is completed.

[0106] Object surface model: a model that can represent the complete surface of an object and objects attached to it (such as clothing, accessories, etc.); in this application, it is mainly divided into an initial object surface model and an intermediate object surface model; among them, the initial object surface model is constructed based on the target surface node model of the current frame; the intermediate object surface model is obtained by projecting the initial object surface model into the depth space and then adjusting it.

[0107] Object model: The final target model corresponding to the current frame is obtained by fusing the object surface model with the facial model, which can fully display the surface details and posture of the object.

[0108] The following is a brief introduction to the design concept of the embodiment of the present application:

[0109] With the development of computer technology, people have gradually mastered 3D technology. Based on this technology, models with three-dimensional data can be constructed in a virtual three-dimensional space, so it has been widely used in games, movies, animations, product displays, model design, etc. One of the 3D modeling methods can directly build a model based on a real video, restore the characters and scenes in the video, and the dynamic detection of the motion trajectory of the human body and objects and three-dimensional reconstruction are the two most important aspects involved in this method. Its purpose is to accurately detect the motion trajectory and restore every detail in the real scene as much as possible.

[0110] In related technologies, dynamic detection of motion trajectories is often performed by obtaining each sampling point on the surface of the model, and then detecting the changes of each sampling point between each frame of the video to reflect the changes of the model in space; however, detecting motion trajectories through sampling points is only applicable to slow-moving objects or human bodies. For high-speed motion, it is easy to lose detection during detection, which further leads to inaccurate subsequent modeling. The three-dimensional reconstruction part is based on voxel grid technology, which changes the motion information of the sampling points into "surfaces" and finally generates a surface model of the human body or object. Compared with other parts, the facial details are richer, and the related technologies do not further process the facial area, which makes the detection and reproduction of the face in the modeling process relatively rough, and the construction of facial details is not accurate enough, which is easy to affect the audience's real viewing experience. In this process, the geometry and expression of the face cannot be accurately detected and reproduced.

[0111] Based on this, the embodiments of the present application provide a method, device, electronic device and storage medium for constructing an object model. Since the present application detects the posture change of the target object in the target video based on the initial parameterized object model and the initial surface node model of the current frame at the same time, the parameterized object model represents the joint transformation information of the object joint, so it can well detect the posture change of the target object when the movement before and after is large and the movement is fast, and the surface node model represents the node transformation information of multiple surface nodes on the surface of the model, which has advantages in detecting slow movements. The combination of the two allows the present application to take into account the posture change detection of fast and slow movements at the same time, and there will be no detection loss for high-speed movements, which improves the accuracy of modeling; in addition, the present application will also perform a separate detection on the face of the target object, further restore the position, shape and expression of the face, so that the geometry and expression of the face are accurately reproduced, and finally the face model is merged with the initial object surface model obtained by the posture detection to obtain the final object model.

[0112] The preferred embodiments of the present application are described below in conjunction with the drawings in the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application. In addition, the embodiments and features in the embodiments of the present application may be combined with each other if there is no conflict.

[0113] like Figure 1 As shown, it is a schematic diagram of an application scenario of an embodiment of the present application. The application scenario diagram includes two terminal devices 110 and a server 120.

[0114] In the embodiment of the present application, the terminal device 110 includes but is not limited to mobile phones, tablet computers, laptop computers, desktop computers, e-book readers, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals and other devices; the terminal device may be installed with a client related to building an object model, which may be software (such as a browser, model building software, etc.), or a web page, a small program, etc. The server 120 is a background server corresponding to the software or web page, small program, etc., or a server specifically used for model building, which is not specifically limited in the present application. The server 120 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and basic cloud computing services such as big data and artificial intelligence platforms.

[0115] It should be noted that the object model construction method in each embodiment of the present application can be executed by an electronic device, and the electronic device can be a terminal device 110 or a server 120, that is, the method can be executed by the terminal device 110 or the server 120 alone, or can be executed by the terminal device 110 and the server 120 together. For example, when the server 120 is the execution subject, for each frame in the target video, the server 120 can construct the object model of the current frame in the following manner, specifically: the server 120 obtains the initial parameterized object model and the initial surface node model corresponding to the target object in the target video in the current frame, and detects the posture of the target object in the current frame based on the initial parameterized object model and the initial surface node model to construct the initial object surface model corresponding to the target object in the current frame; in addition, the server 120 also detects the face of the target object in the current frame, further restores the face of the target object, and obtains the target face model; finally, the server 120 fuses the initial object surface model with the target face model to obtain the object model corresponding to the target object in the current frame.

[0116] On the basis of the above, for each frame in the target video, the server 120 can perform the above operation to obtain the object model corresponding to each frame, and combine the object models corresponding to each frame to obtain a model sequence, which is a four-dimensional sequence including three-dimensional space and time.

[0117] In an optional implementation, the terminal device 110 and the server 120 may communicate with each other via a communication network.

[0118] In an optional implementation, the communication network is a wired network or a wireless network.

[0119] It should be noted that Figure 1 The figure is only an example. In fact, the number of terminal devices and servers is not limited and is not specifically limited in the embodiments of the present application.

[0120] In an embodiment of the present application, when there are multiple servers, the multiple servers can be combined into a blockchain, and the servers are nodes on the blockchain; as in the method for constructing an object model disclosed in an embodiment of the present application, the model data involved can be saved on the blockchain, for example, joint transformation information, node transformation information, depth map, vertex comparison differences and other differences, depth values, truncated directed distances, various weights, various models, etc.

[0121] In addition, the embodiments of the present application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, games, digital humans, virtual humans, film and television, live broadcasts and other scenarios.

[0122] The following describes the method for constructing an object model provided by an exemplary embodiment of the present application in combination with the application scenarios described above and with reference to the accompanying drawings. It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principles of the present application, and the implementation methods of the present application are not limited in this regard.

[0123] See also Figure 2 As shown, it is an implementation flow chart of a method for constructing an object model provided in an embodiment of the present application. The specific implementation process of the method is as follows:

[0124] S201: The server obtains an initial parameterized object model and an initial surface node model corresponding to a current frame in a target video for a target object.

[0125] In the above, when the target object is a person, the parameterized object model can also be called a parameterized human body model (Skinned Multi-Person Linear, SMPL), which is a realistic human 3D model based on skinning and blending shapes, and is learned from thousands of 3D body scans. It can represent the joint transformation information of each joint of the target object in the target video relative to the reference frame. Figure 3A , which is a schematic diagram of a parameterized object model corresponding to a certain frame provided in an embodiment of the present application.

[0126] The surface node model is composed of multiple surface nodes and the lines between them, which can represent the node transformation information of multiple surface nodes on the model relative to the reference frame. Figure 3B As shown, it is a schematic diagram of a surface node model corresponding to a certain frame provided in an embodiment of the present application.

[0127] The target video contains multiple video frames, and is a multi-perspective target video captured by multiple acquisition devices located in different directions. The acquisition device can be a camera, etc., and the viewing angle can be 32-degree viewing angle, 64-degree viewing angle, etc., which is not specifically limited in this article.

[0128] An optional implementation is to obtain a target video with 32 viewing angles by capturing the target video with acquisition devices located at 32 positions, such as Figure 4 As shown, a three-primary color (Red, Green, Blue, RGB) image under multiple perspectives corresponding to a certain frame of a target video provided in an embodiment of the present application, and an RGB image can be obtained for each perspective of each frame of the video, that is, each frame corresponds to 32 RGB images, and each RGB image corresponds to one perspective.

[0129] It should be noted that Figure 4 The RGB images in the figure should be color images. Due to the restrictions on the requirements of the attached figures, color images cannot be displayed. Figure 4The image displayed in appears to be grayscale, but it should be in color.

[0130] In addition, infrared acquisition devices located in 64 directions can be used to capture target videos with 64 viewing angles, and the 64 infrared acquisition devices can be divided into 32 groups. The two infrared acquisition devices in each group are parallel, and the shooting directions and viewing angles can be approximately regarded as the same; the viewing angle of each group of infrared acquisition devices is similar to the viewing angle of the acquisition device that captures RGB images in a certain direction, and can be approximately regarded as the same. The video captured by two infrared acquisition devices is also a group, that is, each frame of the video corresponds to 32 groups of infrared radiation (IR) images, and each group of IR images can be used to obtain a depth map of the corresponding viewing angle, and the depth map is used to subsequently construct a depth point cloud model.

[0131] like Figure 5 As shown, it is a schematic diagram of obtaining a depth map from an IR image provided by an embodiment of the present application. The upper part is an IR image, with a total of 32 groups, 64 images, each corresponding to a viewing angle. The viewing angles of the two IR images in each group are parallel. The function of each group of IR images is to obtain a depth map corresponding to a viewing angle, that is, Figure 5 The black image in the lower half can finally get a depth map of 32 viewing angles for each frame.

[0132] The purpose of the RGB image is to provide color information for subsequent texture mapping, and it can also be used to assist in finding matching points and improve the accuracy of finding matching points, but it can be used as a non-essential item in this application.

[0133] It should be noted that the number of viewing angles and the method of obtaining a depth map mentioned above are only examples, and other numbers of viewing angles and methods of obtaining a depth map are applicable to this application, and this application does not make any specific limitations.

[0134] It can be understood that in the specific implementation of the present application, relevant data such as the target video is involved. For example, when modeling the target object in the target video, it is necessary to obtain the appearance information, human body data, etc. of the target object. When the above embodiments of the present application are applied to specific products or technologies, it is necessary to obtain the permission or consent of the object, and the collection, use and processing of relevant data need to comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0135] After obtaining the depth map, a model can be constructed based on the multi-view depth map corresponding to the current frame to obtain a depth point cloud model of the target object in the current frame for subsequent posture detection. In the embodiment of the present application, the depth point cloud model of the current frame is constructed based on the multi-view depth map corresponding to the current frame. Figure 6As shown, a depth point cloud model corresponding to two frames before and after a target video provided in an embodiment of the present application can be seen that there are many missing parts on the surface of the depth point cloud model. Therefore, the depth point cloud model cannot be directly used to construct a model for the target object.

[0136] The reference frame is the 0th frame of the target video (only in the literal sense, it can also be recorded as the 1st frame, and the following description will be based on the reference frame being the 0th frame of the target video). In essence, the target object in the reference frame is generally in a preset initial posture. At the beginning of the video recording, that is, at the 0th frame, the target object needs to pose a preset initial posture to facilitate alignment with the existing initial preset parameterized object model. For example, assuming that the target object is a person, the posture of the initial preset parameterized object model, that is, the initial posture, can be a person standing upright, with both hands stretched out to the left and right, and the two feet apart. In the 0th frame of the target video, the target object also poses upright, with both hands stretched out to the left and right, and the two feet apart. For another example, assuming that the target object is a bird, the posture of the initial preset parameterized object model can be standing with wings spread. In the 0th frame of the target video, the target object also stands with wings spread. The following description will be mainly based on the target object being a person.

[0137] Taking a specific scenario as an example, suppose you need to add a dance video to the 3D animation. Before building the model, the target video has been recorded, and each frame in the target video corresponds to 32 depth maps from different perspectives. The dancer in the target video is the target object.

[0138] Assuming that the current frame is the f-th frame, f>1, the server first obtains the initial parameterized object model and the initial surface node model corresponding to the current frame.

[0139] S202: The server performs posture detection on the target object in the current frame based on the initial parameterized object model and the initial surface node model, and constructs an initial object surface model of the target object in the current frame.

[0140] In the above, the method of obtaining the initial parameterized object model and the initial surface node model varies depending on whether the current frame is a specified frame. If the current frame is not a specified frame, the initial parameterized object model is obtained by transforming the target surface node model of the previous frame, and the initial surface node model is the target surface node model of the previous frame; if the current frame is a specified frame, the initial parameterized object model is obtained by adjusting the preset parameterized object model at least once, and the initial surface node model is obtained by sampling the initial parameterized object model; the preset parameterized object model is constructed based on the reference frame.

[0141] The designated frame mentioned above is the next frame of the reference frame. If the reference frame is frame 0, the designated frame is frame 1. If the reference frame is frame 1, the designated frame is frame 2. The following description mainly takes the reference frame as frame 0. The "target surface node model of the previous frame" mentioned above is obtained by the server performing multiple rounds of posture detection on the target object in the previous frame. The specific implementation process is described in detail later.

[0142] The preset parameterized object model is the "existing initial preset parameterized object model" mentioned above. The posture of the initial preset parameterized object model is very similar to the posture of the target object in the reference frame and can be roughly aligned, but further adjustment is required. The adjustment process is at least once. The specific number of times is not specifically limited in this application and can be set by the model builder according to experience.

[0143] Each time adjustment is made, an optional implementation is that the server adjusts the preset parameterized object model to be updated based on the difference between the current latest preset parameterized object model and the depth point cloud model corresponding to the reference frame, and the difference between the preset parameterized object model and the preset parameterized object model to be updated.

[0144] In the above, if the current adjustment is the first adjustment, the current latest preset parametric object model is the initial preset parametric object model. If it is not the first adjustment, the current latest preset parametric object model is the adjusted preset parametric object model obtained after the last adjustment.

[0145] An optional objective function corresponding to the above process is:

[0146]

[0147] In the above, The variable to be solved, also known as joint transformation information, records the rotation information of each joint and the translation of the root joint in the global space of the preset parametric object model to be solved relative to the initial preset parametric object model in this adjustment. The root joint is the core joint in the parametric object model. Each other joint is directly or indirectly connected to the root joint through its own path. The translation of the root joint in the global space can be further reflected as the translation of the model in the global space. d , r , p , k With λ pri are the preset weights corresponding to each energy item. Each energy item is described in detail below.

[0148] The formula is:

[0149]

[0150] In the above, v m ′ Presets each vertex in the parameterized object model after the last adjustment (if this is the first adjustment, v m ′ is the initial preset parametric object model), v t ′ In the depth point cloud model corresponding to the reference frame, m ′ The matching corresponding points can be obtained through projection matching; p ′ is the set of corresponding point pairs obtained by projection matching, that is, v m ′ With v t ′ The corresponding point pair set of Then v t ′ The corresponding normal vector.

[0151] The formula is:

[0152]

[0153] In the above, J proj2d ′ is the projection position of each joint of the preset parameterized object model to be solved at different viewing angles in this adjustment, J lmk2d ′ is the projection position of the corresponding joint under the corresponding viewpoint estimated by the pose estimation (OpenPose) algorithm. The algorithm is based on convolutional neural network and supervised learning, and uses the open source library developed by the convolutional framework for feature extraction (Convolutional Architecture for Fast Feature Embedding, caffe).

[0154] The formula is:

[0155]

[0156] In the above, is the variable to be solved, is the joint transformation information of the preset parameterized object model obtained at the end of the last adjustment; if this adjustment is the first adjustment, then It is the joint transformation information of the initial preset parametric object model; the joint transformation information corresponding to the initial preset parametric object model is generally directly set to 0, or manually set to other values ​​by the model builder according to actual needs.

[0157] The formula is:

[0158]

[0159] In the above, J defk+1 ′ is the spatial position of each joint of the preset parameterized object model to be solved in this adjustment, J last ′ is the spatial position of each joint of the preset parameterized object model obtained at the end of the last adjustment; if this adjustment is the first adjustment, J last ′ The spatial positions of the joints of the initial preset parameterized object model.

[0160] The formula is:

[0161]

[0162] In the above, is the variable to be solved, The server selects the closest pose parameter from all prior pose parameters to the current A set of parameters, the prior posture parameters represent the joint transformation information that conforms to the preset joint posture.

[0163] After multiple adjustments, the parameterized object model corresponding to the reference frame is obtained, and the surface node model corresponding to the reference frame is obtained by sampling the surface nodes on the parameterized object model corresponding to the reference frame. In addition, taking the human body as an example, since the parameterized object model cannot reflect the external accessories such as clothes, accessories, backpacks, etc. of the human body, in order to obtain the surface node model corresponding to the complete reference frame, in addition to sampling the surface nodes on the parameterized object model, it is also necessary to directly obtain the truncated signed distance function (TSDF) based on the multi-angle depth map corresponding to the reference frame, and then construct the object surface model corresponding to the reference frame based on the TSDF value, and then extract some surface nodes from the object surface model corresponding to the reference frame to further obtain the surface nodes of items such as clothing.

[0164] In the above, the truncated signed distance function measures the orthogonal distance from a given point x in space to the boundary of the set Ω, and the sign of the function is determined by whether x is inside Ω. The function has a positive value at a point x outside Ω, decreases as x approaches the boundary of Ω, and takes a negative value inside Ω.

[0165] After obtaining the initial parameterized object model and the initial surface node model of the current frame, the target object in the current frame can be subjected to pose detection based on the initial parameterized object model and the initial surface node model of the current frame, and finally the initial object surface model of the target object in the current frame is obtained. In the present application, a method of alternately using a parameterized object model and a surface node model to perform pose detection on the target object in the current frame is proposed. After the pose detection of one model is completed, its detection result will be transmitted to another model and play a role in the pose detection process of the other model. This process can effectively improve the robustness and convergence efficiency of the detection process.

[0166] An optional implementation is that the server uses the initial parameterized object model and the initial surface node model, as well as the depth point cloud model, to perform multiple rounds of pose detection on the target object in the current frame, and uses the surface node model obtained from the last round of pose detection as the target surface node model of the target object in the current frame. This process can simultaneously utilize the respective advantages of the parameterized object model and the surface node model, effectively monitor fast and slow pose changes, and make model construction more accurate.

[0167] Among them, the initial parameterized object model and the initial surface node model are updated alternately during each round of posture detection; according to the node transformation information of multiple surface nodes in the target surface node model of the current frame, the initial object surface model of the target object in the current frame is constructed.

[0168] Assume that the current frame is the tth frame, such as Figure 7 As shown, it is a logical schematic diagram of constructing an initial object surface model of a target object in the current frame provided by an embodiment of the present application. First, the initial parameterized object model and the initial surface node model of the t-1 frame are obtained. Then, multiple pose detections are performed based on the initial parameterized object model and the initial surface node model of the t-1 frame. After the pose detection of the parameterized object model is completed, the detection result is transmitted to the surface node model, and then the surface node model performs pose detection. After the pose detection of the surface node model is completed, the detection result is transmitted to the parameterized object model again, ..., and so on. The two models are continuously adjusted and updated to finally obtain the target surface node model. The server constructs the initial object surface model of the t frame based on the target surface node model of the t frame. Figure 7 The front and back sides of the initial object surface model at frame t are shown.

[0169] In the above, the specific process of each round of posture detection is as follows:

[0170] The server adjusts the parameterized object model to be updated based on the difference between the current latest parameterized object model and the depth point cloud model, as well as the difference between the parameterized object model to be updated; and based on the adjusted parameterized object model, adjusts the current latest surface node model to obtain an intermediate surface node model; then, the server adjusts the surface node model to be updated based on the difference between the intermediate surface node model and the depth point cloud model, as well as the difference between the surface node model to be updated; finally, the server adjusts the adjusted parameterized object model again based on the adjusted surface node model.

[0171] In this process, the adjusted result of the parameterized object model will be passed to the surface node model. The surface node model will perform pose detection based on the result and obtain an adjusted surface node model. The adjusted result will also be passed to the parameterized object model. The parameterized object model will perform pose detection based on this. This process repeats the process to achieve alternating updates of the two.

[0172] In the above, if the current round of pose detection is the first round of pose detection in the current frame, then in the step of "based on the difference between the current latest parameterized object model and the depth point cloud model", the current latest parameterized object model is the initial parameterized object model, and in the step of "based on the adjusted parameterized object model, adjust the current latest surface node model", the current latest surface node model is the initial surface node model. In the subsequent rounds of pose detection, the current latest parameterized object model is obtained based on the latest adjusted surface node model in the previous round, and the current latest surface node model is the latest adjusted surface node model in the previous round.

[0173] Still assuming that the current frame is the tth frame, such as Figure 8 As shown, it is a logical schematic diagram of the first round of pose detection of the current frame provided by an embodiment of the present application. The initial parameterized object model corresponding to the t-th frame is recorded as SMPL1, and the initial surface node model corresponding to the t-th frame is recorded as SUR1. Each round of pose detection first performs pose detection and update on the parameterized object model, and then performs pose detection and update on the surface node model. The specific process is as follows.

[0174] In the first round of detection, the current latest parameterized object model is the initial parameterized object model SMPL1 corresponding to the t-th frame. The server obtains the difference between SMPL1 and the depth point cloud model corresponding to the t-th frame, and obtains the difference between SMPL1 and the parameterized object model to be updated corresponding to the first round of the t-th frame. Based on the above two differences, the parameterized object model to be updated is adjusted to obtain the adjusted parameterized object model, recorded as SMPL2. The parameterized object model to be updated is actually a solution result of the parameterized object model corresponding to the current frame and the current round that needs to be obtained through pose detection. SMPL2 is actually the solution result of the parameterized object model obtained by the pose detection of the current frame and the current round.

[0175] After obtaining SMPL2, the server adjusts the latest surface node model, that is, the initial surface node model SUR1 corresponding to the t-th frame, based on SMPL2 to obtain the intermediate surface node model SUR2; the specific implementation method of this process is as follows: a part of the vertices on the parameterized object model corresponds to a part of the surface nodes on the surface node model, so the position of the corresponding surface node on SUR1 can be adjusted according to the position of some vertices on SMPL2; for surface nodes that have no correspondence and are located on the human body, they can be adjusted through the corresponding surface nodes around them, and for surface nodes on accessories such as clothing, the surface nodes on SUR1 are used.

[0176] Afterwards, the server adjusts the surface node model to be updated based on the difference between SUR2 and the depth point cloud model corresponding to the current frame, and the difference with the surface node model to be updated corresponding to the first round of the t-th frame, to obtain the adjusted surface node model SUR3. Similarly, the surface node model to be updated is actually a solution result of the surface node model corresponding to the current frame and the current round that needs to be obtained through pose detection; SUR3 is actually the solution result of the surface node model obtained by pose detection of the current frame and the current round.

[0177] Afterwards, the server adjusts the adjusted parameterized object model, namely SMPL2, based on SUR3 to obtain SMPL3. Afterwards, the server enters the second round of pose detection.

[0178] In the second round of pose detection, the latest parameterized object model becomes SMPL3. The server obtains the difference between SMPL3 and the depth point cloud model corresponding to the t-th frame, and obtains the difference between SMPL3 and the parameterized object model to be updated corresponding to the second round of the t-th frame. Based on the above two differences, the parameterized object model to be updated is adjusted to obtain the adjusted parameterized object model, which is recorded as SMPL4.

[0179] Afterwards, the server adjusts the current latest surface node model, that is, SUR3, based on SMPL4 to obtain the intermediate surface node model SUR4; the server adjusts the surface node model to be updated based on the difference between SUR4 and the depth point cloud model corresponding to the current frame, and the difference with the surface node model to be updated corresponding to the second round of the t-th frame, to obtain the adjusted surface node model SUR5.

[0180] If the preset pose detection is two rounds, SUR5 is the target surface node model corresponding to the t-th frame. The server can adjust SMPL4 based on SUR5 to obtain the initial parameterized object model corresponding to the t+1-th frame, and directly use SUR5 as the initial surface node model corresponding to the t+1-th frame.

[0181] If the preset pose detection is greater than two rounds, the server adjusts SMPL4 based on SUR5 to obtain SMPL5, and starts the third round of pose detection for the tth frame, that is, continues to repeat the pose detection process.

[0182] Assuming that the preset pose detection is two rounds, the server constructs the initial object surface model of the tth frame according to SUR5.

[0183] It should be noted that the number of rounds of posture adjustment can be two to three times, or one time, or can be manually set by the model builder based on experience, and this application does not make specific limitations.

[0184] After introducing the process of alternating updating of parameterized object models and surface nodes in this application, the specific process of obtaining the difference between the models and adjusting the surface node model to be updated in the above alternating update is analyzed. This process is essentially to solve the parameterized object model to be updated and the surface node model to be updated corresponding to the current frame and the current round.

[0185] First, in the process of posture detection and updating of the parameterized object model, an optional implementation method is that the server needs to obtain the first vertex comparison difference between each vertex in the current latest parameterized object model and the corresponding point in the deep point cloud model; and obtain the joint transformation difference between the joint transformation information of the current latest parameterized object model and the parameterized object model to be updated, the joint point position difference between the positions of each joint point, and the first vertex position difference between the positions of each vertex; finally, based on the first vertex comparison difference, joint transformation difference, joint point position difference and first vertex position difference, the parameterized object model to be updated is adjusted. This process can effectively adjust the parameterized object model according to the accurate deep point cloud model, making the model construction more accurate.

[0186] An optional objective function corresponding to the above process is as follows:

[0187]

[0188] in, is the variable to be solved, which can also be called joint transformation information. It records the rotation information of each joint and the translation of the root joint in the global space of the parameterized object model to be updated relative to the parameterized object model corresponding to the reference frame in this adjustment. The translation of the root joint in the global space can be further reflected as the translation of the model in the global space. d , r , p With λ b is the preset weight corresponding to each difference.

[0189] is the first vertex contrast difference, and its specific formula is:

[0190]

[0191] In the above, v m For each vertex in the current latest parameterized object model, v t In the depth point cloud model corresponding to the current frame, m The matching corresponding points can be obtained through projection matching; p is the set of corresponding point pairs obtained through projection matching, that is, v m With v t The corresponding point pair set of Then v t The corresponding normal vector. Taking the first round of detection of the tth frame as an example, v m It should be each vertex in SMPL1. If it is the second round of detection in the tth frame, v m Should be individual vertices in SMPL3.

[0192] is the joint transformation difference, and its specific formula is:

[0193]

[0194] is the joint transformation information of the parameterized object model to be updated, The joint transformation information of the current latest parametric object model.

[0195] is the difference in joint position, and its specific formula is:

[0196]

[0197] J defk+1is the spatial position of each joint of the parameterized object model to be updated, that is, the position of the joint point, J last The current state-of-the-art parametric object model corresponds to the spatial position of the joint.

[0198] is the first vertex position difference, and its specific formula is:

[0199]

[0200] v mi To record the position of the i-th vertex of the parameterized object model to be updated, Records the position of the vertex in the current latest parametric object model.

[0201] In addition, in the process of detecting and updating the pose of the parameterized object model, there is an optional implementation method. After obtaining the first vertex comparison difference, joint transformation difference, joint point position difference and first vertex position difference, the server can continue to obtain the joint transformation information of the latest parameterized object model and the joint posture difference between the prior posture parameters; and obtain the projection position of each joint point in the parameterized object model to be updated under multiple perspectives, and the joint point estimated difference between the projection position obtained based on the posture estimation; finally, the server adjusts the parameterized object model to be updated based on the first vertex comparison difference, joint transformation difference, joint point position difference, joint posture difference, joint point estimated difference and first vertex position difference. This process adds joint posture difference and joint point estimated difference to further detect the pose, improve the accuracy of detection, and avoid the model from having unreasonable postures.

[0202] In the above, the priori posture parameter represents the joint transformation information that conforms to the preset joint posture. The objective function corresponding to the above implementation is as follows:

[0203]

[0204] Among them, λ k Estimating differences for joint points The corresponding weight, λ pri The joint posture difference The corresponding weight.

[0205] Joint point estimation difference The specific formula is:

[0206]

[0207] J proj2d J is the projection position of each joint in the parameterized object model to be updated under different viewing angles (such as the 32 viewing angles mentioned in S201),lmk2d It is the projection position of the corresponding joint estimated by the OpenPose algorithm under the corresponding viewing angle.

[0208] Differences in joint posture The specific formula is:

[0209]

[0210] in, The server selects the closest pose parameter from all prior pose parameters to the current A set of parameters, the prior posture parameters represent the joint transformation information that conforms to the preset joint posture. The range of motion of each joint of the human body has a normal value range. For example, the human head cannot rotate 360 ​​degrees. The prior posture parameters are to avoid unreasonable postures.

[0211] After the pose detection and update of the parameterized object model are completed, the server will adjust the current latest surface node model based on the adjusted parameterized object model to obtain an intermediate surface node model. After that, the server adjusts the surface node model to be updated based on the difference between the intermediate surface node model and the depth point cloud model, as well as the difference with the surface node model to be updated. For this process, an optional implementation method proposed in the present application is that the server obtains the surface vertex comparison difference between each surface vertex in the intermediate surface node model and the corresponding point in the depth point cloud model; and obtains the node transformation difference between the node transformation information in the intermediate surface node model and the surface node model to be updated, and the surface node position difference between the positions of each surface node; finally, the server adjusts the surface node model to be updated based on the surface vertex comparison difference, the node transformation difference and the surface node position difference.

[0212] An optional objective function corresponding to the above process is as follows:

[0213]

[0214] in, is the variable to be solved, which can also be called node transformation information, which records the rotation and translation information of each surface node relative to the surface node model corresponding to the reference frame in this adjustment. egd , egs With λ egb is the preset weight corresponding to each difference. This process can effectively adjust the surface node model according to the accurate depth point cloud model, making the model construction more accurate.

[0215] is the surface vertex contrast difference, and its specific formula is:

[0216]

[0217] Among them, v ni For each surface vertex in the intermediate surface node model, v ti In the depth point cloud model corresponding to the current frame, ni Matching corresponding points, n ti v ni The corresponding normal vector. Taking the first round of detection in the tth frame as an example, v ni It should be each surface vertex in SUR2. If it is the second round of detection in the tth frame, v ni Should be individual surface vertices in SUR4.

[0218] There is a many-to-one relationship between surface nodes and surface vertices. If the positions of all surface nodes in a group of surface nodes corresponding to a surface vertex can be determined, the position of the corresponding surface vertex can be determined based on the positions of the surface nodes in this group.

[0219] is the node transformation difference, and its specific formula is:

[0220]

[0221] in, The node transformation information of the surface node model to be updated, It is the node transformation information of the mid-surface node model.

[0222] is the surface node position difference, and its specific formula is:

[0223]

[0224] Where M is the number of all surface nodes, v hj is the position of the jth surface node in the surface node model to be updated, is the position of the jth surface node in the intermediate surface node model.

[0225] In addition, during the posture detection and update process of the surface node model, there is an optional implementation method. After obtaining the surface vertex comparison differences, node transformation differences and surface node position differences, the server can continue to obtain the positional relationship between each surface node and its adjacent surface nodes in the surface node model to be updated; finally, based on the surface vertex comparison differences, node transformation differences, surface node position differences and positional relationships, the surface node model to be updated is adjusted.

[0226] This process adds a positional relationship, which is to determine the positional relationship between a node and its adjacent nodes. For example, when the adjacent nodes of a node are moving to the left, the node is likely to move to the left as well. Adding this formula can make the model construction more accurate.

[0227] An optional objective function corresponding to the above implementation is as follows:

[0228]

[0229] Among them, λ egr For position relationship The corresponding weight.

[0230] Positional Relationship The specific formula is:

[0231]

[0232] Where M is the number of all surface nodes, H(j) records all neighbor sequences of the j-th surface node, that is, all surface nodes connected to the j-th surface node, and v hk 、v hj The positions of the kth and jth surface nodes in the surface node model to be updated are recorded. is the rotation parameter corresponding to the jth surface node, and are the translation parameters corresponding to the jth surface node and the kth surface node, α jk The dependency weight between the jth surface node and the kth surface node is recorded, indicating that the degree of mutual influence between the two will decrease with increasing distance.

[0233] Based on the above processes, a final surface node model of the current frame can be obtained in the last round of posture detection, that is, the target surface node model corresponding to the current frame.

[0234] In addition, in the above-mentioned method of alternatingly updating the parameterized object model and the surface node model, the parameterized object model can be replaced by other methods or models, such as obtaining a model using deep learning and alternatingly updating it with the surface node model.

[0235] After that, the server can construct the initial object surface model process of the current frame based on the target surface node model, which is mainly completed through the voxel space. The voxel space is to divide the entire space into small blocks, each of which is a voxel. For example, the space is divided into 512×512×512 voxels, and each voxel corresponds to a TSDF value. The TSDF value indicates whether the voxel is inside or outside the model and how far away from the model surface it is. If it is inside the model, the TSDF value of the voxel is negative, and the farther away from the model surface, the smaller the value; if it is outside the model, the TSDF value of the voxel is positive, and the farther away from the model surface, the larger the value. In other words, as long as all voxels with TSDF values ​​of 0 or extremely close to 0 are found, the model surface can be constructed. If there are two voxels with extremely close positions, one positive and one negative TSDF values, and both are extremely close to 0, the one closer to 0 is selected to prevent the interlacing of voxels.

[0236] The server needs to transform the information of the reference frame and the current frame space through constraints such as Euclidean distance or geodesic distance, such as Fig. 9 As shown, a schematic diagram of TSDF value conversion provided by an embodiment of the present application, the upper half of the figure is the voxel space corresponding to the reference frame (for ease of viewing, the figure only shows the voxels near the model surface, and does not show all the voxels in the voxel space), and the process of constructing the initial object surface model corresponding to the reference frame based on the TSDF value in the voxel. The server updates the dependency relationship between each voxel and the surrounding surface nodes in the reference frame space (the closer the distance, the greater the dependency) based on the skeleton skinning animation algorithm (Linear Blending Skinning, LBS) and the node transformation information of each surface node of the target surface node model, and converts the TSDF value stored in each voxel in the reference frame space into the TSDF value stored in each voxel of the current frame. LBS is a method of moving vertices in a single mesh through multiple transformations. Each transformation brings the vertex into the local space of a given "bone", and then transforms the vertex to a new position through the "binding matrix" of the "bone".

[0237] The server can then construct an initial object surface model for the current frame based on the converted voxels. Fig. 9 The lower part is the voxel space corresponding to the current frame, and the process of constructing the initial object surface model corresponding to the current frame based on the TSDF value in the voxel. The specific processing method of this process is as follows:

[0238] The server re-establishes the voxel space in the current frame space coordinate system, transforms the TSDF value obtained above to the current frame space, and then projects it to each depth map perspective, and updates the TSDF value recorded in each voxel.

[0239] Before and after the TSDF value of a voxel is updated, the position of the voxel does not change, only its TSDF value changes. However, during the TSDF value update process, the voxel will briefly change position following the model as the model action changes. This process can be used for collision detection and other processing. Specifically, when voxels in different regions are close to the same position, the model surfaces corresponding to the two voxels will not be fused. For example, when a human body model crosses its left and right hands, the voxels of the left and right hands belong to different regions. Even if they are close to the same position, the model surfaces of the left and right hands will not be fused.

[0240] Finally, after the voxel information is updated, the server will re-extract the model surface in the new voxel space under the current frame space coordinate system to construct the initial object surface model of the current frame.

[0241] In addition, the TSDF value also corresponds to a weight, which can reflect the current frame number. When the TSDF value is updated, the corresponding weight also needs to be updated, and the weight has a preset upper limit. Assuming the upper limit is 50, the weight corresponding to the TSDF value will not exceed 50. If it exceeds 50, it will remain at 50 and will not increase.

[0242] It should be noted that the upper limit of 50 mentioned above is only an example and is not specifically limited in this application.

[0243] Following the assumption in S201, since the current frame is the f-th frame, f>1, the initial parameterized object model corresponding to the f-th frame obtained by the server is obtained by transforming the target surface node model in the f-1-th frame, and the initial surface node model corresponding to the f-th frame is the target surface node model of the f-1-th frame. The initial parameterized object model corresponding to the f-th frame is recorded as A1, and the initial surface node model corresponding to the f-th frame is recorded as B1.

[0244] In the first round of pose detection in the f-th frame, the server first performs pose detection based on A1, and obtains the parameterized object model A2 based on the detection result. Then, the server adjusts B1 based on A2 to obtain the intermediate surface node model B2. Then, the intermediate surface node model B2 performs pose detection to obtain the surface node model B3, and A2 is adjusted based on B3 to obtain the parameterized object model A3.

[0245] Then, the second round of pose detection for the f-th frame begins. The server first performs pose detection based on A3 to obtain a parameterized object model A4. Then, the server adjusts B3 based on A4 to obtain an intermediate surface node model B4. Then, the intermediate surface node model B4 performs pose detection to obtain a surface node model B5. Assuming that the preset number of pose detection rounds is 2, B5 is the target surface node model corresponding to the f-th frame, and the server constructs the initial object surface model corresponding to the f-th frame based on B5.

[0246] S203: The server obtains an initial facial model corresponding to the current frame, and performs facial detection on the target object in the current frame based on the initial facial model to obtain a target facial model.

[0247] In the above, the facial model is also called the Xface model, which is a realistic 3D model of a human face, based on skinning and blending shapes, and learned from thousands of 3D body scans. The facial model represents the key point transformation information of multiple key points on the model relative to the reference frame; the key point transformation information includes: the facial position transformation information, facial morphology transformation information, and facial expression transformation information of the target object in the current frame relative to the reference frame.

[0248] First of all, to reproduce facial details, you need to find the most appropriate multiple perspectives so that the combination of multiple perspectives can fully reflect the facial situation. This process can be completed by using OpenPose to detect multiple facial key points at each perspective of the current frame (such as the 32 perspectives mentioned above). The number of key points can be set according to actual needs. For example, 68 facial key points are set. OpenPose will also output the confidence of each facial key point. The confidence reflects OpenPose's score for the accuracy of the position of the key point. The server combines the confidence with the orientation of the face on the SMPL model corresponding to the current frame, and selects multiple optimal perspectives that can fully construct the entire face after combining with each other. The number of perspectives is not specifically limited in this application, and can generally be 6-10.

[0249] In addition, the specific number of facial key points mentioned above is only an example and is not specifically limited in this application.

[0250] After that, the server starts to perform facial detection and obtains the target facial model. An optional implementation is that the server obtains the deep point cloud model of the target object and the initial facial model corresponding to the current frame. After that, the server adjusts the facial model to be updated based on the difference between the facial model to be updated and the facial part in the deep point cloud model, as well as the difference between the facial model to be updated and the initial facial model, to obtain the target facial model. Fig.10 , which is a schematic diagram of a face detection result provided in an embodiment of the present application.

[0251] In the above, the depth point cloud model is constructed based on the multi-view depth map corresponding to the current frame; if the current frame is a specified frame, the initial facial model is the preset facial model corresponding to the reference frame; if the current frame is not a specified frame, the initial facial model is the target facial model of the previous frame; the preset facial model corresponding to the reference frame is generally an average face under natural expressions, and its key point transformation information is set to 0.

[0252] That is, the face detection of the current frame is performed based on the face detection result of the previous frame. Next, the acquisition of each difference in the above face detection process is specifically analyzed. This process is actually solving the objective function corresponding to face detection.

[0253] An optional implementation is that the server obtains the second vertex comparison difference between each vertex in the facial model to be updated and the corresponding point on the deep point cloud model; and obtains the key point transformation difference between the key point transformation information in the facial model to be updated and the initial facial model; then, the server adjusts the facial model to be updated based on the second vertex comparison difference and the key point transformation difference to obtain the target facial model. This process adjusts the facial model to be updated based on the accurate deep point cloud model, making the facial model construction more accurate.

[0254] In this implementation, an optional objective function corresponding to face detection is as follows:

[0255]

[0256] in, It is the variable to be solved, also known as the key point transformation information, which records the facial position transformation information of the facial model to be updated in the current frame relative to the preset facial model corresponding to the reference frame, as well as the facial morphology transformation information and facial expression transformation information. Facial position information is also the facial space transformation information, which mainly reflects its rotation and translation. fd With λ fr is the weight corresponding to each item. The specific formula for each item is as follows:

[0257]

[0258]

[0259] In the above, is the second vertex contrast difference, is the key point transformation difference, is the position of each vertex in the facial model to be updated in the current frame, v tar In the depth point cloud model corresponding to the current frame, The position of the matching corresponding point, n tar vtar The corresponding normal vector is, is the facial morphology transformation information and facial expression transformation information corresponding to the initial facial model. Each facial key point in the facial model is a part of the vertices selected from each vertex in the facial model. When each vertex in a facial model is determined, its facial key points are also determined.

[0260] Through the above target formula, the key point transformation difference of the facial model to be updated corresponding to the current frame can be solved, that is, the target surface node model corresponding to the current frame can be obtained.

[0261] In addition, there is another optional implementation method, the server can further obtain the key point transformation information of the facial model to be updated, and the facial motion difference between the key point transformation information and the prior key point transformation information; and obtain the projection position of the facial key points of the facial model to be updated under multiple perspectives, and the estimated facial key point difference between the projection position of the facial key points under the corresponding perspectives obtained based on the posture estimation; finally, the server adjusts the facial model to be updated based on the second vertex comparison difference, the key point transformation difference, the facial motion difference and the facial key point estimated difference to obtain the target facial model. This process adds the facial motion difference and the facial key point estimated difference, which can control the facial morphology and expression of the model within a reasonable and normal range, avoid unnatural facial morphology and expression, and further improve the accuracy of facial model construction based on the facial key point estimated difference.

[0262] In the above, the prior key point transformation information includes prior expression information and prior morphological information; an optional objective function corresponding to the above process is:

[0263]

[0264] Among them, λ fk Estimating differences for facial landmarks The corresponding weight, λ fpri For facial movement differences The corresponding weight.

[0265] The specific formula is:

[0266]

[0267] in, is the projection position of each key point in the facial model to be updated in the current frame under different viewing angles, The projection positions of the corresponding key points estimated by the OpenPose algorithm under the corresponding viewing angles. In addition, the different viewing angles here are multiple optimal viewing angles selected previously.

[0268] The specific formula is:

[0269]

[0270] in, It is the prior key point transformation information, including prior expression information and prior morphology information. The prior key point transformation information can represent the normal morphology and expressions that the face can make in reality. The purpose of this item is to prevent a part of the facial model from exceeding its normal area and to control the facial morphology and expression of the model within a reasonable and normal range, such as Fig.11 As shown, it is a schematic diagram of an abnormal facial model provided in an embodiment of the present application, wherein in the left half of the picture, the position of the model's mouth is obviously abnormal, while in the right half, the mouth shape is abnormal.

[0271] The prior key point transformation information is the mean of the data contained in a large number of expressions in the preset expression library. For example, the expression library contains 100 objects, each object contains 50 expressions, and assuming that the expression data is 30-dimensional, a 5000×30 matrix is ​​obtained. The prior key point transformation information can be obtained by taking the mean of the data in each dimension.

[0272] In addition, in addition to using the above-mentioned solutions, other methods such as data-driven can also be used to build a facial model.

[0273] Following the assumption in S202, after obtaining the initial object surface model corresponding to the f-th frame, the server needs to further restore the face of the target object in the f-th frame. The server first obtains the initial object surface model corresponding to the f-th frame, that is, the target facial model in the f-1-th frame. Then, the server detects multiple facial key points under various viewing angles of the f-th frame through OpenPose, and obtains the confidence of each facial key point. Then, the server combines the confidence with the orientation of the face on the SMPL model corresponding to the f-th frame, and selects 6 optimal viewing angles to solve the facial model objective function.

[0274] Afterwards, the server adjusts the facial model to be updated based on the differences between each vertex in the facial model to be updated corresponding to the f-th frame and the corresponding points on the depth point cloud model; the differences between the key point transformation information of the facial model to be updated and the initial facial model; the differences between the key point transformation information of the facial model to be updated and the prior key point transformation information; and the differences between the projection positions of the facial key points of the facial model to be updated under 6 viewing angles and the projection positions of the facial key points obtained based on posture estimation under the corresponding viewing angles, to obtain the target facial model corresponding to the f-th frame.

[0275] S204: The server fuses the initial object surface model with the target face model to obtain an object model of the target object in the current frame.

[0276] The server can fuse the initial object surface model with the target facial model to obtain the object model. In this process, the model can also be based on the depth point cloud model to further reproduce the target object in the target video, further improving the accuracy of the constructed model.

[0277] One implementation method of the above method is that the server obtains a deep point cloud model of the target object, and fuses the deep point cloud model, the initial object surface model and the target facial model to obtain an object model of the target object in the current frame.

[0278] Compared with the object surface model, the model information reproduced by the deep point cloud model is more accurate and has more details. However, due to its own reasons, the deep point cloud model has many holes and omissions, so the server first merges the deep point cloud model with the object surface model to make up for the omissions. However, since it is more complicated to directly merge the two model surfaces, in this application, the above process can be completed through the depth space, that is, the depth map image space.

[0279] An optional implementation is that the server projects the initial object surface model into the depth space and regains the truncated signed distance of each voxel in the space; then, the server determines the distance weight corresponding to each truncated signed distance and the depth weight corresponding to each depth value based on the difference between each truncated signed distance and each depth value in the depth space; and constructs an intermediate object surface model based on each truncated directed distance and its corresponding distance weight, and each depth value and its corresponding depth weight; finally, the server fuses the intermediate object surface model with the target facial model to obtain the object model of the target object in the current frame.

[0280] In the above, the depth space is determined based on the depth point cloud model, which is used to represent the depth value of each point in the depth point cloud model. It can also be obtained by spatially constructing the multi-view depth map corresponding to the current frame, which can replace the depth point cloud model to complete the fusion. Since the depth point cloud model is more accurate, fusing the initial object surface model with it can improve the accuracy of model construction.

[0281] In the related technology, the initial object surface model is often used directly without fusion. However, since the voxels of the initial object surface model will continuously fuse the information of the new frame during the construction process, the reconstructed model surface will become overly smooth and it will be difficult to maintain high-frequency geometric details such as clothing wrinkles. The present application will fuse the initial object surface model with the depth point cloud model. During the fusion, since the depth point cloud model is more accurate, the principle of depth value priority should be followed to retain as much detail information as possible and maximize the accuracy of model construction.

[0282] In addition, the truncated signed distance and the depth value cannot be compared directly. The depth value must first be converted into the corresponding truncated signed distance. The principle of this process is that after the intermediate object surface model is projected into the depth space, if a part of the surface is larger than the depth value, it means that the part corresponding to the depth point cloud model is outside the intermediate object surface model, then the truncated signed distance corresponding to the depth value should be a positive value.

[0283] The difference between the cut-off directed distance and the depth value is negatively correlated with the distance weight and positively correlated with the depth weight; the difference is the absolute value of the difference between the two; when the initial object surface model and the depth space, or the depth point cloud model, are not far apart, the distance weight and the depth weight are also similar. As the gap between the initial object surface model and the depth space increases, or the difference between the cut-off directed distance and the depth value increases, the depth weight gradually increases and the distance weight gradually decreases. For the missing part of the depth point cloud model, it is directly supplemented by the initial object surface model. Fig.12 Shown is a schematic diagram of an intermediate object surface model provided in an embodiment of the present application.

[0284] Through the above method, the body part of the target object is basically reproduced, while the facial details need to be further enhanced through the target facial model, that is, the intermediate object surface model needs to be fused with the target facial model to further reproduce the facial details, such as Fig.13 , which is a geometrical schematic diagram of a target facial model provided in an embodiment of the present application, wherein the portion having a solid surface is the target facial model.

[0285] An optional implementation is that the server determines the first facial weight corresponding to the truncated directed distance of each voxel in the target facial model and the second facial weight corresponding to the truncated directed distance of each voxel in the face of the intermediate object surface model based on the distance between each voxel in the target facial model and the facial edge; then, based on the truncated directed distance of each voxel in the target facial model and the first facial weight corresponding to each voxel, and the truncated directed distance of each voxel in the face of the intermediate object surface model and the second facial weight corresponding to each voxel, the object model of the target object in the current frame is obtained. This process fuses the target facial model with the intermediate object surface model, making the model facial details richer and the model construction more accurate.

[0286] The server integrates the target facial model corresponding to the current frame into the intermediate object surface model face. This process follows the principle of target facial model priority, and in order to ensure the continuity of the target facial model and the intermediate object surface model facial boundary area, weight transition is required at the edge of the target facial model, such as Fig.14 As shown, a schematic diagram of a first facial weight provided in an embodiment of the present application is shown, in which the darker the color, the smaller the first facial weight. That is, the closer the voxel is to the center of the face (for example, the tip of the nose can be taken as the center of the face), the larger the first facial weight corresponding to the truncated directed distance corresponding to the voxel, and the smaller the second facial weight corresponding to the truncated directed distance of the intermediate object surface model. At the very center, the second facial weight is 0.

[0287] In addition, if Fig.15A FIG. 1 is a schematic diagram of a preliminary fusion result of an intermediate object surface model and a target face model provided in an embodiment of the present application. Fig.15A The preliminary fusion results of the intermediate object surface model and the target face model are shown, including schematic diagrams of the entire human body and enlarged images of the face in different representations. It can be seen that after the preliminary fusion, the two models cannot be completely connected, and there may be gaps in the edge areas of the two, such as Fig. 15B As shown, a schematic diagram of eliminating the seam between the target facial model and the intermediate object surface model provided by an embodiment of the present application shows the fusion result after filling the gap. This process uses the surface information of the gap edge through the Poisson reconstruction algorithm to fill the gap, which can eliminate the seam between the target facial model and the intermediate object surface model, thereby providing a topologically continuous and consistent model for the subsequent texture mapping and higher quality model construction. Poisson reconstruction is an implicit surface reconstruction scheme, with a set of directed point clouds of the object surface as input and a three-dimensional mesh of the object surface as output. The vertices of the output mesh do not need to come from the original point cloud, and the result is smoother. In addition, due to the global solution, the watertightness of the mesh can be guaranteed.

[0288] In addition to the current frame, the server can obtain the object model of the target object in each frame of the target video, and obtain a complete object model sequence for describing the position change of the target object in the target video. Fig.16A As shown, it is a model construction result of other frames of a target video provided by an embodiment of the present application, and these models can together constitute an object model sequence. The sequence is a four-dimensional sequence, including three-dimensional space and time, and can be applied to digital entertainment fields represented by game development, film and television special effects, and virtual live broadcasts. It can also be applied to commercial application fields represented by three-dimensional fitting, virtual plastic surgery, and online meetings, as well as the metaverse field built based on virtual reality and augmented reality technologies. For example, after re-lighting the model sequence, the model can be viewed in an immersive 360° manner, or it can be rendered to generate a promotional video. In addition, it can be freely edited, such as adjusting lighting, changing textures, colors, etc., to meet different needs; such as Fig. 16B As shown, it is a result of rendering an object model corresponding to a partial frame provided by an embodiment of the present application. The viewing angle, the color of the model's clothes, etc. are freely changed. Due to the restrictions of the accompanying drawings, Fig. 16B The colors in the video cannot be displayed. Fig. 15B In the four models in the upper part, the characters are wearing red skirts, while in the four models in the lower part, they are wearing white skirts.

[0289] Following the assumption in S203, after obtaining the initial object surface model corresponding to the f-th frame and the target facial model corresponding to the f-th frame, the server projects the initial object surface model corresponding to the f-th frame into the depth space to obtain the truncated signed distance of each voxel in the space; and determines the weights of the depth values ​​and the truncated signed distances, based on the truncated signed distances and their corresponding distance weights, and the depth values ​​and their corresponding depth weights, the fusion of the initial object surface model and the depth point cloud model is achieved, and the detailed information in the depth point cloud model is retained as much as possible. The missing surface in the depth point cloud model is directly filled by the initial object surface model, and finally the intermediate object surface model corresponding to the f-th frame is obtained.

[0290] Afterwards, the server fuses the intermediate object surface model corresponding to the f-th frame with the target face model corresponding to the f-th frame, and fills the seam between the edges of the two models to obtain the object model corresponding to the f-th frame.

[0291] An optional implementation is that the Fig.17 The flowchart shown implements S201 to S204, including the following steps:

[0292] S1701: The server obtains an initial parameterized object model and an initial surface node model corresponding to the current frame.

[0293] If the current frame is a specified frame, the initial parameterized object model is obtained by adjusting the preset parameterized object model at least once, and the initial surface node model is obtained by sampling the initial parameterized object model; the preset parameterized object model is constructed based on the reference frame; if the current frame is not a specified frame, the initial parameterized object model is transformed based on the target surface node model of the previous frame, and the initial surface node model is the target surface node model of the previous frame.

[0294] S1702: The server uses the initial parameterized object model, the initial surface node model, and the depth point cloud model to perform multiple rounds of posture detection on the target object in the current frame to obtain the target surface node model of the current frame.

[0295] In each round of posture detection, the service period first performs posture detection based on the parameterized object model, then passes the detection results to the surface node model, the surface node model performs posture detection, and then passes the detection results to the parameterized object model. This process repeats and completes multiple rounds of posture detection.

[0296] S1703: Based on the target surface node model, the server transforms each truncated directed distance function value in the voxel space corresponding to the reference frame to the current frame to obtain an initial object surface model corresponding to the current frame.

[0297] The server converts the TSDF value stored in each voxel in the reference frame space to the TSDF value stored in each voxel in the current frame, and re-establishes the voxel space in the current frame space coordinate system. The TSDF value obtained in the above conversion is transformed to the current frame space, and then projected to each depth map perspective, and the TSDF value recorded in each voxel is updated. After the voxel information is updated, the server will re-extract the model surface in the new voxel space in the current frame space coordinate system to construct the initial object surface model of the current frame.

[0298] S1704: The server obtains an initial facial model corresponding to the current frame.

[0299] If the current frame is a designated frame, the initial facial model is a preset facial model corresponding to the reference frame; if the current frame is not a designated frame, the initial facial model is a target facial model of the previous frame.

[0300] S1705: The server performs facial detection on the target object in the current frame based on the initial facial model to obtain a target facial model.

[0301] The server obtains the second vertex comparison difference between each vertex in the facial model to be updated and the corresponding point on the depth point cloud model; obtains the key point transformation difference between the key point transformation information in the facial model to be updated and the initial facial model; obtains the facial motion difference between the key point transformation information of the facial model to be updated and the prior key point transformation information; obtains the facial key point estimated difference between the projection position of the facial key points of the facial model to be updated under multiple perspectives and the projection position of the facial key points obtained based on posture estimation under corresponding perspectives; finally, the server adjusts the facial model to be updated based on the second vertex comparison difference, the key point transformation difference, the facial motion difference and the facial key point estimated difference to obtain the target facial model.

[0302] S1706: The server projects the initial object surface model to the depth space corresponding to the current frame.

[0303] Since it is relatively complicated to directly fuse the two model surfaces, the initial object surface model is projected into the depth space to achieve the fusion of the depth point cloud model and the initial object surface model.

[0304] S1707: The server fuses the initial object surface model with the depth point cloud model to obtain an intermediate object surface model.

[0305] S1708: The server fuses the intermediate object surface model with the target face model to obtain the object model of the current frame.

[0306] See also Fig.18 As shown, it is an overall logic diagram of the construction process of an object model provided by an embodiment of the present application. The server constructs a depth map corresponding to the tth frame based on the IR image at 2X viewing angles of the tth frame. According to the depth map, the depth point cloud model and depth space corresponding to the tth frame can be further constructed. The depth point cloud model participates in the pose detection and alternating update between the parameterized object model and the surface node model. Based on the pose detection and alternating update of the parameterized object model and the surface node model, the target surface node model corresponding to the tth frame can be obtained. Based on the skeleton skinning animation algorithm, the TSDF value of the reference frame voxel can be converted to the TSDF value of each voxel of the f frame, and the initial object surface model corresponding to the tth frame can be further obtained.

[0307] At the same time, the server can also perform further detection on the face to build a more accurate facial model. The specific process is that the server detects facial key points and selects the best viewing angle based on the openpose algorithm, and then performs facial detection on the tth frame based on these best viewing angles and facial key points to obtain the target facial model.

[0308] Finally, based on the depth space, the server fuses the depth point cloud model with the initial object surface model into an intermediate object surface model corresponding to the t-th frame, and then fuses the intermediate object surface model with the target face model into an object model corresponding to the t-th frame.

[0309] Compared with other methods, the object model established based on the method provided by this application has more accurate model details and more natural surface. Fig.19 As shown, it is a schematic diagram comparing the model construction results of the present application and the model construction results of other solutions provided in an embodiment of the present application, wherein the first column is two frames of images in the target video, the second column is the construction result using only the SMPL model, the third column is the construction result of the general solution DynamicFusion, and the fourth column is the construction result of the present application.

[0310] Based on the same inventive concept, the present application embodiment also provides a device for constructing an object model. Fig. 20 As shown, it is a schematic diagram of the structure of the object model construction device, which may include:

[0311] The acquisition unit 2001 is used to acquire, for the target object, an initial parameterized object model and an initial surface node model corresponding to the current frame in the target video; the initial parameterized object model represents joint transformation information of each joint of the target object relative to the reference frame, and the initial surface node model represents node transformation information of multiple surface nodes on the model relative to the reference frame;

[0312] A construction unit 2002 is used to perform posture detection on the target object in the current frame based on the initial parameterized object model and the initial surface node model, and to construct an initial object surface model of the target object in the current frame;

[0313] The detection unit 2003 is used to obtain an initial facial model corresponding to the current frame, and perform facial detection on the target object in the current frame based on the initial facial model to obtain a target facial model; the initial facial model represents transformation information of multiple key points on the model relative to the key points of the reference frame;

[0314] The fusion unit 2004 is used to fuse the initial object surface model with the target face model to obtain the object model of the target object in the current frame.

[0315] Optionally, the acquisition unit 2001 is further used for:

[0316] Obtain a depth point cloud model of the target object; the depth point cloud model is constructed based on the multi-view depth map corresponding to the current frame;

[0317] The construction unit 2002 is specifically used for:

[0318] Using the initial parameterized object model, the initial surface node model, and the deep point cloud model, multiple rounds of pose detection are performed on the target object in the current frame, and the surface node model obtained from the last round of pose detection is used as the target surface node model of the target object in the current frame; wherein the initial parameterized object model and the initial surface node model are alternately updated during each round of pose detection;

[0319] An initial object surface model of the target object in the current frame is constructed according to node transformation information of multiple surface nodes in the target surface node model of the current frame.

[0320] Optionally, the construction unit 2002 is specifically configured to perform the following process in each round of posture detection:

[0321] Based on the difference between the current latest parameterized object model and the depth point cloud model, and the difference between the current latest parameterized object model and the parameterized object model to be updated, adjusting the parameterized object model to be updated;

[0322] Based on the adjusted parameterized object model, the current latest surface node model is adjusted to obtain an intermediate surface node model; in the first round of posture detection, the current latest parameterized object model is the initial parameterized object model, and the current latest surface node model is the initial surface node model;

[0323] Based on the difference between the intermediate surface node model and the depth point cloud model, and the difference between the intermediate surface node model and the surface node model to be updated, adjusting the surface node model to be updated;

[0324] Based on the adjusted surface node model, the adjusted parameterized object model is adjusted again.

[0325] Optionally, if the current frame is a designated frame, the initial parameterized object model is obtained by adjusting at least once with a preset parameterized object model, and the initial surface node model is obtained by sampling the initial parameterized object model; the preset parameterized object model is constructed based on a reference frame;

[0326] If the current frame is not a specified frame, the initial parameterized object model is obtained by transforming the target surface node model of the previous frame, and the initial surface node model is the target surface node model of the previous frame.

[0327] Optionally, if the current frame is a designated frame, the initial facial model is a preset facial model corresponding to the reference frame; if the current frame is not a designated frame, the initial facial model is a target facial model of a previous frame;

[0328] The detection unit is specifically used for:

[0329] Obtain a deep point cloud model of the target object and an initial facial model corresponding to the current frame; the deep point cloud model is constructed based on the multi-view depth map corresponding to the current frame;

[0330] Based on the difference between the facial model to be updated and the facial part in the depth point cloud model, as well as the difference between the facial model to be updated and the initial facial model, the facial model to be updated is adjusted to obtain a target facial model.

[0331] Optionally, the fusion unit 2004 is specifically used for:

[0332] Obtain a depth point cloud model of the target object; the depth point cloud model is constructed based on the multi-view depth map corresponding to the current frame;

[0333] The depth point cloud model, the initial object surface model and the target face model are fused to obtain the object model of the target object in the current frame.

[0334] Optionally, the construction unit 2002 is specifically used for:

[0335] Get the first vertex difference between each vertex in the latest parameterized object model and the corresponding point in the depth point cloud model;

[0336] Obtaining the joint transformation difference between the joint transformation information of the latest parameterized object model and the parameterized object model to be updated, the joint point position difference between the positions of each joint point, and the first vertex position difference between the positions of each vertex;

[0337] The parameterized object model to be updated is adjusted based on the first vertex comparison difference, the joint transformation difference, the joint point position difference and the first vertex position difference.

[0338] Optionally, the construction unit 2002 is specifically used for:

[0339] Obtain the surface vertices in the intermediate surface node model and compare the surface vertices with the corresponding points in the depth point cloud model;

[0340] Obtaining node transformation differences between node transformation information in the intermediate surface node model and the surface node model to be updated, and surface node position differences between positions of each surface node;

[0341] Based on the surface vertex comparison difference, the node transformation difference and the surface node position difference, the surface node model to be updated is adjusted.

[0342] Optionally, the detection unit 2003 is specifically used for:

[0343] Obtain the second vertex comparison difference between each vertex in the facial model to be updated and the corresponding point on the depth point cloud model;

[0344] Obtaining key point transformation differences between key point transformation information in the face model to be updated and the initial face model;

[0345] Based on the second vertex comparison difference and the key point transformation difference, the facial model to be updated is adjusted to obtain a target facial model.

[0346] Optionally, the key point transformation information includes: facial position transformation information, facial morphology transformation information and facial expression transformation information of the target object in the current frame relative to the reference frame.

[0347] Optionally, the construction unit 2002 is specifically used for:

[0348] Obtain the joint transformation information of the latest parameterized object model and the joint posture difference between the prior posture parameters; the prior posture parameters represent the joint transformation information that conforms to the preset joint posture;

[0349] Obtain the estimated difference between the projection position of each joint point in the parameterized object model to be updated under multiple viewing angles and the projection position obtained based on the posture estimation;

[0350] The parameterized object model to be updated is adjusted based on the first vertex comparison difference, the joint transformation difference, the joint point position difference, the joint posture difference, the joint point estimation difference and the first vertex position difference.

[0351] Optionally, the construction unit 2002 is specifically used for:

[0352] Obtaining the positional relationship between each surface node and its adjacent surface nodes in the surface node model to be updated;

[0353] Based on the surface vertex comparison differences, node transformation differences, surface node position differences and position relationships, the surface node model to be updated is adjusted.

[0354] Optionally, the detection unit 2003 is specifically used for:

[0355] Obtaining the key point transformation information of the face model to be updated and the facial motion difference between the key point transformation information and the prior key point transformation information; the prior key point transformation information includes the prior expression information and the prior morphology information;

[0356] Obtaining the estimated differences between the projection positions of the facial key points of the facial model to be updated under multiple viewing angles and the projection positions of the facial key points under corresponding viewing angles obtained based on posture estimation;

[0357] Based on the second vertex comparison difference, the key point transformation difference, the facial motion difference and the facial key point estimation difference, the facial model to be updated is adjusted to obtain a target facial model.

[0358] Optionally, the fusion unit 2004 is specifically used for:

[0359] The initial object surface model is projected into the depth space to obtain the truncated directed distance of each voxel in the initial object surface model; the depth space is determined based on the depth point cloud model and is used to represent the depth value of each point in the depth point cloud model;

[0360] Based on the difference between each truncated directed distance and each depth value in the depth space, determine the distance weight corresponding to each truncated directed distance and the depth weight corresponding to each depth value; the difference is negatively correlated with the distance weight and positively correlated with the depth weight;

[0361] Based on each truncated directed distance and its corresponding distance weight, and each depth value and its corresponding depth weight, construct an intermediate object surface model;

[0362] The intermediate object surface model is fused with the target face model to obtain the object model of the target object in the current frame.

[0363] Optionally, the fusion unit 2004 is specifically used for:

[0364] Determine, based on the distance between each voxel in the target facial model and the facial edge, a first facial weight corresponding to the truncated directed distance of each voxel in the target facial model and a second facial weight corresponding to the truncated directed distance of each voxel in the face of the intermediate object surface model;

[0365] Based on the truncated signed distances of each voxel in the target facial model and the corresponding first facial weights, and the truncated signed distances of each voxel in the intermediate object surface model face and the corresponding second facial weights, an object model of the target object in the current frame is obtained.

[0366] Optionally, the device further comprises:

[0367] The combining unit 2005 is used to obtain the object model of the target object in each frame of the target video, and obtain an object model sequence for describing the position and posture changes of the target object in the target video.

[0368] For the convenience of description, the above parts are divided into modules (or units) according to their functions and described separately. Of course, when implementing this application, the functions of each module (or unit) can be implemented in the same or multiple software or hardware.

[0369] After introducing the method and apparatus for constructing an object model according to an exemplary embodiment of the present application, next, an electronic device according to another exemplary embodiment of the present application is introduced.

[0370] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to as "circuit", "module" or "system" herein.

[0371] Based on the same inventive concept as the above method embodiment, an electronic device is also provided in the embodiment of the present application. In one embodiment, the electronic device may be a server, such as Figure 1 In this embodiment, the structure of the electronic device can be as follows: Fig.21 As shown, it includes a memory 2101 , a communication module 2103 and one or more processors 2102 .

[0372] The memory 2101 is used to store computer programs executed by the processor 2102. The memory 2101 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and programs required for running the instant messaging function, etc.; the data storage area may store various instant messaging information and operation instruction sets, etc.

[0373] The memory 2101 may be a volatile memory, such as a random-access memory (RAM); the memory 2101 may also be a non-volatile memory, such as a read-only memory, a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); or the memory 2101 may be any other medium that can be used to carry or store a desired computer program in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2101 may be a combination of the above memories.

[0374] The processor 2102 may include one or more central processing units (CPU) or a digital processing unit, etc. The processor 2102 is used to implement the above object model construction method when calling the computer program stored in the memory 2101 .

[0375] The communication module 2103 is used to communicate with terminal devices and other servers.

[0376] The specific connection medium between the memory 2101, the communication module 2103 and the processor 2102 is not limited in the embodiment of the present application. Fig.21 In the embodiment, the memory 2101 and the processor 2102 are connected via a bus 2104. The bus 2104 is connected to the processor 2102 via a bus 2104. Fig.21 The connections between the other components are only for illustration and are not intended to be limiting. The bus 2104 can be divided into an address bus, a data bus, a control bus, etc. For ease of description, Fig.21 The diagram shows that only one thick line is used, but this does not mean that there is only one bus or only one type of bus.

[0377] The memory 2101 stores a computer storage medium, and the computer storage medium stores computer executable instructions, and the computer executable instructions are used to implement the object model construction method of the embodiment of the present application. The processor 2102 is used to execute the above-mentioned object model construction method, such as Figure 2 shown.

[0378] In another embodiment, the electronic device may also be other electronic devices, such as Figure 1 The terminal device 110 shown in FIG. 1 is a terminal device 110 shown in FIG. 1 . In this embodiment, the structure of the electronic device can be as follows: Fig. 22 As shown, it includes: a communication component 2210, a memory 2220, a display unit 2230, a camera 2240, a sensor 2250, an audio circuit 2260, a Bluetooth module 2270, a processor 2280 and other components.

[0379] The communication component 2210 is used to communicate with the server. In some embodiments, a circuit wireless fidelity (Wireless Fidelity, WiFi) module may be included. The WiFi module belongs to a short-range wireless transmission technology. The electronic device can help an object (such as a user) to send and receive information through the WiFi module.

[0380] The memory 2220 can be used to store software programs and data. The processor 2280 executes various functions and data processing of the terminal device 110 by running the software programs or data stored in the memory 2220. The memory 2220 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. The memory 2220 stores an operating system that enables the terminal device 110 to run. In the present application, the memory 2220 can store an operating system and various application programs, and can also store a computer program for executing the construction method of the object model of the embodiment of the present application.

[0381] The display unit 2230 can also be used to display information input by the object or information provided to the object and a graphical user interface (GUI) of various menus of the terminal device 110. Specifically, the display unit 2230 may include a display screen 2232 disposed on the front of the terminal device 110. The display screen 2232 may be configured in the form of a liquid crystal display, a light emitting diode, etc. The display unit 2230 may be used to display the construction interface of the object model in the embodiment of the present application, etc.

[0382] The display unit 2230 can also be used to receive input digital or character information, and generate signal input related to the object setting and function control of the terminal device 110. Specifically, the display unit 2230 may include a touch screen 2231 set on the front of the terminal device 110, which can collect touch operations of objects on or near it, such as clicking a button, dragging a scroll box, etc.

[0383] The touch screen 2231 can be covered on the display screen 2232, or the touch screen 2231 and the display screen 2232 can be integrated to realize the input and output functions of the terminal device 110. The integrated touch screen can be referred to as a touch display screen. In this application, the display unit 2230 can display the application and the corresponding operation steps.

[0384] The camera 2240 can be used to capture static images, and the object can publish the image taken by the camera 2240 through the application. The camera 2240 can be one or more. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the processor 2280 to convert it into a digital image signal.

[0385] The terminal device may further include at least one sensor 2250, such as an acceleration sensor 2251, a distance sensor 2252, a fingerprint sensor 2253, and a temperature sensor 2254. The terminal device may also be configured with other sensors such as a gyroscope, a barometer, a hygrometer, a thermometer, an infrared sensor, a light sensor, and a motion sensor.

[0386] The audio circuit 2260, the speaker 2261, and the microphone 2262 can provide an audio interface between the object and the terminal device 110. The audio circuit 2260 can transmit the electrical signal converted from the received audio data to the speaker 2261, which is converted into a sound signal for output. The terminal device 110 can also be configured with a volume button for adjusting the volume of the sound signal. On the other hand, the microphone 2262 converts the collected sound signal into an electrical signal, which is received by the audio circuit 2260 and converted into audio data, and then the audio data is output to the communication component 2210 to be sent to, for example, another terminal device 110, or the audio data is output to the memory 2220 for further processing.

[0387] The Bluetooth module 2270 is used to exchange information with other Bluetooth devices having Bluetooth modules through the Bluetooth protocol. For example, the terminal device can establish a Bluetooth connection with a wearable electronic device (such as a smart watch) that also has a Bluetooth module through the Bluetooth module 2270 to exchange data.

[0388] The processor 2280 is the control center of the terminal device. It uses various interfaces and lines to connect various parts of the entire terminal. It executes various functions of the terminal device and processes data by running or executing software programs stored in the memory 2220 and calling data stored in the memory 2220. In some embodiments, the processor 2280 may include one or more processing units; the processor 2280 may also integrate an application processor and a baseband processor, wherein the application processor mainly processes the operating system, object interface and application program, and the baseband processor mainly processes wireless communication. It is understandable that the above-mentioned baseband processor may not be integrated into the processor 2280. In the present application, the processor 2280 can run the operating system, application program, object interface display and touch response, as well as the object model construction method of the embodiment of the present application. In addition, the processor 2280 is coupled to the display unit 2230.

[0389] In some possible implementations, various aspects of the object model construction method provided in the present application may also be implemented in the form of a program product, which includes a computer program. When the program product is run on an electronic device, the computer program is used to enable the electronic device to execute the steps of the object model construction method according to various exemplary embodiments of the present application described above in this specification. For example, the electronic device may execute the following steps: Figure 2 Follow the steps shown in .

[0390] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0391] The program product of the embodiment of the present application may adopt a portable compact disk read-only memory (CD-ROM) and include a computer program, and can be run on an electronic device. However, the program product of the present application is not limited thereto, and in this document, a readable storage medium may be any tangible medium containing or storing a program, which can be used by or in combination with a command execution system, apparatus, or device.

[0392] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein a readable computer program is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with a command execution system, apparatus, or device.

[0393] The computer program embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0394] Computer programs for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The computer program may be executed entirely on the subject electronic device, partially on the subject electronic device, as a separate software package, partially on the subject electronic device and partially on a remote electronic device, or entirely on a remote electronic device or server. In cases involving remote electronic devices, the remote electronic device may be connected to the subject electronic device via any type of network including a local area network (LAN) or a wide area network (WAN), or may be connected to an external electronic device (e.g., via the Internet using an Internet service provider).

[0395] It should be noted that, although several units or subunits of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into multiple units to be embodied.

[0396] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0397] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain a computer-usable computer program.

[0398] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program commands. These computer program commands can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the commands executed by the processor of the computer or other programmable data processing device generate commands for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0399] These computer program commands may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the commands stored in the computer readable memory produce an article of manufacture comprising a command device, the command device implementing the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0400] These computer program commands can also be loaded onto a computer or other programmable data processing device so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the commands executed on the computer or other programmable device provide the instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0401] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0402] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for constructing an object model, characterized in that: The method comprises: For the target object, an initial parameterized object model and an initial surface node model corresponding to the current frame are obtained in the target video; the initial parameterized object model represents the joint transformation information of each joint of the target object relative to the reference frame, and the initial surface node model represents the node transformation information of multiple surface nodes on the model relative to the reference frame; Based on the initial parameterized object model and the initial surface node model, performing pose detection on the target object in the current frame, and constructing an initial object surface model of the target object in the current frame; Obtaining an initial facial model corresponding to the current frame, and performing facial detection on the target object in the current frame based on the initial facial model to obtain a target facial model; the initial facial model represents transformation information of multiple key points on the model relative to the key points of the reference frame; The initial object surface model is fused with the target face model to obtain an object model of the target object in the current frame.

2. The method according to claim 1, wherein The method further comprises: Acquire a depth point cloud model of the target object; the depth point cloud model is constructed based on a multi-view depth map corresponding to the current frame; The performing pose detection on the target object in the current frame based on the initial parameterized object model and the initial surface node model, and constructing the initial object surface model of the target object in the current frame, includes: Using the initial parameterized object model, the initial surface node model, and the depth point cloud model, perform multiple rounds of pose detection on the target object in the current frame, and use the surface node model obtained from the last round of pose detection as the target surface node model of the target object in the current frame; wherein the initial parameterized object model and the initial surface node model are alternately updated during each round of pose detection; An initial object surface model of the target object in the current frame is constructed according to node transformation information of a plurality of surface nodes in the target surface node model of the current frame.

3. The method according to claim 2, wherein Each round of pose detection performs the following process: Adjusting the parameterized object model to be updated based on a difference between the current latest parameterized object model and the depth point cloud model, and a difference between the current latest parameterized object model and the parameterized object model to be updated; Based on the adjusted parameterized object model, the current latest surface node model is adjusted to obtain an intermediate surface node model; in the first round of posture detection, the current latest parameterized object model is the initial parameterized object model, and the current latest surface node model is the initial surface node model; Adjusting the surface node model to be updated based on a difference between the intermediate surface node model and the depth point cloud model, and a difference between the intermediate surface node model and the surface node model to be updated; Based on the adjusted surface node model, the adjusted parameterized object model is adjusted again.

4. The method according to claim 2, wherein If the current frame is a designated frame, the initial parameterized object model is obtained by adjusting at least once with a preset parameterized object model, and the initial surface node model is obtained by sampling the initial parameterized object model; the preset parameterized object model is constructed based on a reference frame; If the current frame is not a designated frame, the initial parameterized object model is obtained by transforming the target surface node model of the previous frame, and the initial surface node model is the target surface node model of the previous frame.

5. The method according to claim 1, wherein If the current frame is a designated frame, the initial facial model is the preset facial model corresponding to the reference frame; if the current frame is not a designated frame, the initial facial model is the target facial model of the previous frame; The acquiring an initial facial model corresponding to the current frame, and performing facial detection on the target object in the current frame based on the initial facial model to obtain a target facial model, includes: Obtaining a depth point cloud model of the target object and an initial facial model corresponding to the current frame; the depth point cloud model is constructed based on a multi-view depth map corresponding to the current frame; Based on the difference between the facial model to be updated and the facial part in the depth point cloud model, and the difference between the facial model to be updated and the initial facial model, the facial model to be updated is adjusted to obtain the target facial model.

6. The method according to claim 1, wherein The fusing the initial object surface model with the target face model to obtain an object model of the target object in the current frame includes: Acquire a depth point cloud model of the target object; the depth point cloud model is constructed based on a multi-view depth map corresponding to the current frame; The depth point cloud model, the initial object surface model and the target face model are fused to obtain an object model of the target object in the current frame.

7. The method according to claim 3, wherein The adjusting the parameterized object model to be updated based on the difference between the current latest parameterized object model and the depth point cloud model, and the difference between the current latest parameterized object model and the parameterized object model to be updated, includes: Obtaining a first vertex comparison difference between each vertex in the current latest parameterized object model and a corresponding point in the depth point cloud model; Obtaining joint transformation differences between the joint transformation information of the current latest parameterized object model and the parameterized object model to be updated, joint point position differences between the positions of the joint points, and first vertex position differences between the positions of the vertices; The parameterized object model to be updated is adjusted based on the first vertex comparison difference, the joint transformation difference, the joint point position difference and the first vertex position difference.

8. The method according to claim 3, wherein The adjusting the surface node model to be updated based on the difference between the intermediate surface node model and the depth point cloud model, and the difference between the intermediate surface node model and the surface node model to be updated, comprises: Obtaining surface vertex differences between each surface vertex in the intermediate surface node model and corresponding points in the depth point cloud model; Obtaining node transformation differences between node transformation information in the intermediate surface node model and the surface node model to be updated, and surface node position differences between positions of respective surface nodes; The surface node model to be updated is adjusted based on the surface vertex comparison difference, the node transformation difference and the surface node position difference.

9. The method according to claim 5, wherein The step of adjusting the facial model to be updated based on the difference between the facial model to be updated and the facial part in the depth point cloud model, as well as the difference between the facial model to be updated and the initial facial model, to obtain the target facial model includes: Obtaining a second vertex comparison difference between each vertex in the facial model to be updated and a corresponding point on the depth point cloud model; Obtaining a key point transformation difference between key point transformation information in the face model to be updated and the initial face model; Based on the second vertex comparison difference and the key point transformation difference, the facial model to be updated is adjusted to obtain the target facial model.

10. The method according to claim 9, wherein The key point transformation information includes: facial position transformation information, facial morphology transformation information, and facial expression transformation information of the target object in the current frame relative to the reference frame.

11. The method according to claim 7, wherein The adjusting the parameterized object model to be updated based on the first vertex comparison difference, the joint transformation difference, the joint point position difference and the first vertex position difference includes: Obtaining the joint transformation information of the latest parameterized object model and the difference between the joint posture and the prior posture parameters; the prior posture parameters represent the joint transformation information that conforms to the preset joint posture; Obtaining a difference between a projection position of each joint point in the parameterized object model to be updated under multiple viewing angles and a projection position obtained based on posture estimation; The parameterized object model to be updated is adjusted based on the first vertex comparison difference, the joint transformation difference, the joint point position difference, the joint posture difference, the joint point estimated difference and the first vertex position difference.

12. The method according to claim 8, wherein The adjusting the surface node model to be updated based on the surface vertex comparison difference, the node transformation difference and the surface node position difference includes: Obtaining a positional relationship between each surface node and its adjacent surface nodes in the surface node model to be updated; The surface node model to be updated is adjusted based on the surface vertex comparison difference, the node transformation difference, the surface node position difference and the position relationship.

13. The method according to claim 9, wherein The step of adjusting the facial model to be updated based on the second vertex comparison difference and the key point transformation information to obtain a target facial model includes: Obtaining facial motion differences between key point transformation information of the face model to be updated and prior key point transformation information; the prior key point transformation information includes prior expression information and prior morphology information; Obtaining the estimated differences between the projection positions of the facial key points of the facial model to be updated under multiple viewing angles and the projection positions of the facial key points under corresponding viewing angles obtained based on posture estimation; Based on the second vertex comparison difference, the key point transformation difference, the facial motion difference and the facial key point estimated difference, the facial model to be updated is adjusted to obtain the target facial model.

14. The method according to any one of claims 6, wherein: The fusing the depth point cloud model, the initial object surface model, and the target face model to obtain an object model of the target object in the current frame includes: Projecting the initial object surface model into a depth space to obtain a truncated signed distance of each voxel in the initial object surface model; the depth space is determined based on the depth point cloud model and is used to represent the depth value of each point in the depth point cloud model; Determining, based on a difference between each truncated directed distance and each depth value in the depth space, a distance weight corresponding to each truncated directed distance and a depth weight corresponding to each depth value; wherein the difference is negatively correlated with the distance weight and positively correlated with the depth weight; constructing an intermediate object surface model based on the truncated directed distances and their corresponding distance weights, and the depth values ​​and their corresponding depth weights; The intermediate object surface model is fused with the target face model to obtain an object model of the target object in the current frame.

15. The method according to claim 14, wherein The fusing the intermediate object surface model with the target face model to obtain an object model of the target object in the current frame includes: Determining, based on the distance between each voxel in the target facial model and the facial edge, a first facial weight corresponding to the truncated signed distance of each voxel in the target facial model and a second facial weight corresponding to the truncated signed distance of each voxel in the face of the intermediate object surface model; An object model of the target object in the current frame is obtained based on the truncated signed distances of each voxel in the target facial model and the corresponding first facial weights, and the truncated signed distances of each voxel in the face of the intermediate object surface model and the corresponding second facial weights.

16. The method according to any one of claims 1 to 13, wherein: The method further comprises: An object model of a target object in each frame of the target video is obtained to obtain an object model sequence for describing a posture change of the target object in the target video.

17. A device for constructing an object model, characterized in that: include: An acquisition unit, configured to acquire, for a target object, an initial parameterized object model and an initial surface node model corresponding to a current frame in a target video; The initial parameterized object model represents joint transformation information of each joint of the target object relative to the reference frame, and the initial surface node model represents node transformation information of multiple surface nodes on the model relative to the reference frame; A construction unit, configured to perform pose detection on the target object in the current frame based on the initial parameterized object model and the initial surface node model, and construct an initial object surface model of the target object in the current frame; a detection unit, configured to obtain an initial facial model corresponding to a current frame, and perform facial detection on the target object in the current frame based on the initial facial model to obtain a target facial model; the initial facial model represents transformation information of multiple key points on the model relative to the key points of a reference frame; A fusion unit is used to fuse the initial object surface model with the target face model to obtain an object model of the target object in the current frame.

18. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor is enabled to perform the steps of any one of the methods of claims 1 to 16.

19. A computer-readable storage medium, characterized in that The method comprises a computer program. When the computer program is run on an electronic device, the computer program is used to enable the electronic device to execute the steps of any one of the methods of claims 1 to 16.

20. A computer program product, characterized in that The method comprises a computer program stored in a computer-readable storage medium; when a processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, so that the electronic device performs the steps of any one of the methods described in claims 1 to 16.