Method, device, computer equipment and storage medium for three-dimensional reconstruction of target object

Abstract

The present application relates to artificial intelligence and computer vision technology, and to a method, device, computer equipment and storage medium for three-dimensional reconstruction of a target object. The method comprises: acquiring a two-dimensional image including a target object; creating a three-dimensional parameterized model of the target object according to the two-dimensional image; generating a three-dimensional continuous surface of the target object according to the two-dimensional image; the three-dimensional continuous surface is a three-dimensional surface obtained by continuously representing the surface of the target object; meshing the three-dimensional continuous surface to obtain a three-dimensional mesh surface; adjusting the model parameters of the parameterized model to register the mesh surface in the parameterized model with the three-dimensional mesh surface to obtain the final target parameterized model of the target object. The use of this method can improve the accuracy of three-dimensional reconstruction.

Description

Technical Field

[0001] The present application relates to the fields of computer vision and artificial intelligence, and in particular to a method, apparatus, computer device and storage medium for three-dimensional reconstruction of a target object. Background Art

[0002] With the development of computer vision and artificial intelligence technology, there are more and more scenarios for 3D reconstruction of target objects. For example, 3D reconstruction of the human body is very important in application scenarios such as human animation, virtual reality and games. There are many methods for 3D reconstruction of target objects, and 3D reconstruction based on the 2D image of the target object is one of them.

[0003] At present, the method of 3D reconstruction based on the 2D image of the target object generally reconstructs a parametric model of the target object based on the 2D image. However, due to the limitations of the parametric model itself, the reconstructed parametric model of the target object has limited expression of the shape of the target object and often lacks detailed information, resulting in inaccurate results of the 3D reconstruction of the target object. Summary of the invention

[0004] Based on this, it is necessary to provide a method, device, computer equipment and storage medium for three-dimensional reconstruction of a target object that can improve accuracy in response to the above technical problems.

[0005] A method for three-dimensional reconstruction of a target object, the method comprising:

[0006] acquiring a two-dimensional image including a target object;

[0007] Creating a three-dimensional parameterized model of the target object based on the two-dimensional image;

[0008] Generate a three-dimensional continuous surface of the target object according to the two-dimensional image; the three-dimensional continuous surface is a three-dimensional surface obtained by continuously representing the surface of the target object;

[0009] Meshing the three-dimensional continuous surface to obtain a three-dimensional mesh surface;

[0010] The model parameters of the parameterized model are adjusted to perform registration processing on the mesh surface in the parameterized model and the three-dimensional mesh surface to obtain a final target parameterized model of the target object.

[0011] In one embodiment, the method further comprises:

[0012] Adding an offset parameter to the parameterized model to obtain a deformable parameterized model;

[0013] The step of adjusting the model parameters of the parameterized model to register the mesh surface in the parameterized model with the three-dimensional mesh surface includes:

[0014] The offset parameters corresponding to the surface vertices of the deformable parameterized model are adjusted to align the mesh surface in the parameterized model with the three-dimensional mesh surface.

[0015] In one of the embodiments, the model parameters of the deformable parameterized model further include global feature parameters, posture parameters and shape parameters;

[0016] The step of adjusting the offset parameters corresponding to the surface vertices of the deformable parameterized model to register the mesh surface in the parameterized model with the three-dimensional mesh surface includes:

[0017] adjusting global feature parameters of the deformable parameterized model to perform depth registration of a mesh surface in the parameterized model with the three-dimensional mesh surface;

[0018] Adjusting the posture parameters and shape parameters of the parameterized model after the depth registration to coincide and register the mesh surface in the parameterized model with the three-dimensional mesh surface;

[0019] The offset parameters corresponding to the surface vertices of the parameterized model after the coincidence and registration are adjusted to perform feature part registration between the mesh surface in the parameterized model and the three-dimensional mesh surface.

[0020] In one embodiment, the adjusting the offset parameters corresponding to the surface vertices of the deformable parameterized model to register the mesh surface in the parameterized model with the three-dimensional mesh surface includes:

[0021] Obtaining a semantic segmentation result of surface vertices of the parameterized model;

[0022] Determining, according to the surface vertex semantic segmentation result, surface vertices of the deformable parameterized model whose geometric shapes change dramatically;

[0023] The offset parameters corresponding to the determined surface vertices are fixed, and the offset parameters corresponding to the surface vertices other than the determined surface vertices in the deformable parameterized model are adjusted to align the mesh surface in the parameterized model with the three-dimensional mesh surface.

[0024] In one embodiment, the two-dimensional image is a single-frame two-dimensional image; the parameterized model created according to the single-frame two-dimensional image is a first parameterized model;

[0025] The obtaining of the surface vertex semantic segmentation result of the parameterized model comprises:

[0026] Acquire multiple frames of two-dimensional images including the target object;

[0027] Acquire a second parameterized model of the target object created corresponding to each of the two-dimensional image frames in the multiple two-dimensional image frames;

[0028] Performing semantic segmentation on each of the multiple two-dimensional image frames to obtain a surface vertex semantic segmentation result of a corresponding second parameterized model;

[0029] The semantic segmentation result of the surface vertices of the first parameterized model is determined according to the semantic segmentation result of the surface vertices of each of the second parameterized models.

[0030] In one of the embodiments, after determining the semantic segmentation result of the surface vertices of the first parameterized model according to the semantic segmentation results of the surface vertices of each of the second parameterized models, the method further includes:

[0031] Determining invisible surface vertices for which semantic segmentation results are not determined in the first parameterized model;

[0032] Determine a visible surface vertex within a preset neighborhood of the invisible surface vertex;

[0033] According to the determined semantic segmentation result of the visible surface vertices, the semantic segmentation result of the invisible surface vertices is determined.

[0034] In one embodiment, adjusting the model parameters of the parameterized model to register the mesh surface in the parameterized model with the three-dimensional mesh surface includes:

[0035] Obtaining a target loss function of multiple constraints; the target loss function includes a mesh difference loss function, a surface topology structure loss function and a deformation loss function;

[0036] The model parameters of the parameterized model are iteratively adjusted in a direction to minimize the objective loss function, so as to perform registration processing on the mesh surface in the parameterized model and the three-dimensional mesh surface.

[0037] In one embodiment, the two-dimensional image is a single-frame two-dimensional image; the parameterized model created according to the single-frame two-dimensional image is a first parameterized model;

[0038] The method further comprises:

[0039] Acquire multiple frames of two-dimensional images including the target object;

[0040] Acquire a second parameterized model of the target object created corresponding to each of the two-dimensional image frames in the multiple two-dimensional image frames;

[0041] According to the texture coordinates corresponding to the surface vertices of each of the second parameterized models, the points corresponding to the target object in the corresponding two-dimensional image are mapped to the texture space to obtain the initial texture map of the target object corresponding to each frame of the two-dimensional image;

[0042] Merging the initial texture maps to obtain a texture map of the target object;

[0043] The target parameterized model is texture rendered according to the texture map.

[0044] In one embodiment, the fusing of the initial texture maps to obtain the texture map of the target object includes:

[0045] Determining the fusion order corresponding to each frame of the two-dimensional image according to the root node direction of each second parameterized model;

[0046] The initial texture maps corresponding to the two-dimensional image frames are fused in the fusion order to obtain the texture map of the target object.

[0047] In one embodiment, the method further comprises:

[0048] Obtaining visibility maps of the target object corresponding to each frame of the two-dimensional image;

[0049] The step of fusing the initial texture maps corresponding to the two-dimensional images of each frame in the fusion order to obtain the texture map of the target object includes:

[0050] According to the fusion order, the current two-dimensional image is selected from the first two-dimensional image, and the texture map corresponding to the current two-dimensional image is generated according to the initial texture map and visibility map corresponding to the current two-dimensional image;

[0051] Fusing the texture map corresponding to the current two-dimensional image with the accumulated fused texture map;

[0052] After fusion, the next two-dimensional image is used as the current two-dimensional image in the fusion order, and the step of generating the texture map corresponding to the current two-dimensional image according to the initial texture map and visibility map corresponding to the current two-dimensional image is iteratively returned to continue execution until the iteration stops, and the texture map of the target object is obtained.

[0053] In one embodiment, obtaining visibility maps corresponding to each frame of two-dimensional image includes:

[0054] According to the normal vectors of the surfaces of the second parameterized models, a normal vector map corresponding to the corresponding two-dimensional image is generated;

[0055] For each frame of a two-dimensional image, the visibility of each point corresponding to the target object in the normal vector map is determined based on the proximity between the normal vector direction in the normal vector map corresponding to the two-dimensional image and the shooting direction of the two-dimensional image, and a visibility map corresponding to the two-dimensional image is obtained.

[0056] In one of the embodiments, the target parameterized model is a drivable parameterized model;

[0057] The method further comprises:

[0058] Get action parameters;

[0059] The action parameters are substituted into the target parameterized model to drive the target parameterized model to perform corresponding actions.

[0060] A device for three-dimensional reconstruction of a target object, the device comprising:

[0061] An image acquisition module, used to acquire a two-dimensional image including a target object;

[0062] A parameterized model creation module, used to create a three-dimensional parameterized model of the target object according to the two-dimensional image;

[0063] A continuous surface generation module, used to generate a three-dimensional continuous surface of the target object according to the two-dimensional image; the three-dimensional continuous surface is a three-dimensional surface obtained by continuously representing the surface of the target object;

[0064] A meshing module, used for meshing the three-dimensional continuous surface to obtain a three-dimensional mesh surface;

[0065] The model parameter adjustment module is used to adjust the model parameters of the parameterized model to perform registration processing on the mesh surface in the parameterized model and the three-dimensional mesh surface to obtain the final target parameterized model of the target object.

[0066] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps in the target object three-dimensional reconstruction method described in each embodiment of the present application.

[0067] A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the processor executes the steps in the method for three-dimensional reconstruction of a target object described in each embodiment of the present application.

[0068] A computer program product or a computer program, wherein the computer program product or the computer program comprises computer instructions, wherein the computer instructions are stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the steps in the method for three-dimensional reconstruction of a target object described in each embodiment of the present application.

[0069] The above-mentioned method, device, computer equipment and storage medium for three-dimensional reconstruction of the target object create a three-dimensional parametric model of the target object according to the two-dimensional image of the target object, generate a three-dimensional continuous surface of the target object according to the two-dimensional image, mesh the three-dimensional continuous surface to obtain a three-dimensional mesh surface, and then adjust the model parameters of the parametric model to align the mesh surface in the parametric model with the three-dimensional mesh surface to obtain the final target parametric model of the target object. Because the three-dimensional continuous surface has the detailed information of the target object, the target parametric model obtained by aligning the mesh surface in the parametric model with the three-dimensional mesh surface has a stronger ability to express the shape of the target object and can have the detailed information of the target object, avoiding the limitation that the parametric model of the target object obtained by three-dimensional reconstruction of the parametric model has limited expression of the shape of the target object and lacks detailed information, thereby improving the accuracy of three-dimensional reconstruction of the target object. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 A diagram showing an application environment of a method for three-dimensional reconstruction of a target object in an embodiment;

[0071] Figure 2 A schematic diagram of a process of a method for three-dimensional reconstruction of a target object in one embodiment;

[0072] Figure 3 A schematic diagram of creating a parametric model and a three-dimensional continuous surface in one embodiment;

[0073] Figure 4 A schematic diagram of performing registration processing on a parameterized model and a three-dimensional continuous surface in one embodiment;

[0074] Figure 5 A schematic diagram of determining a semantic segmentation result of surface vertices of a first parameterized model in one embodiment;

[0075] Figure 6 A schematic diagram of determining an initial texture map and a visibility map in one embodiment;

[0076] Figure 7 A schematic diagram of obtaining a texture map of a target object by fusing an initial texture map and a visibility map in one embodiment;

[0077] Figure 8 A schematic diagram of the effect of a target parameterized model in one embodiment;

[0078] Fig. 9 A schematic diagram of performing action driving on a target parameterized model in one embodiment;

[0079] Fig.10 A schematic diagram of the overall process of a method for three-dimensional reconstruction of a target object in one embodiment;

[0080] Fig.11 is a structural block diagram of a target object three-dimensional reconstruction device in one embodiment;

[0081] Fig.12 is a structural block diagram of a device for three-dimensional reconstruction of a target object in another embodiment;

[0082] Fig.13 is an internal structure diagram of a computer device in one embodiment;

[0083] Fig.14 FIG. 4 is a diagram showing the internal structure of a computer device in another embodiment. DETAILED DESCRIPTION

[0084] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0085] The target object three-dimensional reconstruction method provided in this application can be applied to Figure 1 In the application environment shown. Among them, the image acquisition device 102 can perform image acquisition on the target object 104, and the computer device 106 can obtain the two-dimensional image acquired by the image acquisition device 102, and according to the two-dimensional image, adopt the target object three-dimensional reconstruction method in each embodiment of the present application to perform three-dimensional reconstruction on the target object to obtain a target parameterized model of the target object. The computer device 106 can acquire the two-dimensional image from the image acquisition device 102 by communicating with the image acquisition device 102 through the network. The two-dimensional image acquired by the image acquisition device 102 can also be manually stored in the computer device 106.

[0086] The image acquisition device 102 may be, but is not limited to, various cameras, webcams, video cameras, and video recorders. The target object 104 may be, but is not limited to, a human body, an animal, and a robot. The computer device 106 may be a terminal or a server, or may be implemented by a terminal and a server. The terminal may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, vehicle-mounted computers, and portable wearable devices. The server may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It may also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0087] It should be noted that, in other embodiments, the computer device 106 may also directly obtain an existing two-dimensional image of the target object 104 from a database or storage device, etc., instead of using the image acquisition device 102 to acquire the two-dimensional image of the target object 104 .

[0088] It can be understood that the three-dimensional reconstruction method of the target object in each embodiment of the present application adopts computer vision technology and machine learning technology in artificial intelligence technology, etc., which can effectively realize the three-dimensional reconstruction of the target object.

[0089] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0090] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0091] Computer Vision (CV) is a science that studies how to make machines "see". To put it more specifically, it refers to the use of cameras and computers to replace human eyes to identify, follow and measure targets, and further perform graphics processing so that the computer processing becomes an image that is more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0092] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0093] In one embodiment, Figure 2 As shown, a method for three-dimensional reconstruction of a target object is provided. In the embodiment of the present application, the method is applied to Figure 1 The computer device in the example is used to illustrate, including the following steps:

[0094] Step 202: Acquire a two-dimensional image including the target object.

[0095] The target object is the object for 3D reconstruction, and the 2D image including the target object refers to the image content of the target object contained in the 2D image.

[0096] In one embodiment, the target object may be any one of a human body, an animal, and a robot.

[0097] In one embodiment, the image acquisition device may acquire an image of the target object to obtain a two-dimensional image of the target object, and then the computer device may acquire the two-dimensional image acquired by the image acquisition device.

[0098] In one embodiment, the computer device can obtain the two-dimensional image of the target object from the image acquisition device through the network. In another embodiment, the two-dimensional image acquired by the image acquisition device can be manually stored in the computer device through any one of a memory card, a mobile hard disk, and a USB flash drive.

[0099] In one embodiment, the computer device may also directly obtain an existing two-dimensional image of the target object. For example, the computer device may obtain an existing two-dimensional image of the target object from a database or locally.

[0100] In one embodiment, if the target object is a human body, the target object can be placed in a standard posture suitable for three-dimensional reconstruction, and the image acquisition device acquires images of the human body in the standard posture to obtain a two-dimensional image of the target object. For example, the standard posture can be Figure 3 The figure in the figure is in a posture shaped like the letter "A". It can be understood that the standard posture can be set arbitrarily according to the requirements of three-dimensional reconstruction, and there is no limitation on this.

[0101] In one embodiment, the computer device may first acquire multiple two-dimensional images including the target object, then select one of the two-dimensional images from the multiple two-dimensional images, and execute step 204 and subsequent steps according to the selected single two-dimensional image.

[0102] In one embodiment, the selected single-frame two-dimensional image is a two-dimensional image suitable for three-dimensional reconstruction, for example, it may be a two-dimensional image of the front side of the target object.

[0103] In one embodiment, the multiple frames of two-dimensional images may include two-dimensional images taken at different directions of the target object. In one embodiment, the multiple frames of two-dimensional images may include two-dimensional images taken at different directions around the target object.

[0104] In one embodiment, the image acquisition device may circle the target object and acquire multiple frames of two-dimensional images of the target object during the process. In another embodiment, the target object may rotate once, and during the process, the image acquisition device at a fixed position acquires multiple frames of two-dimensional images of the target object.

[0105] In one embodiment, the multiple frames of two-dimensional images may be two-dimensional images extracted from continuously shot videos. In another embodiment, the multiple frames of two-dimensional images may also be two-dimensional images obtained by taking multiple photos.

[0106] In one embodiment, the computer device may first crop a two-dimensional image of the area where the target object is located from the two-dimensional image including the target object, and then perform step 204 and subsequent steps based on the cropped two-dimensional image of the area where the target object is located. The two-dimensional image of the area where the target object is located refers to a two-dimensional image in which the image content corresponding to the target object occupies the main area. In this embodiment, cropping the two-dimensional image of the area where the target object is located can avoid problems such as the presence of multiple objects in the original two-dimensional image including the target object, or the target object is not in the center of the two-dimensional image, and can improve the accuracy of subsequent three-dimensional reconstruction.

[0107] In one embodiment, the computer device may first detect key points of the target object or a target frame where the target object is located from the original two-dimensional image including the target object, and then cut out the two-dimensional image of the area where the target object is located from the original two-dimensional image according to the key points or the target frame. Figure 3 As shown, 302 is the original two-dimensional image, and the person in it is the target object. By performing target detection and cropping on 302, a two-dimensional image 304 of the area where the target object is located can be cropped.

[0108] In one embodiment, the computer device may use a human key point detection network (Keypoint R-CNN) to detect key points of a target object or a target box where the target object is located from a two-dimensional image including the target object.

[0109] Step 204: Create a three-dimensional parameterized model of the target object based on the two-dimensional image.

[0110] The three-dimensional parametric model refers to a three-dimensional model of a target object obtained by extracting model parameters from a two-dimensional image of the target object and modeling according to the model parameters. The surface of the three-dimensional parametric model is composed of a grid (ie, mesh) composed of vertices and facets.

[0111] In one embodiment, the three-dimensional parameterized model can be a parameterized model of a human body or a parameterized model of an animal. In one embodiment, the parameterized model of a human body can be any one of the SMPL model (Skinned multi-person linear model), the SMPLH model (Skinned multi-person linear model with hand), and the SMPLX model (Skinned multi-person linear model with expression). It can be understood that there are many kinds of parameterized models, and the parameterized models used are not limited here, as long as the parameterized models that separately model the shape and posture can be used.

[0112] In one embodiment, the computer device can extract model parameters from the two-dimensional image, and then input the model parameters into the initial parameterized model to create a parameterized model of the target object. Specifically, the computer device can extract shape parameters and posture parameters of the target object from the two-dimensional image, and then input the shape parameters and posture parameters into the initial parameterized model to create a parameterized model of the target object.

[0113] The initial parametric model refers to a parametric model without inputting the shape parameters and posture parameters of the target object. It can be understood that the initial parametric model does not have the specific shape and posture of the target object. After inputting the shape parameters and posture parameters of the target object into the initial parametric model, the parametric model corresponding to the target object can be created.

[0114] In one embodiment, if the target object is a human body, the computer device may use the Expose method (a method for creating an SMPLX model) to create an SMPLX model (a multi-person linear skin model with gestures and facial expressions) of the target object according to the two-dimensional image. The parameterized model created by this method, compared to the SMPL model, adds human gestures and facial expressions, and improves the resolution, thereby improving the accuracy of three-dimensional reconstruction.

[0115] Step 206 , generating a three-dimensional continuous surface of the target object according to the two-dimensional image; the three-dimensional continuous surface is a three-dimensional surface obtained by continuously representing the surface of the target object.

[0116] It can be understood that the three-dimensional continuous surface has detailed information of the target object.

[0117] In one embodiment, the three-dimensional continuous surface may be an implicit surface, wherein an implicit surface (Implicit Surface, a way of expressing a surface) is a surface defined by an implicit function, that is, an isosurface defined by an implicit function.

[0118] In one embodiment, the computer device may generate an implicit surface of a target object based on a two-dimensional image by a PIFuHD method (a method for generating an implicit surface of a target object through an end-to-end trainable coarse-to-fine model framework).

[0119] like Figure 3 As shown in FIG. 3 , the computer device can create a parameterized model 308 of the target object and a three-dimensional continuous surface 306 of the target object according to the two-dimensional image 304 of the area where the target object is located. It can be understood that because the three-dimensional continuous surface is continuous and has a high resolution, it has detailed information of the target object. Figure 3 As can be seen from the figure, the three-dimensional continuous surface 306 contains the detailed information of the target object, such as the detailed information of clothes, hair and shoes. The parametric model is created by inputting shape parameters and posture parameters into the initial parametric model, so it lacks detailed information. The parametric model 308 of the target human body is generally as follows: Figure 3 The nude human model shown lacks detailed information such as the target person's specific clothes, shoes, and hair.

[0120] Step 208, meshing the three-dimensional continuous surface to obtain a three-dimensional mesh surface.

[0121] Among them, the three-dimensional mesh surface (Mesh) refers to the mesh surface composed of vertices and facets.

[0122] In one embodiment, the computer device may use the Marching cubes method (isosurface extraction algorithm, a method for extracting meshes) to mesh the three-dimensional continuous surface to obtain a three-dimensional mesh surface.

[0123] In other embodiments, the computer device may also use other methods of extracting meshes to mesh the three-dimensional continuous surface to obtain a three-dimensional mesh surface, without limitation.

[0124] It can be understood that the form of the parameterized model and the three-dimensional continuous surface are different. The parameterized model is a mesh composed of vertices and facets, while the three-dimensional continuous surface is a three-dimensional surface represented continuously. Therefore, in this embodiment, the three-dimensional continuous surface is meshed to obtain a three-dimensional mesh surface, which can make the form of the parameterized model consistent with that of the three-dimensional continuous surface, and facilitate the subsequent registration processing between the parameterized model and the three-dimensional continuous surface. That is, in the registration process, it is only necessary to register the mesh surface in the parameterized model with the three-dimensional mesh surface.

[0125] Step 210 , adjusting the model parameters of the parameterized model to register the mesh surface in the parameterized model with the three-dimensional mesh surface to obtain a final target parameterized model of the target object.

[0126] Among them, registration refers to the process of adjusting the shape, posture and details of two mesh surfaces to make them consistent. The target parameterized model is the parameterized model finally obtained by three-dimensional reconstruction of the target object.

[0127] In one embodiment, an offset parameter may be added to the parameterized model, and the model parameters of the adjusted parameterized model may include the offset parameter. In one embodiment, the model parameters of the adjusted parameterized model may also include at least one of a global feature parameter, a posture parameter, and a shape parameter. Among them, the offset parameter is a model parameter used to characterize the offset of the surface vertices of the parameterized model. The global feature parameter is a model parameter used to characterize the global features of the parameterized model. The posture parameter is a model parameter used to characterize the posture of the parameterized model. The shape parameter is a model parameter used to characterize the shape of the parameterized model.

[0128] In one embodiment, the computer device can adjust the model parameters used to characterize the features of the parameterized model at the corresponding level in order from global to local, so as to align the mesh surface in the parameterized model with the three-dimensional mesh surface to obtain the final target parameterized model of the target object. For example: the computer device can first adjust the global feature parameters (corresponding to the global level), then adjust the shape parameters and posture parameters (corresponding to the level between the global and local), and finally adjust the offset parameters (corresponding to the local level). It can be understood that after each adjustment of the model parameters used to characterize the features of a certain level, the parameter values ​​of the model parameters corresponding to the adjusted level are fixed, and then the model parameters corresponding to the next level are adjusted. In this embodiment, the model parameters are adjusted in order from global to local, which can avoid the problem of excessive local deformation caused by directly adjusting the model parameters of the local level, improve the accuracy of the registration, and then improve the accuracy of the three-dimensional reconstruction.

[0129] In one embodiment, the computer device may obtain a pre-constructed target loss function, and iteratively adjust the model parameters of the parameterized model in a direction that minimizes the target loss function, so as to align the mesh surface in the parameterized model with the three-dimensional mesh surface.

[0130] In one embodiment, in the process of adjusting different model parameters by the computer device, the target loss functions used respectively may be the same or different. That is, the corresponding target loss function may be designed according to the difference in the characteristics of the parameterized model represented by the adjusted model parameters.

[0131] In one embodiment, the target loss function may include at least one of a mesh difference loss function, a surface topology loss function, a deformation loss function, and the like.

[0132] like Figure 4 As shown, the computer device can Figure 3 The parameterized model 308 created in the above is registered with the three-dimensional mesh surface of the three-dimensional continuous surface 306, and the result is as follows: Figure 4 The target parameterized model 402 is shown. As can be seen from the figure, the target parameterized model 402 has the detail information of a three-dimensional continuous surface, avoiding the limitation of the parameterized model in expressing the shape of the target object and lacking detail information, and improving the accuracy of the three-dimensional reconstruction of the target object. In one embodiment, the computer device can obtain the texture map of the target object, and then perform texture rendering on the target parameterized model according to the texture map to obtain the target parameterized model after texture rendering. The texture map is used to characterize the high-frequency details and color information of the surface of the parameterized model.

[0133] In one embodiment, the computer device may acquire multiple frames of two-dimensional images including the target object, and then obtain a texture map of the target object based on the multiple frames of two-dimensional images.

[0134] In one embodiment, the computer device may drive the target parameterized model to perform an action according to the action parameter, wherein the action parameter is a model parameter used to characterize the action of the target parameterized model.

[0135] In the above-mentioned target object three-dimensional reconstruction method, a three-dimensional parameterized model of the target object is created according to the two-dimensional image of the target object, and a three-dimensional continuous surface of the target object is generated according to the two-dimensional image, and the three-dimensional continuous surface is meshed to obtain a three-dimensional mesh surface, and then the model parameters of the parameterized model are adjusted to align the mesh surface in the parameterized model with the three-dimensional mesh surface to obtain the final target parameterized model of the target object. Because the three-dimensional continuous surface has the detail information of the target object, the target parameterized model obtained by aligning the mesh surface in the parameterized model with the three-dimensional mesh surface has a stronger ability to express the shape of the target object, and can have the detail information of the target object, such as: the detail information of the target human body's clothes, hair or shoes, etc., avoiding the limitation that the parameterized model of the target object obtained by the three-dimensional reconstruction of the parameterized model has limited expression of the shape of the target object and lacks the limitation of detail information, thereby improving the accuracy of the three-dimensional reconstruction of the target object. In addition, the parameterized model often has ambiguity in depth. For example, the legs of the parameterized model of a target person with a shorter height are often incomplete. The target parameterized model obtained by the method of the present application also avoids ambiguity in depth and improves the accuracy of three-dimensional reconstruction.

[0136] In addition, the three-dimensional model of the target object modeled by the three-dimensional continuous surface cannot be driven by motion and has limitations, while the parametric model can achieve motion driving. Therefore, the model parameters of the parametric model are adjusted, and the mesh surface in the parametric model is registered with the three-dimensional mesh surface extracted from the three-dimensional continuous surface to obtain the target parametric model, which can have the advantage of the parametric model being able to be driven by motion, avoiding the limitation that the three-dimensional continuous surface cannot be driven, so that the target parametric model has both detailed information, improves the accuracy of three-dimensional reconstruction, and can be driven by motion. Because the model can be flexibly driven by motion, the applicability of the target parametric model obtained by three-dimensional reconstruction is improved, and the scope of application is broadened.

[0137] In one embodiment, the method further includes: adding an offset parameter to the parameterized model to obtain a deformable parameterized model. In this embodiment, step 210 of adjusting the model parameters of the parameterized model to register the mesh surface in the parameterized model with the three-dimensional mesh surface includes: adjusting the offset parameters corresponding to the surface vertices of the deformable parameterized model to register the mesh surface in the parameterized model with the three-dimensional mesh surface.

[0138] The deformable parameterized model refers to a parameterized model in which the parameter values ​​of the offset parameters of the surface vertices can be adjusted, that is, a parameterized model in which the surface vertices can be offset.

[0139] In one embodiment, the computer device may adjust the offset parameters corresponding to the surface vertices of the deformable parameterized model to align the mesh surface in the parameterized model with the three-dimensional mesh surface to obtain the final target parameterized model of the target object.

[0140] In one embodiment, before adjusting the offset parameters corresponding to the surface vertices of the deformable parameterized model, the computer device may first adjust the model parameters of the global level of the deformable parameterized model to register the mesh surface in the parameterized model with the three-dimensional mesh surface, and then fix the adjusted model parameters of the global level, and adjust the offset parameters corresponding to the surface vertices of the deformable parameterized model to register the mesh surface in the parameterized model with the three-dimensional mesh surface to obtain the final target parameterized model of the target object. For example: the computer device may first adjust the global feature parameters, then adjust the shape parameters and the posture parameters, and then adjust the offset parameters. In this embodiment, adjusting the model parameters at the global level first, and then adjusting the offset parameters corresponding to the surface vertices, can avoid the problem of excessive local offset caused by directly adjusting the offset parameters, improve the accuracy of the registration, and then improve the accuracy of the three-dimensional reconstruction.

[0141] In one embodiment, a computer device can determine the surface vertices corresponding to the incomplete parts expressed by the three-dimensional continuous surface based on the surface vertex semantic segmentation results of the parameterized model, and then fix the offset parameters of the determined surface vertices, and adjust the offset parameters corresponding to the surface vertices other than the determined surface vertices in the deformable parameterized model to align the mesh surface in the parameterized model with the three-dimensional mesh surface.

[0142] The surface vertex semantic segmentation result is used to characterize the part of the target object to which the surface vertex in the parameterized model belongs. For example, if the target object is a human body, the surface vertex semantic segmentation result may include at least one of face, hand, foot, and torso. That is, for example, if the semantic segmentation result of the surface vertex is a hand, it means that the surface vertex belongs to the hand of the target human body.

[0143] It can be understood that the three-dimensional continuous surface does not fully express certain areas in the target object, and these areas are the parts that are incompletely expressed by the three-dimensional continuous surface.

[0144] In the above embodiment, a deformable parameterized model is obtained by adding an offset parameter to the parameterized model, and the offset parameter corresponding to the surface vertices of the deformable parameterized model is adjusted to align the mesh surface in the parameterized model with the three-dimensional mesh surface. This can make the alignment of the mesh surface in the parameterized model with the three-dimensional mesh surface more accurate, achieve the alignment of local detail features, and enable the target parameterized model to have the detail information of a three-dimensional continuous surface, thereby avoiding the limitations of the parameterized model in expressing the shape of the target object and lacking detail information, and improving the accuracy of three-dimensional reconstruction of the target object.

[0145] In one embodiment, the model parameters of the deformable parameterized model also include global feature parameters, posture parameters and shape parameters. In this embodiment, adjusting the offset parameters corresponding to the surface vertices of the deformable parameterized model to register the mesh surface in the parameterized model with the three-dimensional mesh surface includes: adjusting the global feature parameters of the deformable parameterized model to perform depth registration of the mesh surface in the parameterized model with the three-dimensional mesh surface; adjusting the posture parameters and shape parameters of the parameterized model after the depth registration to perform coincidence registration of the mesh surface in the parameterized model with the three-dimensional mesh surface; adjusting the offset parameters corresponding to the surface vertices of the parameterized model after the coincidence registration to perform feature part registration of the mesh surface in the parameterized model with the three-dimensional mesh surface.

[0146] Among them, depth registration refers to the registration of the mesh surface in the parameterized model with the 3D mesh surface in the depth direction. Coincidence registration refers to the registration of the mesh surface in the parameterized model with the 3D mesh surface in the direction of coincidence. Feature part registration refers to the registration of the mesh surface in the parameterized model with the 3D mesh surface at the local detail feature parts.

[0147] In one embodiment, the global characteristic parameters may include a global direction parameter and a global transformation parameter of the parameterized model. The global direction parameter is a model parameter used to characterize the overall rotation direction of the parameterized model. The global transformation parameter is a model parameter used to characterize the overall displacement of the parameterized model.

[0148] In one embodiment, a computer device may first adjust the global feature parameters of a deformable parameterized model to perform depth alignment of a mesh surface in the parameterized model with a three-dimensional mesh surface, then fix the parameter values ​​of the adjusted global feature parameters, adjust the posture parameters and shape parameters of the parameterized model after the depth alignment to perform coincident alignment of the mesh surface in the parameterized model with the three-dimensional mesh surface, then fix the parameter values ​​of the adjusted global feature parameters, posture parameters and shape parameters, adjust the offset parameters corresponding to the surface vertices of the coincidently aligned parameterized model to perform feature part alignment of the mesh surface in the parameterized model with the three-dimensional mesh surface, and obtain a final target parameterized model of the target object.

[0149] In the above embodiment, because the parameterized model created based on the two-dimensional image often has depth ambiguity, resulting in a large deviation in the depth direction between the parameterized model and the three-dimensional mesh surface of the three-dimensional continuous surface, therefore, the global feature parameters of the deformable parameterized model are first adjusted, so that the parameterized model and the three-dimensional mesh surface can be aligned in the global depth to avoid the deviation in the depth direction. Then, by adjusting the shape parameters and posture parameters, the shape and posture of the parameterized model are adjusted, so that the parameterized model and the three-dimensional mesh surface can overlap. After adjusting the global feature parameters, shape parameters and posture parameters, the offset parameters are adjusted, which can avoid the problem of excessive local offset caused by directly adjusting the offset parameters, improve the accuracy of the registration, and then improve the accuracy of the three-dimensional reconstruction.

[0150] In one embodiment, adjusting the offset parameters corresponding to the surface vertices of a deformable parametric model to align the mesh surface in the parametric model with the three-dimensional mesh surface includes: obtaining the surface vertex semantic segmentation result of the parametric model; determining the surface vertices whose geometric shapes change dramatically in the deformable parametric model based on the surface vertex semantic segmentation result; fixing the offset parameters corresponding to the determined surface vertices, and adjusting the offset parameters corresponding to the surface vertices in the deformable parametric model other than the determined surface vertices, to align the mesh surface in the parametric model with the three-dimensional mesh surface.

[0151] Specifically, the target semantic segmentation results corresponding to the surface vertices with drastic changes in geometric shapes can be set in advance. The computer device can determine the surface vertices corresponding to the target semantic segmentation results based on the semantic segmentation results corresponding to each surface vertices of the parameterized model, and use the determined surface vertices as the surface vertices with drastic changes in geometric shapes.

[0152] Among them, the target semantic segmentation result refers to the semantic segmentation result corresponding to the surface vertices with drastic geometric shape changes.

[0153] It can be understood that because the three-dimensional continuous surface has problems such as incomplete or ambiguous expression of the locations where the geometric shapes of the target objects change dramatically, the offset parameters of the surface vertices with drastic geometric shapes in the parametric model are fixed, and only the offset parameters corresponding to the surface vertices other than the surface vertices with drastic geometric shapes are adjusted to align the mesh surface in the parametric model with the three-dimensional mesh surface, which can avoid the above-mentioned limitations of the three-dimensional continuous surface.

[0154] For example, if the target object is a human body, the target semantic segmentation result may include at least one of the face and hands. It is understandable that because the geometric shapes of the face and hands of the human body change dramatically, at least one of the face and hands is preset as the target semantic segmentation result. In the three-dimensional continuous surface of the target human body, the face and hands are prone to incomplete or blurred expressions. Figure 3 and Figure 4 As shown in Figure 1, the three-dimensional continuous surface of the hand has an incomplete problem, and the face has an expression ambiguity problem. Figure 4 As shown, the target parameterized model obtained after the registration processing under the constraint of the surface vertex semantic segmentation result 404 of the parameterized model retains the integrity of the hands and face in the original parameterized model and avoids being affected by the above-mentioned limitations of the three-dimensional continuous surface.

[0155] In one embodiment, the computer device may fix the parameter value of the offset parameter corresponding to the determined surface vertex to 0, and adjust the offset parameter corresponding to the surface vertices other than the determined surface vertex in the deformable parametric model to align the mesh surface in the parametric model with the three-dimensional mesh surface.

[0156] In the above embodiment, the computer device can fix the offset parameters of the surface vertices whose geometric shapes change drastically based on the semantic segmentation results of the surface vertices of the parameterized model, and only adjust the offset parameters corresponding to the surface vertices other than the determined surface vertices, so as to align the mesh surface in the parameterized model with the three-dimensional mesh surface, thereby avoiding the problems of incomplete or ambiguous expression of the three-dimensional continuous surface at the position where the geometric shape changes drastically, so that the target parameterized model can completely reconstruct the position where the geometric shape changes drastically, thereby improving the accuracy of three-dimensional reconstruction.

[0157] In one embodiment, the two-dimensional image is a single-frame two-dimensional image, and the parameterized model created based on the single-frame two-dimensional image is a first parameterized model. In this embodiment, obtaining the surface vertex semantic segmentation result of the parameterized model includes: obtaining multiple frames of two-dimensional images including the target object; obtaining a second parameterized model of the target object created corresponding to each frame of the two-dimensional image in the multiple frames of two-dimensional images; performing semantic segmentation on each frame of the two-dimensional image in the multiple frames of two-dimensional images to obtain the partially visible surface vertex semantic segmentation result of the corresponding second parameterized model; determining the surface vertex semantic segmentation result of the first parameterized model based on the partially visible surface vertex semantic segmentation results of each second parameterized model.

[0158] The partially visible surface vertex semantic segmentation result refers to the semantic segmentation result of only a part of the surface vertices, while the speech segmentation result of the other part of the surface vertices is missing.

[0159] In one embodiment, the multiple frames of two-dimensional images may include the single frame of two-dimensional image, or may not include the single frame of two-dimensional image.

[0160] In one embodiment, multiple frames of two-dimensional images including the target object may be acquired by using the methods in the aforementioned embodiments of acquiring multiple frames of two-dimensional images.

[0161] In one embodiment, the computer device can use the method in each embodiment of the aforementioned method of creating a three-dimensional parameterized model of a target object according to a two-dimensional image to create a second parameterized model of the target object according to each two-dimensional image in the plurality of two-dimensional images. A second parameterized model is created for each two-dimensional image.

[0162] In one embodiment, the computer device may perform semantic segmentation on each of the multiple two-dimensional image frames to obtain semantic segmentation results corresponding to each of the two-dimensional image frames. Then, the computer device may determine the semantic segmentation results of the partially visible surface vertices of the corresponding second parameterized model according to the semantic segmentation results corresponding to each of the two-dimensional image frames. That is, for each two-dimensional image frame, the computer device may determine the semantic segmentation results of the partially visible surface vertices of the second parameterized model corresponding to the two-dimensional image frame according to the semantic segmentation results of the two-dimensional image frame.

[0163] In one embodiment, the computer device may use a Human Parsing Network (RP-R-CNN, a deep learning network for semantic segmentation of images) to perform semantic segmentation on each frame of the two-dimensional image.

[0164] It can be understood that because the two-dimensional image can only express the image of the target object in a certain direction, the surface vertex semantic segmentation result of the second parameterized model obtained based on the two-dimensional image is partially visible, and the complete surface vertex semantic segmentation result of the parameterized model cannot be obtained.

[0165] In one embodiment, the computer device may perform voting fusion based on the semantic segmentation results in each second parameterized model corresponding to each surface vertex of the first parameterized model, and determine the semantic segmentation results of the surface vertices of the first parameterized model. Specifically, for the surface vertices of the first parameterized model, the computer device may correspond the surface vertex to each semantic segmentation result in each second parameterized model as a candidate semantic segmentation result of the surface vertex, and then determine the final semantic segmentation result of the surface vertex based on each candidate semantic segmentation result. It can be understood that after obtaining the final semantic segmentation results of each surface vertices in the first parameterized model, the semantic segmentation results of the surface vertices of the first parameterized model are obtained.

[0166] In one embodiment, the computer device may select the final semantic segmentation result of the surface vertex from each candidate semantic segmentation result according to the frequency of occurrence of each candidate semantic segmentation result.

[0167] In one embodiment, the computer device may determine the candidate semantic segmentation result with the highest occurrence frequency as the final semantic segmentation result of the surface vertex.

[0168] For example: the surface vertex A of the first parameterized model corresponds to the semantic segmentation results of the second parameterized models B, C and D, which are hand, arm and hand respectively. Since "hand" has the highest frequency of occurrence, "hand" can be determined as the semantic segmentation result of surface vertex A.

[0169] like Figure 5 As shown, the computer device performs semantic segmentation on multiple frames of two-dimensional images, obtains the semantic segmentation results corresponding to each frame of the two-dimensional image, and creates a second parameterized model according to each frame of the two-dimensional image. Then, the computer device performs voting fusion according to the semantic segmentation results corresponding to each frame of the two-dimensional image and the corresponding second parameterized model to obtain the surface vertex semantic segmentation results of the first parameterized model. Each surface vertex can be represented by a different color according to the different semantic segmentation results. In order to clearly illustrate, Figure 5 Different numbers are used to mark the regions of different semantic segmentation results in the first parameterized model.

[0170] In the above embodiment, the computer device determines the semantic segmentation results of the partially visible surface vertices of the corresponding second parameterized models according to the semantic segmentation results corresponding to the multiple frames of two-dimensional images, and then determines the semantic segmentation results of the surface vertices of the first parameterized model according to the semantic segmentation results of the partially visible surface vertices of each second parameterized model. It can be understood that the semantic segmentation results of the surface vertices in each second parameterized model are not complete, but partially visible, and then the semantic segmentation results of the surface vertices of the first parameterized model obtained according to the semantic segmentation results of the surface vertices in each second parameterized model are obtained by integrating the semantic segmentation results of the partially visible surface vertices in each second parameterized model, so that the semantic segmentation results of the surface vertices of the parameterized model are more complete and the accuracy of the semantic segmentation results of the surface vertices of the parameterized model is improved.

[0171] In one embodiment, after determining the semantic segmentation results of the surface vertices of the first parameterized model based on the semantic segmentation results of the surface vertices of each second parameterized model, the method also includes: determining the invisible surface vertices for which the semantic segmentation results have not been determined in the first parameterized model; determining the visible surface vertices within a preset neighborhood range of the invisible surface vertices; and determining the semantic segmentation results of the invisible surface vertices based on the determined semantic segmentation results of the visible surface vertices.

[0172] The invisible surface vertex is a surface vertex for which the semantic segmentation result is not determined in the first parameterized model. The visible surface vertex is a surface vertex for which the semantic segmentation result is determined in the first parameterized model. The preset neighborhood range is a preset range adjacent to the surface vertex.

[0173] For example, since it is difficult to capture the sole of the target object in the multi-frame two-dimensional image of the target object, the surface vertex at the sole of the target object is an invisible surface vertex.

[0174] Specifically, after determining the semantic segmentation results of the surface vertices of the first parameterized model based on the semantic segmentation results of the surface vertices of each second parameterized model, the computer device can determine the invisible surface vertices for which the semantic segmentation results have not been determined in the first parameterized model, and then determine the visible surface vertices for which the semantic segmentation results have been determined within a preset neighborhood range of the invisible surface vertices, and then determine the semantic segmentation results of the invisible surface vertices based on the determined semantic segmentation results of the visible surface vertices.

[0175] In one embodiment, the preset neighborhood range may be an area within a preset neighborhood radius. The preset neighborhood radius refers to the radius of the preset neighborhood range. For example, the preset neighborhood radius is 3.

[0176] In one embodiment, the computer device may select the semantic segmentation results of the invisible surface vertices from the determined semantic segmentation results of the visible surface vertices according to the occurrence frequency of the determined semantic segmentation results of the visible surface vertices.

[0177] In one embodiment, the computer device may select the semantic segmentation result with the highest occurrence frequency from the determined semantic segmentation results of the visible surface vertices as the semantic segmentation result of the invisible surface vertices.

[0178] For example, the visible surface vertices within the preset neighborhood of the invisible surface vertex A are B, C, and D, and the semantic segmentation results of B, C, and D are hand, face, and hand, respectively. Since the semantic segmentation result of "hand" appears most frequently, "hand" can be determined as the semantic segmentation result of the invisible surface vertex A.

[0179] In the above embodiment, the semantic segmentation results of the invisible surface vertices are determined according to the semantic segmentation results of the visible surface vertices within the preset neighborhood of the invisible surface vertices, thereby avoiding the problem of invisible surface vertices in the semantic segmentation results of the surface vertices of the parameterized model. The integrity of the semantic segmentation results of the surface vertices of the parameterized model is improved, and the accuracy of the semantic segmentation results of the surface vertices of the parameterized model is improved.

[0180] In one embodiment, step 210 adjusts the model parameters of the parameterized model to align the mesh surface in the parameterized model with the three-dimensional mesh surface, including: obtaining a target loss function with multiple constraints; the target loss function includes a mesh difference loss function, a surface topology structure loss function and a deformation loss function; iteratively adjusting the model parameters of the parameterized model in the direction of minimizing the target loss function to align the mesh surface in the parameterized model with the three-dimensional mesh surface.

[0181] Among them, the mesh difference loss function is a loss function used to characterize the mesh difference between the mesh surface in the parameterized model and the three-dimensional mesh surface. The surface topology loss function is a loss function used to maintain the topological structure of the surface of the parameterized model during deformation. The deformation loss function is a loss function used to constrain the deformation amplitude of the parameterized model. It can be understood that the deformation loss function can be used to constrain the deformation amplitude of the parameterized model to prevent the parameterized model from deforming too much.

[0182] In one embodiment, the target loss function used in the deep registration process includes a mesh difference loss function. The computer device can iteratively adjust the global feature parameters of the parameterized model in a direction that minimizes the target loss function including the mesh difference loss function to perform deep registration of the mesh surface in the parameterized model with the three-dimensional mesh surface.

[0183] In one embodiment, the objective loss function used in the coincidence registration process includes a mesh difference loss function. The computer device can iteratively adjust the pose parameters and shape parameters of the parameterized model in a direction that minimizes the objective loss function including the mesh difference loss function to coincidentally register the mesh surface in the parameterized model with the three-dimensional mesh surface.

[0184] In one embodiment, the target loss function used in the feature part registration process includes a mesh difference loss function, a surface topology loss function, and a deformation loss function. The computer device can iteratively adjust the offset parameters corresponding to the surface vertices of the parameterized model in a direction that minimizes the target loss function including the mesh difference loss function, the surface topology loss function, and the deformation loss function, so as to perform feature part registration on the mesh surface in the parameterized model and the three-dimensional mesh surface.

[0185] In one embodiment, the mesh difference loss function may include the sum of the minimum distances from each surface vertex in the parameterized model to the three-dimensional mesh surface divided by the three-dimensional continuous surface, and the sum of the minimum distances from each surface vertex of the three-dimensional mesh surface divided by the three-dimensional continuous surface to the mesh surface of the parameterized model. In this embodiment, by bidirectionally optimizing the minimum distances from each vertex of the two models (i.e., the parameterized model and the three-dimensional mesh surface divided by the three-dimensional continuous surface) to each other's surface, the two models are overlapped, thereby improving the accuracy of the target parameterized model and thus improving the accuracy of the three-dimensional reconstruction.

[0186] In one embodiment, the grid difference loss function is expressed as follows:

[0187]

[0188] Among them, L p2s Represents the mesh difference loss function, S represents the surface vertex of the parameterized model, p∈S represents p is the surface vertex of the parameterized model, M represents the patch of the mesh surface of the parameterized model, and f∈M represents f is the patch of the mesh surface of the parameterized model. The surface vertices representing the three-dimensional mesh surface divided by the three-dimensional continuous surface, Indicates that q is the surface vertex of the three-dimensional mesh surface divided by the three-dimensional continuous surface, Represents the face of a three-dimensional mesh surface divided by a three-dimensional continuous surface. express It is a patch of a three-dimensional mesh surface divided by a three-dimensional continuous surface. Indicates p to The minimum distance is the minimum distance from the surface vertex of the parameterized model to the three-dimensional mesh surface divided by the three-dimensional continuous surface. Represents the sum of the minimum distances from each surface vertex in the parameterized model to the three-dimensional mesh surface divided by the three-dimensional continuous surface. It represents the minimum distance from q to f, that is, the minimum distance from the surface vertex of the three-dimensional mesh surface divided by the three-dimensional continuous surface to the mesh surface of the parameterized model. It represents the sum of the minimum distances from each surface vertex of the three-dimensional mesh surface divided by the three-dimensional continuous surface to the mesh surface of the parameterized model.

[0189] In one embodiment, the surface topology loss function may include a surface topology difference loss function and an intrinsic surface topology loss function. The surface topology difference loss function is a loss function used to characterize the change in the surface topology of the parameterized model before and after the model parameters are adjusted. The intrinsic surface topology loss function is a loss function used to characterize the surface topology of the parameterized model itself after the model parameters are adjusted.

[0190] In one embodiment, the surface topology difference loss function can be a regularization of the difference in Laplace values ​​of the mesh surface of the parameterized model before and after the model parameters are adjusted. The Laplace value is used to characterize the local detail features of the mesh surface. In this embodiment, by constraining the difference in Laplace values ​​of the mesh surface of the parameterized model before and after the model parameters are adjusted, the situation in which the surface topology structure of the parameterized model changes too much due to the registration process is avoided, the rationality and accuracy of the target parameterized model are improved, and thus the accuracy of the three-dimensional reconstruction is improved.

[0191] In one embodiment, the surface topology difference loss function can be expressed by the following formula (2):

[0192]

[0193] in, represents the surface topology difference loss function, p represents the surface vertex of the parameterized model, δ p represents the Laplace value at the surface vertex p of the parameterized model after adjusting the model parameters, δ' p Represents the Laplace value at the surface vertex p of the parameterized model before adjusting the model parameters. Laplace value δ p It can be expressed by the following formula (3):

[0194]

[0195] Among them, N(p) represents the neighbor surface vertex in the neighborhood of surface vertex p, and k∈N(p) represents that k is the neighbor surface vertex in the neighborhood of surface vertex p. It can be understood that δ' p It can also be obtained by using the calculation method of formula (3) above.

[0196] In one embodiment, the surface topology loss function itself can be the regularization of the Laplace value of the mesh surface of the parameterized model after adjusting the model parameters. In this embodiment, by constraining the Laplace value of the mesh surface of the parameterized model after adjusting the model parameters, the drastic change of the surface topology of the target parameterized model is avoided, the surface smoothness of the target parameterized model is enhanced, the rationality and accuracy of the target parameterized model are improved, and the accuracy of the three-dimensional reconstruction is improved.

[0197] In one embodiment, the surface topology loss function itself can be expressed by the following formula (4):

[0198]

[0199] Among them, L lap represents the surface topology loss function itself, δ p Represents the Laplace value of the mesh surface of the parameterized model after adjusting the model parameters. Laplace value δ p The same can be expressed by formula (3).

[0200] In one embodiment, the deformation loss function may be a regularization of the parameter value of the offset parameter of the surface vertices of the parameterized model. In this embodiment, by constraining the parameter value of the offset parameter of the surface vertices of the parameterized model, excessive offset of the surface vertices is avoided, the rationality and accuracy of the target parameterized model are improved, and the accuracy of the three-dimensional reconstruction is improved.

[0201] In one embodiment, the deformation loss function can be expressed as follows:

[0202]

[0203] Among them, L d represents the deformation loss function, p represents the surface vertex of the parameterized model, and d p The parameter value representing the offset parameter of the surface vertex p of the parameterized model after adjusting the model parameters.

[0204] In one embodiment, the objective loss function can be expressed as follows:

[0205]

[0206] Among them, L p2s represents the grid difference loss function, represents the surface topology difference loss function, L lap Represents the surface topology loss function, L d Denotes the deformation loss function.p2s , λ lap and λ d They are the weights of each loss function in the target loss function. Each weight can be set according to the actual situation.

[0207] In one embodiment, each weight can be set to λ p2s =1×10 2 , λ lap =1×10 3 and λ d =1×10 1 .

[0208] In one embodiment, the computer device may iteratively adjust the model parameters of the parameterized model in a direction that minimizes the target loss function through an Adam optimizer (an algorithm for iteratively optimizing model parameters) to register the mesh surface in the parameterized model with the three-dimensional mesh surface. In one embodiment, the learning rate of the process of iteratively adjusting the model parameters of the parameterized model may be set to lr=1×10 -2 .

[0209] In one embodiment, the computer device can implement each loss function in the target loss function through the Kaolin package (an open source tool library applied to deep learning).

[0210] In the above-mentioned embodiments, by iteratively adjusting the model parameters of the parameterized model in the direction of minimizing the target loss function of multiple constraints to align the mesh surface in the parameterized model with the three-dimensional mesh surface, the accuracy of the three-dimensional reconstruction can be improved, and the efficiency of the three-dimensional reconstruction can be improved. Specifically, by constructing a mesh difference loss function, the two models are made to overlap, the accuracy of the target parameterized model is improved, and thus the accuracy of the three-dimensional reconstruction is improved. By constructing a surface topology loss function, the rationality and accuracy of the surface topology of the target parameterized model are improved, and the accuracy of the three-dimensional reconstruction is improved. By constructing a deformation loss function, the parameter value of the offset parameter of the surface vertex of the parameterized model is constrained, so that the excessive offset of the surface vertex is avoided, the rationality and accuracy of the target parameterized model are improved, and the accuracy of the three-dimensional reconstruction is improved.

[0211] In one embodiment, the two-dimensional image is a single-frame two-dimensional image, and the parameterized model created based on the single-frame two-dimensional image is a first parameterized model. In this embodiment, the method further includes: obtaining multiple frames of two-dimensional images including the target object; obtaining second parameterized models of the target object created corresponding to each frame of the two-dimensional image in the multiple frames of the two-dimensional image; mapping the points corresponding to the target object in the corresponding two-dimensional image to the texture space according to the texture coordinates corresponding to the surface vertices of each second parameterized model, and obtaining the initial texture map of the target object corresponding to each frame of the two-dimensional image; fusing the initial texture maps to obtain the texture map of the target object; and texture rendering the target parameterized model according to the texture map.

[0212] Among them, texture coordinates (UV coordinates) are the coordinates corresponding to the surface vertices of the parameterized model on the texture map. Texture space refers to the coordinate space based on the texture map. The initial texture map refers to the texture map obtained based on a single-frame two-dimensional image in multiple frames of two-dimensional images. It can be understood that since the initial texture map is obtained based on a single-frame two-dimensional image, the initial texture map is not a complete texture map of the parameterized model.

[0213] In one embodiment, the multiple frames of two-dimensional images may include the single frame of two-dimensional image, or may not include the single frame of two-dimensional image.

[0214] In one embodiment, multiple frames of two-dimensional images including the target object may be acquired by using the methods in the aforementioned embodiments of acquiring multiple frames of two-dimensional images.

[0215] In one embodiment, the computer device can use the method in each embodiment of the aforementioned method of creating a three-dimensional parameterized model of a target object according to a two-dimensional image to create a second parameterized model of the target object according to each two-dimensional image in the plurality of two-dimensional images. A second parameterized model is created for each two-dimensional image.

[0216] In one embodiment, a computer device can map points corresponding to the target object in the corresponding two-dimensional image to the texture space based on the texture coordinates corresponding to the surface vertices of each second parameterized model and the correspondence between the surface vertices of the second parameterized model and the points in the corresponding two-dimensional image, and obtain the texture coordinates of the points corresponding to the target object in the corresponding two-dimensional image, thereby forming an initial texture map of the target object corresponding to each frame of the two-dimensional image.

[0217] like Figure 6 As shown, for each frame of the two-dimensional image, according to the two-dimensional image and the texture coordinates corresponding to the surface vertices of the corresponding second parameterized model (such as Figure 6602), mapping the point corresponding to the target object in the two-dimensional image to the texture space (i.e., texture expansion), obtaining the texture coordinates of the point corresponding to the target object in the corresponding two-dimensional image, thereby obtaining the texture coordinates corresponding to the two-dimensional image. Figure 6 The initial texture map of the target object shown in 604.

[0218] In one embodiment, the computer device may sort the initial texture maps, and then sequentially merge the initial texture maps in the sorted order to obtain the texture map of the target object.

[0219] In one embodiment, the computer device may sort the initial texture maps according to the global directions of the second parameterized models, wherein the global directions refer to the parameter values ​​of the global direction parameters.

[0220] In one embodiment, the computer device may perform texture rendering on the target parameterized model according to the texture map of the target object to obtain the texture-rendered target parameterized model.

[0221] In one embodiment, the computer device may perform texture rendering on the target parameterized model according to the texture map of the target object through Neural Render (an algorithm for performing texture rendering) to obtain the texture-rendered target parameterized model.

[0222] In one embodiment, the computer device may first segment the foreground (i.e., the target object) from each frame of the two-dimensional image, and then obtain the initial texture map based on the segmented two-dimensional image. In one embodiment, the computer device may use a foreground segmentation model to segment the foreground (i.e., the target object) from multiple frames of two-dimensional images. In this embodiment, by segmenting the foreground, the background in the two-dimensional image is removed, thereby avoiding the problem of background information appearing in the obtained initial texture map, and improving the accuracy of the obtained texture map.

[0223] In one embodiment, the computer device may first perform de-lighting processing on each frame of the two-dimensional image to remove the illumination in the two-dimensional image, thereby improving the accuracy and clarity of the texture map. In one embodiment, the computer device may perform de-lighting processing on each frame of the two-dimensional image using a de-lighting model.

[0224] In the above embodiment, based on multiple frames of two-dimensional images and the corresponding second parameterized model, initial texture maps corresponding to each frame of the two-dimensional image are obtained, and then the initial texture maps are fused to obtain an accurate texture map of the target object. Then, the target parameterized model is texture rendered according to the texture map to obtain a target parameterized model after texture rendering, so that the target parameterized model contains richer information, thereby improving the applicability of the reconstructed target parameterized model.

[0225] In one embodiment, fusing the initial texture maps to obtain the texture map of the target object includes: determining a fusion order corresponding to each frame of two-dimensional image according to the root node direction of each second parameterized model; and fusing the initial texture maps corresponding to each frame of two-dimensional image according to the fusion order to obtain the texture map of the target object.

[0226] The root node direction is used to represent the global direction of the parametric model. The root node is a node on the key point skeleton of the parametric model. The rotation and translation of the root node represent the rotation and translation of the entire parametric model.

[0227] In one embodiment, the computer device may sequentially sort the multiple frames of two-dimensional images according to the root node directions of the second parameterized models to obtain a fusion order corresponding to each frame of the two-dimensional image.

[0228] In one embodiment, the computer device can group each frame of two-dimensional image according to the root node direction of the corresponding second parameterized model and according to each preset main direction, that is, divide the two-dimensional image into a group of the main direction closest to the root node direction. Then, in each group, the two-dimensional images of the group are sorted according to the consistency between the root node direction and the main direction of the group to obtain the fusion order corresponding to each frame of two-dimensional image. Among them, consistency is used to characterize the closeness between the root node direction and the main direction.

[0229] In one embodiment, in one embodiment, the main directions may include four directions: front, back, left and right.

[0230] In one embodiment, the computer device may fuse the initial texture maps corresponding to the two-dimensional image frames in sequence according to the fusion order corresponding to the two-dimensional image frames, so as to obtain the texture map of the target object.

[0231] In one embodiment, a computer device may first determine the visibility of each point on an initial texture map corresponding to a two-dimensional image, and then select a current two-dimensional image from the first two-dimensional image in a fusion order, and generate a texture map corresponding to the current two-dimensional image according to the initial texture map corresponding to the current two-dimensional image and the visibility of each point; fuse the texture map corresponding to the current two-dimensional image with the accumulated fused texture maps; after fusion, use the next two-dimensional image as the current two-dimensional image in a fusion order, and iterate back to the step of generating a texture map corresponding to the current two-dimensional image according to the initial texture map corresponding to the current two-dimensional image and the visibility map to continue execution until the iteration stops, and the texture map of the target object is obtained.

[0232] In one embodiment, the computer device may weight each point on the initial texture map corresponding to the current two-dimensional image by the visibility of each point on the initial texture map to generate a texture map corresponding to the current two-dimensional image.

[0233] In one embodiment, for each frame of a two-dimensional image, a computer device may determine the visibility of each point on an initial texture map corresponding to the two-dimensional image based on the direction of the normal vectors of each surface vertex of the corresponding second parameterized model and the proximity between the direction of the shooting direction of the two-dimensional image.

[0234] In the above embodiment, the fusion order corresponding to each frame of the two-dimensional image is determined according to the root node direction of each second parameterized model, and then the initial texture maps corresponding to each frame of the two-dimensional image are fused according to the fusion order to obtain the texture map of the target object, thereby improving the efficiency of the fusion to obtain the texture map and improving the accuracy of the obtained texture map.

[0235] In one embodiment, the method further includes: obtaining the visibility map of the target object corresponding to each frame of the two-dimensional image. In this embodiment, the initial texture maps corresponding to each frame of the two-dimensional image are fused in a fusion order to obtain the texture map of the target object, including: selecting the current two-dimensional image from the first two-dimensional image in a fusion order, generating the texture map corresponding to the current two-dimensional image according to the initial texture map and visibility map corresponding to the current two-dimensional image; fusing the texture map corresponding to the current two-dimensional image with the accumulated fused texture maps; after fusion, taking the next two-dimensional image as the current two-dimensional image in a fusion order, iterating and returning to the step of generating the texture map corresponding to the current two-dimensional image according to the initial texture map and visibility map corresponding to the current two-dimensional image to continue to execute until the iteration stops, and obtaining the texture map of the target object.

[0236] The visibility map is used to represent the visibility of each point on the initial texture map corresponding to the two-dimensional image. The accumulated fused texture map is the texture map accumulated and fused before the initial texture map corresponding to the current two-dimensional image is fused.

[0237] Specifically, the computer device can first use the first two-dimensional image as the current two-dimensional image, and then generate a texture map corresponding to the current two-dimensional image based on the initial texture map and visibility map corresponding to the current two-dimensional image, and then fuse the texture map corresponding to the current two-dimensional image with the accumulated fused texture maps. After the fusion, the next two-dimensional image is used as the current two-dimensional image in the fusion order, and iteratively returns to the step of generating a texture map corresponding to the current two-dimensional image based on the initial texture map and visibility map corresponding to the current two-dimensional image to continue execution until the iteration stops, that is, the initial texture maps corresponding to each frame of the two-dimensional image are all fused to obtain the texture map of the target object.

[0238] like Figure 7 As shown, 702 is the initial texture map corresponding to each frame of two-dimensional image, 704 is the visibility map corresponding to each frame of two-dimensional image, and after iterative texture fusion, a texture map 706 of the target object is obtained.

[0239] In one embodiment, the computer device may multiply the initial texture map corresponding to the current two-dimensional image and the visibility map to generate the texture map corresponding to the current two-dimensional image. It can be understood that multiplying the initial texture map corresponding to the current two-dimensional image and the visibility map is equivalent to weighting the texture value of each point on the initial texture map by the visibility of each point on the visibility map.

[0240] In one embodiment, fusing the texture map corresponding to the current two-dimensional image with the accumulated fused texture map may include: multiplying the accumulated fused texture map with the invisibility map corresponding to the current two-dimensional image, and then adding the texture map corresponding to the current two-dimensional image.

[0241] In one embodiment, in the above embodiment, the initial texture maps corresponding to each frame of the two-dimensional image are fused in a fusion order to obtain the texture map of the target object in an iterative step which can be expressed by the following formula (7):

[0242] Γ'=(1-α)×Γ+α×P; (7)

[0243] Among them, α represents the visibility map corresponding to the current two-dimensional image, P represents the initial texture map corresponding to the current two-dimensional image, Γ represents the accumulated fused texture map, and Γ' is the accumulated fused texture map obtained after fusing the initial texture map corresponding to the current two-dimensional image.

[0244] In one embodiment, for each frame of a two-dimensional image, a computer device may determine the visibility of each point on an initial texture map corresponding to the two-dimensional image based on the direction of the normal vectors of each surface vertex of the corresponding second parameterized model and the degree of proximity between the direction of the normal vectors and the shooting direction of the two-dimensional image, and generate a visibility map corresponding to the two-dimensional image.

[0245] In the above embodiment, the initial texture maps corresponding to each frame of the two-dimensional image are iteratively fused in sequence according to the fusion order, and the initial texture maps are weighted by the visibility map during the fusion process to obtain the texture map of the target object, thereby improving the accuracy of the obtained texture map.

[0246] In one embodiment, obtaining visibility maps corresponding to each frame of two-dimensional image includes: generating a normal vector map corresponding to the corresponding two-dimensional image according to the normal vector of each second parameterized model surface; for each frame of two-dimensional image, determining the visibility of each point corresponding to the target object in the normal vector map according to the proximity between the normal vector direction in the normal vector map corresponding to the two-dimensional image and the shooting direction of the two-dimensional image, and obtaining the visibility map corresponding to the two-dimensional image.

[0247] The normal vector map is used to represent the normal vector directions of the surface vertices corresponding to each point on the target object in the two-dimensional image in the corresponding second parameterized model.

[0248] In one embodiment, for each frame of the two-dimensional image, the computer device may generate a normal vector map corresponding to the two-dimensional image according to the normal vectors of each surface vertex of the corresponding second parameterized model, such as Figure 6 The reference numeral 606 in FIG. 8 is a normal vector map.

[0249] In one embodiment, for each frame of a two-dimensional image, the computer device may determine the visibility of each point in the initial texture map corresponding to the two-dimensional image according to the proximity between the normal vector direction of each point in the normal vector map corresponding to the two-dimensional image and the shooting direction of the two-dimensional image, and obtain a visibility map corresponding to the two-dimensional image. Specifically, the computer device may determine the visibility corresponding to each point according to the proximity between the normal vector direction of each point in the normal vector map and the shooting direction of the two-dimensional image, and obtain a visibility map, such as Figure 6 608 in the image is the visibility map.

[0250] In one embodiment, the computer device may indicate the proximity degree according to a cosine value between the normal vector direction and the shooting direction of the two-dimensional image.

[0251] In the above embodiment, a visibility map corresponding to the two-dimensional image is obtained based on the degree of proximity between the normal vector direction in the normal vector map corresponding to the two-dimensional image and the shooting direction of the two-dimensional image, so that an accurate visibility map can be obtained, thereby improving the accuracy of the obtained texture map.

[0252] like Figure 8 As shown, the target parametric models are obtained by three-dimensionally reconstructing three target human bodies respectively through the target object three-dimensional reconstruction method in each embodiment of the present application. It can be seen that the reconstruction result is accurate, and the target parametric model has detailed information such as clothes, hair and shoes at all angles, and the shape expression is complete and clear, without incomplete or blurred parts.

[0253] In one embodiment, the target parameterized model is a drivable parameterized model. In this embodiment, the method further comprises: obtaining action parameters; substituting the action parameters into the target parameterized model to drive the target parameterized model to perform corresponding actions.

[0254] The drivable parametric model refers to a parametric model that can input action parameters to drive the execution of actions.

[0255] In one embodiment, the drivable parameterized model may include at least an SMPL model, an SMPLH model, an SMPLX model, and the like.

[0256] In one embodiment, the computer device may extract motion parameters from a motion video or a motion image.

[0257] In one embodiment, the motion parameters may be captured by a motion capture device, and then the computer device may obtain the motion parameters captured by the motion capture device.

[0258] In another embodiment, the computer device can extract the motion parameters of the parameterized model from the two-dimensional image through a motion capture algorithm, and then substitute the motion parameters into the target parameterized model to drive the target parameterized model to perform the corresponding action. In one embodiment, the moving target detection algorithm can be a VIBE algorithm (a motion capture algorithm that obtains the motion parameters of the SMPL model by inputting a video).

[0259] In one embodiment, if the target parameterized model is obtained based on the deformation of the SMPLX model, the computer device can first extract the motion parameters of the SMPL model from the two-dimensional image through the VIBE algorithm, and then convert the motion parameters of the SMPL model into the motion parameters of the SMPLX model.

[0260] In one embodiment, a computer device may obtain a sequence of action parameters, substitute the sequence of action parameters into a target parametric model, obtain multiple frames of images of the target parametric model performing actions, and form a continuous video of the target parametric model performing a series of actions.

[0261] In one embodiment, the computer device may perform texture rendering on the image of the target parameterized model that performs the action according to the texture map, and obtain an image of the target parameterized model that performs the action and has the texture map. Fig. 9 As shown, according to the sequence 902 of action parameters in the figure, the target parameterized model with texture mapping is driven to perform actions, and multiple frames of images 904 of the target parameterized model performing actions are obtained.

[0262] In the above embodiment, the obtained target parametric model can perform corresponding actions according to the input action parameters to realize action drive, avoiding the problem that the three-dimensional continuous surface has detail information but cannot realize action drive. The obtained target parametric model has both detail information and can realize action drive, which improves the applicability of the target parametric model obtained by three-dimensional reconstruction and broadens the scope of application.

[0263] like Fig.10As shown, it is a schematic diagram of the overall process of the three-dimensional reconstruction method of the target object in each embodiment of the present application. The computer device can obtain a single-frame two-dimensional image of the target object, and then create a first parameterized model based on the single-frame two-dimensional image, and generate a three-dimensional continuous surface, and divide a three-dimensional mesh surface from the three-dimensional continuous surface. The computer device can adjust the model parameters of the first parameterized model to register the mesh surface of the first parameterized model with the three-dimensional mesh surface of the three-dimensional continuous surface to obtain a target parameterized model. During the registration process, the computer device can fix the parameter values ​​of some offset parameters according to the surface vertex semantic segmentation result of the first parameterized model. The steps of determining the surface vertex semantic segmentation result of the first parameterized model are as follows: the computer device can obtain multiple frames of two-dimensional images of the target object, respectively create a second parameterized model of each frame of the two-dimensional image, and then obtain the surface vertex semantic segmentation result of the first parameterized model according to the semantic segmentation result of each frame of the two-dimensional image and the corresponding second parameterized model. After obtaining the target parameterized model, the computer device can drive the target parameterized model through action parameters, and texture render the target parameterized model according to the texture map. The steps for determining the texture map are as follows: the computer device can obtain multiple frames of two-dimensional images of the target object, create a second parameterized model for each frame of the two-dimensional image, and then generate an initial texture map and a visibility map corresponding to each frame of the two-dimensional image based on each frame of the two-dimensional image and the corresponding second parameterized model, and then each initial texture map is weighted by the visibility map in a fusion order and iteratively fused to obtain a texture map.

[0264] The present application also provides an application scenario, which is a scenario of three-dimensional reconstruction of a target human body, and the application scenario applies the above-mentioned three-dimensional reconstruction method of the target object. Specifically, the application of the three-dimensional reconstruction method of the target object in this application scenario is as follows:

[0265] A video of a target human body rotating once is captured by a camera, and then a two-dimensional image of the front side is selected from multiple two-dimensional image frames of the video. The computer device can create a three-dimensional parametric model of the target human body based on the two-dimensional image of the front side, and generate a three-dimensional continuous surface of the target human body based on the two-dimensional image, and mesh the three-dimensional continuous surface to obtain a three-dimensional mesh surface. Then, the computer device can adjust the model parameters of the parametric model to align the mesh surface in the parametric model with the three-dimensional mesh surface to obtain a final target parametric model of the target human body, thereby realizing three-dimensional reconstruction of the target human body.

[0266] Furthermore, the computer device can also obtain a texture map of the target human body based on multiple frames of two-dimensional images of the video and a parametric model of the target human body created according to each frame of the two-dimensional images, and perform texture rendering on the target parametric model according to the texture map to obtain a three-dimensional reconstruction result of the target human body with a texture map, such as a three-dimensional reconstruction result of the target human body with a pattern of clothes worn by the target human body.

[0267] Furthermore, the computer device can also extract action parameters from the action video used for action reference, substitute the action parameters into the target parameterized model to drive the target parameterized model to perform the corresponding action, and perform texture rendering on the target parameterized model that performs the action according to the texture map, to obtain a three-dimensional reconstruction result of the target human body to be textured, an image performing the action, or a continuous video of the execution of the action composed of multiple frames of images performing the action. For example, the action parameters can be extracted from a dance video, and finally a three-dimensional reconstruction result of the target human body dancing according to the action in the dance video is obtained.

[0268] The present application also provides an application scenario, which is a scenario of three-dimensional reconstruction of a target animal, and the application scenario applies the above-mentioned three-dimensional reconstruction method of the target object. Specifically, the application of the three-dimensional reconstruction method of the target object in this application scenario is as follows:

[0269] The image acquisition device is used to circle around the target animal to collect multiple frames of two-dimensional images of the target animal, and then a two-dimensional image of the front is selected from the multiple frames of two-dimensional images. The computer device can create a three-dimensional parametric model of the target animal based on the two-dimensional image of the front of the target animal, and generate a three-dimensional continuous surface of the target animal based on the two-dimensional image, and mesh the three-dimensional continuous surface to obtain a three-dimensional mesh surface. Then, the computer device can adjust the model parameters of the parametric model to align the mesh surface in the parametric model with the three-dimensional mesh surface to obtain the final target parametric model of the target animal, thereby realizing three-dimensional reconstruction of the target animal.

[0270] Furthermore, the computer device can also obtain a texture map of the target animal based on multiple two-dimensional image frames of the video and a parametric model of the target animal created according to each two-dimensional image frame, and perform texture rendering on the target parametric model according to the texture map to obtain a three-dimensional reconstruction result of the target animal with a texture map, such as a three-dimensional reconstruction result of the target animal with a pattern of the target animal's fur color and patterns.

[0271] Furthermore, the computer device can also extract action parameters from the action video used for action reference, substitute the action parameters into the target parameterized model to drive the target parameterized model to perform the corresponding action, and perform texture rendering on the target parameterized model that performs the action according to the texture map, to obtain an image of the target animal to be textured that performs the action, or a continuous video of the action composed of multiple frames of images that perform the action. The action video can be an animal action video or a human action video. For example, the action parameters can be extracted from the human action video, and finally a three-dimensional reconstruction result of the target animal is obtained, which performs the action according to the human action video, to achieve the animal anthropomorphism effect.

[0272] It should be understood that, although the steps in each flowchart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in each flowchart may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0273] In one embodiment, Fig.11 As shown, a target object three-dimensional reconstruction device 1100 is provided. The device can adopt a software module or a hardware module, or a combination of the two to become a part of a computer device. The device specifically includes: an image acquisition module 1102, a parameterized model creation module 1104, a continuous surface generation module 1106, a mesh division module 1108 and a model parameter adjustment module 1110, wherein:

[0274] The image acquisition module 1102 is used to acquire a two-dimensional image including the target object.

[0275] The parameterized model creation module 1104 is used to create a three-dimensional parameterized model of the target object according to the two-dimensional image.

[0276] The continuous surface generation module 1106 is used to generate a three-dimensional continuous surface of the target object according to the two-dimensional image; the three-dimensional continuous surface is a three-dimensional surface obtained by continuously representing the surface of the target object.

[0277] The meshing module 1108 is used to mesh the three-dimensional continuous surface to obtain a three-dimensional mesh surface.

[0278] The model parameter adjustment module 1110 is used to adjust the model parameters of the parameterized model so as to align the mesh surface in the parameterized model with the three-dimensional mesh surface to obtain the final target parameterized model of the target object.

[0279] In one embodiment, the model parameter adjustment module 1110 is also used to add an offset parameter to the parameterized model to obtain a deformable parameterized model, and adjust the offset parameter corresponding to the surface vertices of the deformable parameterized model to align the mesh surface in the parameterized model with the three-dimensional mesh surface.

[0280] In one embodiment, the model parameters of the deformable parameterized model also include global feature parameters, posture parameters and shape parameters. In this embodiment, the model parameter adjustment module 1110 is also used to adjust the global feature parameters of the deformable parameterized model to perform depth registration of the mesh surface in the parameterized model with the three-dimensional mesh surface; adjust the posture parameters and shape parameters of the parameterized model after the depth registration to perform coincidence registration of the mesh surface in the parameterized model with the three-dimensional mesh surface; adjust the offset parameters corresponding to the surface vertices of the parameterized model after the coincidence registration to perform feature part registration of the mesh surface in the parameterized model with the three-dimensional mesh surface.

[0281] In one embodiment, the model parameter adjustment module 1110 is also used to obtain the semantic segmentation results of the surface vertices of the parameterized model; determine the surface vertices whose geometric shapes change drastically in the deformable parameterized model based on the surface vertex semantic segmentation results; fix the offset parameters corresponding to the determined surface vertices, and adjust the offset parameters corresponding to the surface vertices other than the determined surface vertices in the deformable parameterized model, so as to align the mesh surface in the parameterized model with the three-dimensional mesh surface.

[0282] In one embodiment, the two-dimensional image is a single-frame two-dimensional image, and the parameterized model created based on the single-frame two-dimensional image is the first parameterized model. In this embodiment, the target object three-dimensional reconstruction device 1100 further includes:

[0283] The semantic segmentation module 1112 is used to obtain multiple frames of two-dimensional images including the target object; obtain a second parameterized model of the target object created according to each frame of the multiple two-dimensional images; perform semantic segmentation on each frame of the multiple two-dimensional images to obtain a semantic segmentation result of partially visible surface vertices of the corresponding second parameterized model; and determine the semantic segmentation result of the surface vertices of the first parameterized model based on the semantic segmentation results of the partially visible surface vertices of each second parameterized model.

[0284] In one embodiment, the semantic segmentation module 1112 is also used to determine invisible surface vertices for which semantic segmentation results have not been determined in the first parameterized model; determine visible surface vertices within a preset neighborhood range of the invisible surface vertices; and determine the semantic segmentation results of the invisible surface vertices based on the determined semantic segmentation results of the visible surface vertices.

[0285] In one embodiment, the model parameter adjustment module 1110 is also used to obtain a target loss function of multiple constraints; the target loss function includes a mesh difference loss function, a surface topology structure loss function and a deformation loss function; in the direction of minimizing the target loss function, the model parameters of the parameterized model are iteratively adjusted to align the mesh surface in the parameterized model with the three-dimensional mesh surface.

[0286] In one embodiment, the two-dimensional image is a single-frame two-dimensional image, and the parameterized model created based on the single-frame two-dimensional image is the first parameterized model. In this embodiment, the target object three-dimensional reconstruction device 1100 further includes:

[0287] The texture rendering module 1114 is used to obtain multiple frames of two-dimensional images including the target object; obtain a second parameterized model of the target object created according to each frame of the two-dimensional image in the multiple frames of the two-dimensional image; map the points corresponding to the target object in the corresponding two-dimensional image to the texture space according to the texture coordinates corresponding to the surface vertices of each second parameterized model, and obtain the initial texture map of the target object corresponding to each frame of the two-dimensional image; fuse the initial texture maps to obtain the texture map of the target object; and perform texture rendering on the target parameterized model according to the texture map.

[0288] In one embodiment, the texture rendering module 1114 is also used to determine the fusion order corresponding to each frame of the two-dimensional image according to the root node direction of each second parameterized model; the initial texture map corresponding to each frame of the two-dimensional image is fused according to the fusion order to obtain the texture map of the target object.

[0289] In one embodiment, the texture rendering module 1114 is also used to obtain the visibility map of the target object corresponding to each frame of the two-dimensional image; the initial texture maps corresponding to each frame of the two-dimensional image are fused in a fusion order to obtain the texture map of the target object, including: selecting the current two-dimensional image from the first two-dimensional image in the fusion order, and generating the texture map corresponding to the current two-dimensional image according to the initial texture map and visibility map corresponding to the current two-dimensional image; fusing the texture map corresponding to the current two-dimensional image with the accumulated fused texture maps; after fusion, taking the next two-dimensional image as the current two-dimensional image in the fusion order, iteratively returning to the step of generating the texture map corresponding to the current two-dimensional image according to the initial texture map and visibility map corresponding to the current two-dimensional image to continue execution until the iteration stops, and the texture map of the target object is obtained.

[0290] In one embodiment, the texture rendering module 1114 is also used to generate a normal vector map corresponding to the corresponding two-dimensional image according to the normal vector of each second parameterized model surface; for each frame of the two-dimensional image, the visibility of each point corresponding to the target object in the normal vector map is determined according to the proximity between the normal vector direction in the normal vector map corresponding to the two-dimensional image and the shooting direction of the two-dimensional image, so as to obtain the visibility map corresponding to the two-dimensional image.

[0291] In one embodiment, the target parameterized model is a drivable parameterized model. Fig.12 As shown, the target object 3D reconstruction device 1100 further includes:

[0292] The action driving module 1116 is used to obtain action parameters and substitute the action parameters into the target parameterized model to drive the target parameterized model to perform corresponding actions.

[0293] In the above-mentioned target object three-dimensional reconstruction device, a three-dimensional parameterized model of the target object is created according to the two-dimensional image of the target object, and a three-dimensional continuous surface of the target object is generated according to the two-dimensional image, and the three-dimensional continuous surface is meshed to obtain a three-dimensional mesh surface, and then the model parameters of the parameterized model are adjusted to align the mesh surface in the parameterized model with the three-dimensional mesh surface to obtain the final target parameterized model of the target object. Because the three-dimensional continuous surface has the detail information of the target object, the target parameterized model obtained by aligning the mesh surface in the parameterized model with the three-dimensional mesh surface has a stronger ability to express the shape of the target object, and can have the detail information of the target object, such as: the detail information of the target human body's clothes, hair or shoes, etc., avoiding the limitation that the parameterized model of the target object obtained by the three-dimensional reconstruction of the parameterized model has limited expression of the shape of the target object and lacks detail information, thereby improving the accuracy of the three-dimensional reconstruction of the target object. In addition, the parameterized model often has ambiguity in depth. For example, the legs of the parameterized model of a target person with a shorter height are often incomplete. The target parameterized model obtained by the method of the present application also avoids ambiguity in depth and improves the accuracy of three-dimensional reconstruction.

[0294] In addition, the three-dimensional model of the target object modeled by the three-dimensional continuous surface cannot be driven by motion and has limitations, while the parametric model can achieve motion driving. Therefore, the model parameters of the parametric model are adjusted, and the mesh surface in the parametric model is registered with the three-dimensional mesh surface extracted from the three-dimensional continuous surface to obtain the target parametric model, which can have the advantage of the parametric model being able to be driven by motion, avoiding the limitation that the three-dimensional continuous surface cannot be driven, so that the target parametric model has both detailed information, improves the accuracy of three-dimensional reconstruction, and can be driven by motion. Because the model can be flexibly driven by motion, the applicability of the target parametric model obtained by three-dimensional reconstruction is improved, and the scope of application is broadened.

[0295] For the specific definition of the target object three-dimensional reconstruction device, please refer to the definition of the target object three-dimensional reconstruction method above, which will not be repeated here. Each module in the above-mentioned target object three-dimensional reconstruction device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0296] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig.13 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store three-dimensional reconstruction data of the target object. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for three-dimensional reconstruction of a target object is implemented.

[0297] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig.14As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a method for three-dimensional reconstruction of a target object is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0298] Those skilled in the art will understand that Fig.13 and 14 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0299] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0300] In one embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0301] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps in the above-mentioned method embodiments.

[0302] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0303] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0304] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for three-dimensional reconstruction of a target object, characterized in that: The method comprises: Acquire a two-dimensional image including a target object; the two-dimensional image is a single-frame two-dimensional image; Creating a first three-dimensional parameterized model of the target object based on the two-dimensional image; generating a three-dimensional continuous surface of the target object according to the two-dimensional image, wherein the three-dimensional continuous surface is a three-dimensional surface obtained by continuously representing the surface of the target object; Meshing the three-dimensional continuous surface to obtain a three-dimensional mesh surface; Adjusting the model parameters of the first parameterized model to perform registration processing on the mesh surface in the first parameterized model and the three-dimensional mesh surface to obtain a final target parameterized model of the target object; Acquire multiple frames of two-dimensional images including the target object; Acquire a second parameterized model of the target object created corresponding to each of the two-dimensional image frames in the multiple two-dimensional image frames; According to the texture coordinates corresponding to the surface vertices of each of the second parameterized models, the points corresponding to the target object in the corresponding two-dimensional image are mapped to the texture space to obtain the initial texture map of the target object corresponding to each frame of the two-dimensional image; Merging the initial texture maps to obtain a texture map of the target object; The target parameterized model is texture rendered according to the texture map.

2. The method according to claim 1, characterized in that: The method further comprises: adding an offset parameter to the first parameterized model to obtain a deformable first parameterized model; The adjusting the model parameters of the first parameterized model to register the mesh surface in the first parameterized model with the three-dimensional mesh surface includes: The offset parameters corresponding to the surface vertices of the deformable first parameterized model are adjusted to align the mesh surface in the first parameterized model with the three-dimensional mesh surface.

3. The method according to claim 2, characterized in that The model parameters of the deformable first parameterized model also include global feature parameters, posture parameters and shape parameters; The step of adjusting the offset parameters corresponding to the surface vertices of the deformable first parameterized model to register the mesh surface in the first parameterized model with the three-dimensional mesh surface includes: adjusting global feature parameters of the deformable first parameterized model to perform depth registration of a mesh surface in the first parameterized model with the three-dimensional mesh surface; Adjusting the posture parameters and shape parameters of the first parameterized model after the depth registration to coincide and register the mesh surface in the first parameterized model with the three-dimensional mesh surface; The offset parameters corresponding to the surface vertices of the first parameterized model after the coincidence registration are adjusted to perform feature part registration between the mesh surface in the first parameterized model and the three-dimensional mesh surface.

4. The method according to claim 2, characterized in that: The step of adjusting the offset parameters corresponding to the surface vertices of the deformable first parameterized model to register the mesh surface in the first parameterized model with the three-dimensional mesh surface includes: Obtaining a semantic segmentation result of surface vertices of the first parameterized model; Determining, according to the surface vertex semantic segmentation result, surface vertices of the deformable first parameterized model whose geometric shapes change dramatically; The offset parameters corresponding to the determined surface vertices are fixed, and the offset parameters corresponding to the surface vertices other than the determined surface vertices in the deformable first parameterized model are adjusted to align the mesh surface in the first parameterized model with the three-dimensional mesh surface.

5. The method according to claim 4, characterized in that The two-dimensional image is a single-frame two-dimensional image; the first parameterized model created according to the single-frame two-dimensional image is a first parameterized model; The obtaining of the surface vertex semantic segmentation result of the first parameterized model comprises: Acquire multiple frames of two-dimensional images including the target object; Acquire a second parameterized model of the target object created corresponding to each of the two-dimensional image frames in the multiple two-dimensional image frames; Performing semantic segmentation on each of the multiple two-dimensional image frames to obtain semantic segmentation results of partially visible surface vertices of the corresponding second parameterized model; The semantic segmentation result of the surface vertices of the first parameterized model is determined according to the semantic segmentation result of the partially visible surface vertices of each of the second parameterized models.

6. The method according to claim 5, characterized in that After determining the semantic segmentation result of the surface vertices of the first parameterized model according to the semantic segmentation results of the surface vertices of each of the second parameterized models, the method further includes: Determining invisible surface vertices for which semantic segmentation results are not determined in the first parameterized model; Determine a visible surface vertex within a preset neighborhood of the invisible surface vertex; According to the determined semantic segmentation result of the visible surface vertices, the semantic segmentation result of the invisible surface vertices is determined.

7. The method according to claim 1, characterized in that The adjusting the model parameters of the first parameterized model to register the mesh surface in the first parameterized model with the three-dimensional mesh surface includes: Obtaining a target loss function of multiple constraints; the target loss function includes a mesh difference loss function, a surface topology structure loss function and a deformation loss function; Iteratively adjusting the model parameters of the first parameterized model in a direction that minimizes the objective loss function, so as to perform registration processing on the mesh surface in the first parameterized model and the three-dimensional mesh surface; The mesh difference loss function includes the sum of the minimum distances from each surface vertex in the first parameterized model to the three-dimensional mesh surface divided by the three-dimensional continuous surface, and the sum of the minimum distances from each surface vertex of the three-dimensional mesh surface divided by the three-dimensional continuous surface to the mesh surface of the parameterized model; The surface topology loss function includes a surface topology difference loss function and an intrinsic surface topology loss function; the surface topology difference loss function is a loss function used to characterize the change in the surface topology of the parameterized model before and after the model parameters are adjusted; the intrinsic surface topology loss function is a loss function used to characterize the surface topology of the first parameterized model itself after the model parameters are adjusted; The deformation loss function is a regularization of the parameter values ​​of the offset parameters of the surface vertices of the first parameterized model.

8. The method according to claim 7, characterized in that The step of fusing the initial texture maps to obtain the texture map of the target object includes: Determining the fusion order corresponding to each frame of the two-dimensional image according to the root node direction of each second parameterized model; The initial texture maps corresponding to the two-dimensional image frames are fused in the fusion order to obtain the texture map of the target object.

9. The method according to claim 8, characterized in that The method further comprises: Obtaining visibility maps of the target object corresponding to each frame of the two-dimensional image; The step of fusing the initial texture maps corresponding to the two-dimensional images of each frame in the fusion order to obtain the texture map of the target object includes: According to the fusion order, the current two-dimensional image is selected from the first two-dimensional image, and the texture map corresponding to the current two-dimensional image is generated according to the initial texture map and visibility map corresponding to the current two-dimensional image; Fusing the texture map corresponding to the current two-dimensional image with the accumulated fused texture map; After fusion, the next two-dimensional image is used as the current two-dimensional image in the fusion order, and the step of generating the texture map corresponding to the current two-dimensional image according to the initial texture map and visibility map corresponding to the current two-dimensional image is iteratively returned to continue execution until the iteration stops, and the texture map of the target object is obtained.

10. The method according to claim 9, characterized in that The obtaining of visibility maps corresponding to each frame of two-dimensional image includes: According to the normal vectors of the surfaces of the second parameterized models, a normal vector map corresponding to the corresponding two-dimensional image is generated; For each frame of a two-dimensional image, the visibility of each point corresponding to the target object in the normal vector map is determined based on the proximity between the normal vector direction in the normal vector map corresponding to the two-dimensional image and the shooting direction of the two-dimensional image, and a visibility map corresponding to the two-dimensional image is obtained.

11. The method according to any one of claims 1 to 10, characterized in that The target parameterized model is a drivable parameterized model; The method further comprises: Get action parameters; The action parameters are substituted into the target parameterized model to drive the target parameterized model to perform corresponding actions.

12. A three-dimensional reconstruction device for a target object, characterized in that: The device comprises: An image acquisition module, used to acquire a two-dimensional image including a target object; the two-dimensional image is a single-frame two-dimensional image; A parameterized model creation module, used to create a first three-dimensional parameterized model of the target object according to the two-dimensional image; A continuous surface generation module, used to generate a three-dimensional continuous surface of the target object according to the two-dimensional image; the three-dimensional continuous surface is a three-dimensional surface obtained by continuously representing the surface of the target object; A meshing module, used for meshing the three-dimensional continuous surface to obtain a three-dimensional mesh surface; A model parameter adjustment module, used for adjusting the model parameters of the first parameterized model, so as to register the mesh surface in the first parameterized model with the three-dimensional mesh surface, and obtain a final target parameterized model of the target object; A texture rendering module is used to obtain multiple frames of two-dimensional images including the target object; obtain a second parameterized model of the target object created according to each frame of the two-dimensional image in the multiple frames of the two-dimensional image; map the points corresponding to the target object in the corresponding two-dimensional image to the texture space according to the texture coordinates corresponding to the surface vertices of each second parameterized model, and obtain the initial texture map of the target object corresponding to each frame of the two-dimensional image; fuse each of the initial texture maps to obtain the texture map of the target object; and perform texture rendering on the target parameterized model according to the texture map.

13. The device according to claim 12, characterized in that The model parameter adjustment module is also used to add an offset parameter to the first parameterized model to obtain a deformable first parameterized model; adjust the offset parameter corresponding to the surface vertices of the deformable first parameterized model to align the mesh surface in the first parameterized model with the three-dimensional mesh surface.

14. The device according to claim 13, characterized in that The model parameters of the deformable first parameterized model also include global feature parameters, posture parameters and shape parameters; The model parameter adjustment module is further used to adjust the global feature parameters of the deformable first parameterized model to perform depth registration between the mesh surface in the first parameterized model and the three-dimensional mesh surface; Adjusting the posture parameters and shape parameters of the first parameterized model after the depth registration to coincide and register the mesh surface in the first parameterized model with the three-dimensional mesh surface; The offset parameters corresponding to the surface vertices of the first parameterized model after the coincidence registration are adjusted to perform feature part registration between the mesh surface in the first parameterized model and the three-dimensional mesh surface.

15. The device according to claim 13, characterized in that The model parameter adjustment module is further used to obtain a surface vertex semantic segmentation result of the first parameterized model; and determine, based on the surface vertex semantic segmentation result, surface vertices of the deformable first parameterized model whose geometric shapes change dramatically; The offset parameters corresponding to the determined surface vertices are fixed, and the offset parameters corresponding to the surface vertices other than the determined surface vertices in the deformable first parameterized model are adjusted to align the mesh surface in the first parameterized model with the three-dimensional mesh surface.

16. The device according to claim 15, characterized in that The model parameter adjustment module is also used to obtain multiple frames of two-dimensional images including the target object; obtain a second parameterized model of the target object created according to each frame of the multiple two-dimensional images; perform semantic segmentation on each frame of the multiple two-dimensional images to obtain a corresponding semantic segmentation result of partially visible surface vertices of the second parameterized model; and determine the semantic segmentation result of the surface vertices of the first parameterized model based on the semantic segmentation results of the partially visible surface vertices of each of the second parameterized models.

17. The device according to claim 16, characterized in that The device also includes: A semantic segmentation module is used to determine invisible surface vertices for which semantic segmentation results have not been determined in the first parameterized model; determine visible surface vertices within a preset neighborhood range of the invisible surface vertices; and determine the semantic segmentation results of the invisible surface vertices based on the determined semantic segmentation results of the visible surface vertices.

18. The device according to claim 12, characterized in that The model parameter adjustment module is also used to obtain a target loss function of multiple constraints; the target loss function includes a mesh difference loss function, a surface topology structure loss function and a deformation loss function; in a direction that minimizes the target loss function, iteratively adjust the model parameters of the first parameterized model to align the mesh surface in the first parameterized model with the three-dimensional mesh surface; The mesh difference loss function includes the sum of the minimum distances from each surface vertex in the first parameterized model to the three-dimensional mesh surface divided by the three-dimensional continuous surface, and the sum of the minimum distances from each surface vertex of the three-dimensional mesh surface divided by the three-dimensional continuous surface to the mesh surface of the parameterized model; The surface topology loss function includes a surface topology difference loss function and an intrinsic surface topology loss function; the surface topology difference loss function is a loss function used to characterize the change in the surface topology of the parameterized model before and after the model parameters are adjusted; the intrinsic surface topology loss function is a loss function used to characterize the surface topology of the first parameterized model itself after the model parameters are adjusted; The deformation loss function is a regularization of the parameter values ​​of the offset parameters of the surface vertices of the first parameterized model.

19. The device according to claim 18, characterized in that The texture rendering module is also used to determine the fusion order corresponding to each frame of two-dimensional image according to the root node direction of each second parameterized model; and fuse the initial texture maps corresponding to each frame of two-dimensional image according to the fusion order to obtain the texture map of the target object.

20. The device according to claim 19, characterized in that The texture rendering module is also used to obtain the visibility map of the target object corresponding to each frame of the two-dimensional image; select the current two-dimensional image from the first two-dimensional image according to the fusion order, and generate the texture map corresponding to the current two-dimensional image according to the initial texture map and the visibility map corresponding to the current two-dimensional image; Fusing the texture map corresponding to the current two-dimensional image with the accumulated fused texture map; After fusion, the next two-dimensional image is used as the current two-dimensional image in the fusion order, and the step of generating the texture map corresponding to the current two-dimensional image according to the initial texture map and visibility map corresponding to the current two-dimensional image is iteratively returned to continue execution until the iteration stops, and the texture map of the target object is obtained.

21. The device according to claim 20, characterized in that The texture rendering module is also used to generate a normal vector map corresponding to the corresponding two-dimensional image according to the normal vector of each second parameterized model surface; for each frame of the two-dimensional image, the visibility of each point corresponding to the target object in the normal vector map is determined according to the proximity between the normal vector direction in the normal vector map corresponding to the two-dimensional image and the shooting direction of the two-dimensional image, so as to obtain the visibility map corresponding to the two-dimensional image.

22. The device according to any one of claims 12 to 21, characterized in that The target parameterized model is a drivable parameterized model; the device further comprises: The action driving module is used to obtain action parameters; substitute the action parameters into the target parameterized model to drive the target parameterized model to perform corresponding actions.

23. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 11 are implemented.

24. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

25. A computer program product comprising computer instructions, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 11.

Images