Three-dimensional model construction method and apparatus

Abstract

The embodiments of the present specification provide a three-dimensional model construction method and device, wherein the three-dimensional model construction method comprises: constructing a basic three-dimensional model of a target object based on multiple depth images of the target object; determining texture coordinates of multiple key points of the target object in a three-dimensional space corresponding to the basic three-dimensional model according to two-dimensional space coordinates and depth values corresponding to the multiple key points in the multiple depth images; generating a texture displacement map corresponding to the target object based on the texture coordinates; obtaining color images corresponding to the multiple depth images respectively; and determining a texture map corresponding to the target object according to color values corresponding to the multiple key points in the color images and a mapping relationship between the texture coordinates and the two-dimensional space coordinates. The basic three-dimensional model, the texture displacement map and the texture map are superimposed to generate a target three-dimensional model of the target object.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of computer technology, and in particular to a method for constructing a three-dimensional model. Background Technology

[0002] Currently, 3D model building is widely used in fields such as virtual reality, gaming experiences, and virtual try-on.

[0003] In the process of 3D model construction, implicit representation methods based on pixel alignment dominate. This method implicitly represents the 3D model as an occupied field in 3D space, aligning sampled points in 3D space with the 2D image to determine whether they are inside the occupied field. However, this method ignores the spatial geometric dependencies between points, resulting in unrealistic local reconstruction details such as clothing folds and facial wrinkles. Therefore, an effective method is urgently needed to solve this problem. Summary of the Invention

[0004] In view of this, embodiments of this specification provide a method for constructing a three-dimensional model. One or more embodiments of this specification also relate to a three-dimensional model construction apparatus, a computing device, a computer-readable storage medium, and a computer program, to address the technical deficiencies existing in the prior art.

[0005] According to a first aspect of the embodiments of this specification, a method for constructing a three-dimensional model is provided, comprising:

[0006] Based on multiple depth images of the target object, a basic 3D model of the target object is constructed.

[0007] Based on the two-dimensional spatial coordinates and depth values ​​corresponding to multiple key points of the target object in the multiple depth images, the texture coordinates of the multiple key points in the three-dimensional space corresponding to the basic three-dimensional model are determined, and a texture displacement map corresponding to the target object is generated based on the texture coordinates.

[0008] Obtain the color images corresponding to the multiple depth images respectively, and determine the texture map corresponding to the target object based on the color values ​​corresponding to the multiple key points in the color images and the mapping relationship between the texture coordinates and the two-dimensional spatial coordinates.

[0009] The base 3D model, the texture displacement map, and the texture map are superimposed to generate the target 3D model of the target object.

[0010] According to a second aspect of the embodiments of this specification, a three-dimensional model building apparatus is provided, comprising:

[0011] The construction module is configured to construct a basic 3D model of the target object based on multiple depth images of the target object;

[0012] The determination module is configured to determine the texture coordinates of the multiple key points in the three-dimensional space corresponding to the basic three-dimensional model based on the two-dimensional spatial coordinates and depth values ​​of the multiple key points of the target object in the multiple depth images, and generate a texture displacement map corresponding to the target object based on the texture coordinates.

[0013] The acquisition module is configured to acquire color images corresponding to the multiple depth images respectively, and determine the texture map corresponding to the target object based on the color values ​​corresponding to the multiple key points in the color images and the mapping relationship between the texture coordinates and the two-dimensional space coordinates.

[0014] The generation module is configured to overlay the base 3D model, the texture displacement map, and the texture map to generate a target 3D model of the target object.

[0015] According to a third aspect of the embodiments of this specification, a computing device is provided, comprising:

[0016] Memory and processor;

[0017] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement any of the steps of the three-dimensional model construction method.

[0018] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores computer-executable instructions, which, when executed by a processor, implement the steps of any of the three-dimensional model construction methods described herein.

[0019] According to a fifth aspect of the embodiments of this specification, a computer program is provided, wherein when the computer program is executed in a computer, it causes the computer to perform the steps of the above-described three-dimensional model construction method.

[0020] One embodiment of this specification constructs a basic 3D model of a target object based on multiple depth images of the target object. According to the 2D spatial coordinates and depth values ​​corresponding to multiple key points of the target object in the multiple depth images, the texture coordinates of the multiple key points in the 3D space corresponding to the basic 3D model are determined. A texture displacement map corresponding to the target object is generated based on the texture coordinates. Color images corresponding to the multiple depth images are obtained, and a texture map corresponding to the target object is determined based on the color values ​​corresponding to the multiple key points in the color images and the mapping relationship between the texture coordinates and the 2D spatial coordinates. The basic 3D model, the texture displacement map, and the texture map are superimposed to generate a target 3D model of the target object.

[0021] This specification's embodiments establish a correlation between multi-view color images and depth images and the UV space corresponding to the basic 3D model. Specifically, UV space transformation is performed on the multi-view color images and depth images based on the UV space, resulting in UV-displacement maps and UV-texture maps, respectively. Then, the basic 3D model, UV-displacement maps, and UV-texture maps can be superimposed to generate the target 3D model of the target object. Through this processing method, the realism of the constructed target 3D model can be guaranteed without the need for a time-consuming texture reconstruction and optimization process. This reduces the computational load of the model construction process and simultaneously obtains high-quality UV-displacement maps and UV-texture maps, thereby improving the accuracy and efficiency of the model construction results. Attached Figure Description

[0022] Figure 1 This is a schematic diagram illustrating a three-dimensional model construction process provided in one embodiment of this specification;

[0023] Figure 2 This is a flowchart illustrating a three-dimensional model construction method provided in one embodiment of this specification;

[0024] Figure 3a This is a schematic diagram of a texture displacement map generation process provided in one embodiment of this specification;

[0025] Figure 3b This is a schematic diagram of an image fusion process provided in one embodiment of this specification;

[0026] Figure 3c This is a schematic diagram illustrating a process for constructing a three-dimensional human body model according to one embodiment of this specification;

[0027] Figure 3d This is a schematic diagram illustrating another three-dimensional model construction process provided in one embodiment of this specification;

[0028] Figure 4 This is a flowchart illustrating the processing steps of a three-dimensional model construction method provided in one embodiment of this specification.

[0029] Figure 5 This is a schematic diagram of the structure of a three-dimensional model building device provided in one embodiment of this specification;

[0030] Figure 6 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0031] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0032] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0033] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0034] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0035] UV mapping space: In 3D modeling, the space in which a 2D image is projected onto a 3D surface for texture mapping.

[0036] Texture mapping: Converts object space coordinates into texture coordinates, associating texture pixels with pixels in screen space.

[0037] RGBD: Color image plus depth image.

[0038] Multi-view fusion: fusing observation signals from multiple viewpoints to obtain an overall result.

[0039] This specification provides a method for constructing a three-dimensional model, and also relates to a three-dimensional model construction apparatus, a computing device, a computer-readable storage medium, and a computer program, which will be described in detail in the following embodiments.

[0040] Figure 1 A schematic diagram of a three-dimensional model construction process provided according to an embodiment of this specification is shown.

[0041] Currently, most methods for constructing 3D models use general-purpose monocular RGBD fusion algorithms. This approach, given an input depth image, involves two steps: motion tracking and range field update. Motion tracking calculates the motion 3D field corresponding to the current frame using a non-rigid registration algorithm. After calculating the updated motion model field, it is used to update the range field function. However, this approach suffers from inconsistent convergence during motion model calculation. For complex motions, the tracking of the motion field model is prone to local convergence, leading to fusion failure. Furthermore, obtaining a complete 3D human body model requires scanning the entire human viewpoint, a time-consuming process that significantly limits its application in virtual reality.

[0042] Additionally, there are multi-view RGBD reconstruction algorithms based on implicit functions. These algorithms typically model the mapping relationship between multi-view RGBD input and the 3D model as an implicit function and represent it using a multilayer perceptron (MLP) mechanism. During 3D model construction, multi-view depth images are used as input. Depth features are extracted through a convolutional network, and these features are fused using a self-attention mechanism to obtain the corresponding geometric information. Finally, a mesh model extraction algorithm is used to obtain the final geometric surface. However, this approach requires generating a high-resolution 3D voxel field for mesh generation during inference. The model extraction algorithm, for each voxel point, needs to use the network to infer whether it is within the model. This process is not easily parallelized, resulting in slow inference speed and failing to meet the real-time requirements of current virtual reality applications.

[0043] Based on this, the embodiments of this specification first acquire multiple depth images and color images of the target object by capturing them with an RGBD camera. A basic three-dimensional model of the target object is constructed based on the multiple depth images. Then, according to the two-dimensional spatial coordinates and depth values ​​corresponding to multiple key points of the target object in the multiple depth images, the texture coordinates of the multiple key points in the three-dimensional space corresponding to the basic three-dimensional model are determined, and a texture displacement map corresponding to the target object is generated based on the texture coordinates. The color images corresponding to the multiple depth images are acquired, and the texture map corresponding to the target object is determined according to the color values ​​corresponding to multiple key points in the color images and the mapping relationship between the texture coordinates and the two-dimensional spatial coordinates. Finally, the basic three-dimensional model, the texture displacement map, and the texture map are superimposed to generate the target three-dimensional model of the target object.

[0044] In practical applications, the three-dimensional model construction method provided in the embodiments of this specification is specifically implemented in the cloud, but it is not limited to being implemented on the device side only when there are sufficient computing resources on the device side.

[0045] This specification's embodiments establish a correlation between multi-view color images and depth images and the UV space corresponding to the basic 3D model. Specifically, UV space transformation is performed on the multi-view color images and depth images based on the UV space, resulting in UV-displacement maps and UV-texture maps, respectively. Then, the basic 3D model, UV-displacement maps, and UV-texture maps can be superimposed to generate the target 3D model of the target object. Through this processing method, the realism of the constructed target 3D model can be guaranteed without the need for a time-consuming texture reconstruction and optimization process. This reduces the computational load of the model construction process and simultaneously obtains high-quality UV-displacement maps and UV-texture maps, thereby improving the accuracy and efficiency of the model construction results.

[0046] Figure 2 A flowchart of a three-dimensional model construction method according to an embodiment of this specification is shown, which specifically includes the following steps.

[0047] Step 202: Construct a basic 3D model of the target object based on multiple depth images of the target object.

[0048] Specifically, the target object can be the human body, a virtual character in a virtual game, or other three-dimensional objects. In this embodiment of the specification, a three-dimensional model of the target object needs to be constructed. Therefore, multiple depth images and multiple color images of the target object can be obtained by taking pictures of the target object with an RGBD camera. Furthermore, when taking pictures with an RGBD camera, the target object can be simultaneously obtained from any angle by taking pictures from any angle. In this embodiment of the specification, the color images and depth images obtained by taking pictures from the same angle have a corresponding relationship.

[0049] After capturing multiple depth images, a basic 3D model of the target object can be constructed based on these depth images. This basic 3D model does not include texture maps.

[0050] When the target object is the human body, the basic three-dimensional model constructed is the naked 3D model of the target human body.

[0051] In specific implementation, a basic 3D model of the target object is constructed based on multiple depth images of the target object, including:

[0052] Acquire multiple depth images of the target object;

[0053] The multiple depth images are input into a 3D model construction network. The 3D model construction sub-network in the 3D model construction network processes the multiple depth images to generate a basic 3D model of the target object.

[0054] In practical applications, while acquiring multiple depth images of the target object, color images of the target object corresponding to each of the multiple depth images are also acquired. The subsequent target depth image is any one of the multiple depth images. The target depth image and the color image corresponding to the target depth image are generated by shooting from the same shooting angle.

[0055] Specifically, as mentioned earlier, before constructing a 3D model, multiple depth images and multiple color images of the target object can be obtained by taking pictures of the target object with an RGBD camera. Furthermore, the target color images and target depth images obtained by taking pictures from the same angle have a corresponding relationship.

[0056] After capturing depth and color images, multiple depth images can be input into a 3D model building network. The 3D model building subnetwork within the 3D model building network processes the multiple depth images to generate a basic 3D model of the target object.

[0057] In the case where the target object is the human body, the sub-network for constructing the 3D model can be a parametric human body sub-network. This parametric human body sub-network can actually be a parametric human body model SMPL (Skinned Multi-Person Linear Model). However, since the embodiments of this specification require the construction of a 3D model of the target object, and the basic 3D model of the target object must first be constructed through the SMPL model, the embodiments of this specification uniformly refer to the SMPL model as a parametric human body sub-network to distinguish it from the 3D model.

[0058] Since SMPL (Skinned Multi-Person Linear Model) is a skinned, vertex-based 3D human body model, it can accurately represent different shapes and poses of the human body. Therefore, the embodiments in this specification can input multiple depth images of the target human body into the SMPL model for processing, to determine the shape and pose of the target human body based on the depth images, and to construct a basic 3D model of the target human body, i.e., a nude 3D model, based on the determination results. This basic 3D model is then used to overlay texture displacement maps and texture maps to generate a 3D human body model with texture mapping.

[0059] In specific implementation, the target object includes the target human body, and the three-dimensional model construction sub-network includes a parameterized human body sub-network;

[0060] Accordingly, after constructing the basic three-dimensional model of the target object, a reference three-dimensional model of the target object can also be obtained, and the first three-dimensional spatial coordinates of multiple key points in the reference three-dimensional model can be determined.

[0061] Determine the second three-dimensional spatial coordinates of multiple key points in the basic three-dimensional model;

[0062] Based on the first three-dimensional spatial coordinates and the second three-dimensional spatial coordinates, the loss value corresponding to the parameterized human sub-network is determined;

[0063] The shape and pose parameters of the parameterized human subnetwork are optimized based on the loss value.

[0064] Specifically, as mentioned earlier, when the target object is the human body, the 3D model construction sub-network can be the SMPL model. After constructing the basic 3D model of the target human body, a reference 3D model of the target human body can also be obtained, that is, the accurate 3D model of the target human body. The accuracy of the basic 3D model output by the SMPL model can be evaluated based on the reference 3D model, and the model parameters of the SMPL model can be optimized according to the evaluation results.

[0065] Specifically, the reference 3D model of the target object can be obtained first, and the first 3D spatial coordinates of multiple key points in the reference 3D model can be determined. Then, the second 3D spatial coordinates of multiple key points in the base 3D model can be determined. Based on the first and second 3D spatial coordinates, the loss value corresponding to the parameterized human sub-network can be determined. Then, the shape parameters and pose parameters of the parameterized human sub-network can be optimized according to the loss value.

[0066] For example, the identified key points are key point 1, key point 2, and key point 3. The first three-dimensional spatial coordinates of key point 1 in the reference three-dimensional model are (x... 11 y 11 , z 11 The first three-dimensional spatial coordinate of key point 2 in the reference three-dimensional model is (x...). 21 y 21 , z 21 The first three-dimensional spatial coordinate of key point 3 in the reference three-dimensional model is (x...). 31 y 31 , z 31 Key point 1's second 3D spatial coordinates in the basic 3D model are (x...). 12 y 12 , z 12 The second three-dimensional spatial coordinate of key point 2 in the reference three-dimensional model is (x...). 22 y 22 , z 22 The key point 3 has the following second three-dimensional spatial coordinates in the reference three-dimensional model: (x...) 32 y 32 , z 32 Then the first three-dimensional spatial coordinates (x, y) of key point 1 can be calculated.11 y 11 , z 11 ) and the second three-dimensional spatial coordinates (x) 12 y 12 , z 12 The first distance between the two points is used to calculate the first three-dimensional spatial coordinates (x, y) of key point 2. 21 y 21 , z 21 ) and the second three-dimensional spatial coordinates (x) 22 y 22 , z 22 The second distance between the two points is used to calculate the first three-dimensional spatial coordinates (x, y) of key point 3. 31 y 31 , z 31 ) and the second three-dimensional spatial coordinates (x) 32 y 32 , z 32 The third distance between the first, second and third distances is calculated, and the sum of the first, second and third distances is used as the loss value of the parameterized human subnetwork. The shape and pose parameters of the parameterized human subnetwork are then optimized based on the loss value.

[0067] As can be seen, in practical applications, the SMPL model estimation under multi-view input is to optimize the body shape parameter β and pose parameter θ of the parameterized human body model SMPL from multi-view image input, thus serving as the geometric prior for subsequent fusion algorithms. The optimization function is shown in Equation 1.

[0068] E(β, θ) = E J +λ θ E θ +λ α E α +λ β E β Formula 1

[0069] Among them, E J E is the reprojection error between the 3D keypoints in the base 3D model output by the SMPL model and the 2D keypoints in the depth image. θ The prior knowledge of human pose regarding θ is obtained through the SMPL model. α It is a human posture penalty item used to avoid impossible postures. E β λ represents the difference between the predicted human body shape parameter β and the prior on the training set, avoiding impossible body shape parameters. θ , λ α , λ βThe weights for the corresponding terms are constants. Because the SMPL model has a fixed topology, processing multiple depth images using the SMPL model allows for the determination of a unique UV mapping space based on the resulting basic 3D model. This space is consistent with the topology of the SMPL model.

[0070] Alternatively, the basic 3D model can be converted into a 2D planar image. Based on the coordinates of each keypoint in the 2D planar image and its actual coordinates, the loss value corresponding to the parameterized human subnetwork can be determined. Then, the shape and pose parameters of the parameterized human subnetwork can be optimized based on the loss value. The specific method for calculating the loss value can be determined according to actual needs and will not be elaborated here.

[0071] Step 204: Based on the two-dimensional spatial coordinates and depth values ​​corresponding to multiple key points of the target object in the multiple depth images, determine the texture coordinates of the multiple key points in the three-dimensional space corresponding to the basic three-dimensional model, and generate a texture displacement map corresponding to the target object based on the texture coordinates.

[0072] Among them, the texture displacement map is used to characterize the displacement of key points in the normal direction, such as the displacement of the first key point in the normal direction of the second key point.

[0073] Specifically, texture coordinates, also known as UV coordinates, are used to map the vertices of a 3D model to pixels on the depth image in order to locate texture maps on the surface of the 3D model.

[0074] Key points include, but are not limited to, key points of the target object itself (such as key points of the target human body such as joints and fingers) or key points of the target object's decorations (such as key points of clothing and hats). In the embodiments of this specification, the basic three-dimensional model can be composed of thousands of adjacent meshes. The meshes can be polygons such as triangles and quadrilaterals. Therefore, the key points described in the embodiments of this specification can actually be the mesh vertices in the basic three-dimensional model.

[0075] Since the basic 3D model is a bare 3D model, but the target object may contain other decorations besides the object itself, in order to ensure the construction effect of the 3D model, after constructing the basic 3D model, this embodiment of the specification can also determine the texture coordinates of multiple key points in the corresponding 3D space of the basic 3D model based on the two-dimensional spatial coordinates and depth values ​​corresponding to the key points of the target object itself or the key points of the decorations in the depth image, and generate the texture displacement map corresponding to the target object based on the texture coordinates.

[0076] As mentioned earlier, texture coordinates are used to map the vertices of a 3D model to pixels on a depth image for texture mapping. The base model is a bare 3D model, and the texture information of the target object itself or the texture information of the decoration needs to be added to the base 3D model. Therefore, before texture mapping, the texture coordinates of the decoration in the 3D space of the base 3D model can be determined first, and then the corresponding texture mapping process can be performed according to the texture information and texture coordinates of the decoration.

[0077] In specific implementation, based on the two-dimensional spatial coordinates and depth values ​​corresponding to multiple key points of the target object in the multiple depth images, the texture coordinates of the multiple key points in the three-dimensional space corresponding to the basic three-dimensional model are determined, including:

[0078] Based on the two-dimensional spatial coordinates and depth values ​​corresponding to multiple key points of the target object in the target depth image, the texture coordinates of the multiple key points in the three-dimensional space corresponding to the basic three-dimensional model are determined, wherein the target depth image is any one of the multiple depth images.

[0079] Furthermore, based on the two-dimensional spatial coordinates and depth values ​​corresponding to multiple key points of the target object in the target depth image, the texture coordinates of the multiple key points in the three-dimensional space corresponding to the basic three-dimensional model are determined, including:

[0080] In the target depth image, determine the two-dimensional spatial coordinates and the first depth value corresponding to the first key point of the target object, wherein the first key point is any one of a plurality of key points of the target object in the target depth image;

[0081] In the three-dimensional space corresponding to the basic three-dimensional model, a second key point is determined that corresponds to the first key point, and a second depth value is determined that corresponds to the second key point. In the three-dimensional space, the first key point is located in the normal direction of the second key point.

[0082] Based on the first depth value, the second depth value, and the three-dimensional spatial coordinates of the second key point in the three-dimensional space, the texture coordinates of the first key point in the three-dimensional space are determined.

[0083] Specifically, when acquiring images of a target object using an RGBD camera, multiple depth images can be obtained. These multiple depth images may contain some of the same content, but they are actually independent of each other. Therefore, based on the two-dimensional spatial coordinates and depth values ​​corresponding to multiple key points of the target object in the multiple depth images, the texture coordinates of these key points in the three-dimensional space corresponding to the basic three-dimensional model can be determined. Specifically, any one of the multiple depth images can be processed separately, that is, based on the two-dimensional spatial coordinates and depth values ​​corresponding to multiple key points of the target object in any depth image (target depth image), the texture coordinates of these key points in the three-dimensional space corresponding to the basic three-dimensional model can be determined.

[0084] In practical applications, when determining the texture coordinates of multiple key points in a target depth image, for any key point in the target depth image, i.e., the first key point, we can first determine its two-dimensional spatial coordinates and the first depth value in the target depth image, and then determine the three-dimensional space corresponding to the basic three-dimensional model, i.e., the surface of the basic three-dimensional model and the second key point corresponding to the first key point. In the three-dimensional space, if the surface where the second key point is located is taken as a reference, the first key point is located in the normal direction of the second key point.

[0085] After determining the second keypoint, the corresponding second depth value can be determined. Then, based on the first depth value of the first keypoint and the second depth value of the second keypoint, the distance between the first and second keypoints (the displacement value along the normal direction of the 3D model surface) can be determined. Based on this distance and the 3D spatial coordinates of the second keypoint in 3D space, the 3D spatial coordinates of the first keypoint in 3D space can be determined. Finally, based on these 3D spatial coordinates, the corresponding texture coordinates of the first keypoint can be determined. These texture coordinates define the position information of the first keypoint in the depth image and are simultaneously associated with the 3D model, allowing the determination of the position of each keypoint in the 2D image on the texture map of the 3D model surface based on the texture coordinates.

[0086] In specific implementation, generating a texture displacement map corresponding to the target object based on the texture coordinates includes:

[0087] Generate a texture displacement map corresponding to the target depth image based on the texture coordinates;

[0088] The texture displacement maps corresponding to multiple depth images are fused to generate the texture displacement map corresponding to the target object.

[0089] Specifically, after determining the texture coordinates of any key point in the target depth image in the corresponding 3D space of the base 3D model, a texture displacement map corresponding to the target depth image can be generated based on the texture coordinates of each point. Then, the texture displacement maps corresponding to each target depth image are integrated to generate the texture displacement map corresponding to the target object. Integrating the texture displacement maps corresponding to each target depth image involves identifying the overlapping parts in each texture displacement map and superimposing the texture displacement maps based on the overlapping parts to generate a complete texture displacement map of the target object.

[0090] A schematic diagram of a texture displacement map generation process provided in the embodiments of this specification is shown below. Figure 3a As shown. Figure 3a This demonstrates the process of generating a texture displacement map of a target human body. Taking key point A in the target human body's clothing as an example, due to the wrinkles in the clothing, there is a certain distance between key point A and the surface of the human skin. In order to add the texture information corresponding to the clothing of the target human body to the 3D model to make the constructed target 3D model more realistic, the surface of the basic 3D model and key point B corresponding to key point A can be determined. Then, based on the depth values ​​(distance between key point A and key point B and the camera) and the 3D spatial coordinates (or texture coordinates) of key point B, the texture coordinates of key point A are determined. Thus, based on the texture coordinates of each key point of the clothing, a texture displacement map of the target human body is generated. Then, the texture displacement maps are integrated to generate the texture displacement map corresponding to the target object.

[0091] In specific implementation, when constructing a basic 3D model through a 3D model construction sub-network within a 3D model construction network, generating a texture displacement map corresponding to the target object based on the texture coordinates includes:

[0092] Based on the texture coordinates of the multiple key points in the target depth image in the three-dimensional space, a texture displacement map corresponding to the target depth image is generated, wherein the target depth image is any one of the multiple depth images;

[0093] The displacement fusion subnetwork in the network constructed by the three-dimensional model is used to fuse the texture displacement maps corresponding to the multiple depth images to generate the texture displacement map corresponding to the target object.

[0094] Specifically, the displacement fusion subnetwork can be a D-NET (displacement fusion net).

[0095] After generating the texture displacement map corresponding to the target depth image based on the texture coordinates of multiple key points in the target depth image in three-dimensional space, the D-NET network in the three-dimensional model construction network can be used to fuse the texture displacement maps corresponding to multiple depth images to generate the texture displacement map corresponding to the target object.

[0096] Step 206: Obtain the color images corresponding to the multiple depth images respectively, and determine the texture map corresponding to the target object based on the color values ​​corresponding to the multiple key points in the color images and the mapping relationship between the texture coordinates and the two-dimensional spatial coordinates.

[0097] Specifically, as mentioned earlier, the depth image and color image obtained by an RGBD camera from the same viewpoint have a corresponding relationship. It can be seen that the depth image and its corresponding color image are obtained by capturing the same content; they simply contain different information about that content. The depth image contains the depth value of the keypoint corresponding to the content, and the color image contains the color value of the keypoint. Furthermore, the same keypoint is located in the same position in both the depth and color images, meaning its two-dimensional spatial coordinates are the same. Similarly, after constructing a target 3D model based on this depth image and its corresponding color image, the texture coordinates of the same keypoint in the depth and color images are the same in the target 3D model.

[0098] Therefore, after determining the texture coordinates of multiple key points in the depth image in three-dimensional space, the mapping relationship between the texture coordinates and the two-dimensional spatial coordinates of the key points in the depth image can be determined. Based on this mapping relationship, the texture coordinates of each key point in the color image can be determined. Based on the texture coordinates and color values ​​of each key point in the color image, the texture map corresponding to the target object can be determined.

[0099] In specific implementation, the texture map corresponding to the target object is determined based on the color values ​​corresponding to the multiple key points in the color image, and the mapping relationship between the texture coordinates and the two-dimensional spatial coordinates, including:

[0100] Determine the mapping relationship between the two-dimensional spatial coordinates and the texture coordinates;

[0101] Based on the mapping relationship, the texture coordinates of multiple key points of the target object in the three-dimensional space are determined in the target color image, wherein the target color image is any one of multiple color images;

[0102] Determine the color value corresponding to the target key point among the plurality of key points, and establish the association between the color value and the texture coordinates of the target key point, wherein the target key point is any one of the plurality of key points;

[0103] Based on the texture coordinates, the color values, and the correlation, the texture map corresponding to the target object is determined.

[0104] Furthermore, based on the texture coordinates, the color values, and the correlation, determining the texture map corresponding to the target object includes:

[0105] Based on the texture coordinates, the color values, and the correlation, a texture map corresponding to the target color image is generated;

[0106] The texture maps corresponding to multiple color images are fused to generate the texture map corresponding to the target object.

[0107] Specifically, by processing any one of the multiple depth images individually—that is, by determining the texture coordinates of multiple key points in the three-dimensional space corresponding to the basic three-dimensional model based on the two-dimensional spatial coordinates and depth values ​​of the target object in the arbitrary depth image (target depth image)—the mapping relationship between the two-dimensional spatial coordinates and the texture coordinates can be determined. Based on this mapping relationship, the texture coordinates of multiple key points of the target object in the three-dimensional space in the target color image corresponding to the target depth image can be determined. Then, the color value corresponding to the target key point among the multiple key points can be determined, and the association relationship between the color value and the texture coordinates of the target key point can be established. Here, the target key point is any one of the multiple key points.

[0108] After determining the relationship between color values ​​and texture coordinates, a texture map corresponding to the target color image can be generated based on the texture coordinates of each point. Then, the texture maps corresponding to each target color image are integrated to generate a texture map corresponding to the target object. Integrating the texture maps corresponding to each target color image involves identifying the overlapping parts in each texture map and overlaying them based on these overlapping parts to generate a complete texture map of the target object.

[0109] Furthermore, when constructing a basic 3D model through a 3D model construction subnetwork within a 3D model construction network, the texture map corresponding to the target object is determined based on the color values ​​corresponding to the multiple key points in the color image, and the mapping relationship between the texture coordinates and the 2D spatial coordinates. This includes:

[0110] Based on the texture coordinates of the multiple key points in the three-dimensional space, the color values ​​corresponding to the multiple key points, and the correlation between the color values ​​and the texture coordinates in the target color image, a texture map corresponding to the target color image is generated.

[0111] The color fusion subnetwork in the network is constructed using the 3D model to fuse the texture maps corresponding to multiple color images to generate the texture map corresponding to the target object.

[0112] Specifically, the color fusion sub-network can be T-NET.

[0113] Based on the texture coordinates of multiple key points in the target color image in three-dimensional space, the color values ​​corresponding to the multiple key points, and the correlation between the color values ​​and texture coordinates, after generating the texture map corresponding to the target color image, the T-NET network in the three-dimensional model construction network can be used to fuse the texture maps corresponding to multiple color images to generate the texture map corresponding to the target object.

[0114] A schematic diagram of an image fusion process provided in the embodiments of this specification is shown below. Figure 3b As shown. Figure 3b The image fusion process for the target human body is demonstrated. For multiple depth images, the texture coordinates of multiple key points in the three-dimensional space corresponding to the basic three-dimensional model are first determined based on the two-dimensional spatial coordinates and depth values ​​of multiple key points of the target object in each depth image. Then, a texture displacement map corresponding to the target depth image is generated based on the texture coordinates. The multiple texture displacement maps are then fused through a displacement fusion sub-network to generate the texture displacement map of the target object. For multiple color images, texture maps corresponding to each color image are generated based on the texture coordinates of multiple key points in the three-dimensional space, the color values ​​of the multiple key points, and the correlation between the color values ​​and texture coordinates. Then, a color fusion sub-network in the three-dimensional model construction network is used to fuse the texture maps corresponding to each color image to generate the texture map corresponding to the target object.

[0115] Alternatively, the displacement fusion subnetwork can process multiple texture displacement maps and the texture map output by the color fusion subnetwork together to generate a texture displacement map of the target object, making the generated result more accurate.

[0116] Furthermore, after the displacement fusion subnetwork outputs a texture displacement map, its loss value can be calculated based on the output texture displacement map and the target object's real texture displacement map. The network parameters of the displacement fusion subnetwork can then be adjusted based on the loss value, thereby making the output result of the displacement fusion subnetwork more accurate. Similarly, after the color fusion subnetwork outputs a texture map, its loss value can be calculated based on the output texture map and the target object's real texture map. The network parameters of the color fusion subnetwork can then be adjusted based on the loss value, thereby making the output result of the color fusion subnetwork more accurate.

[0117] In practical applications, the D-NET and T-NET network structures adopt an Encoder-Decoder structure. The Encoder extracts features from the multi-view local input, and the Decoder structure decodes the features to obtain the complete UV-displacement map and UV-texture map of the target object.

[0118] Step 208: Overlay the basic 3D model, the texture displacement map, and the texture map to generate the target 3D model of the target object.

[0119] Specifically, after constructing the basic 3D model of the target object and generating the corresponding texture displacement map and texture map, the three can be superimposed to generate the target 3D model of the target object, which has a texture map.

[0120] This specification provides a schematic diagram of a human body 3D model construction process in its embodiments, as shown below. Figure 3c As shown. Since the basic 3D model does not contain texture maps, i.e., it does not contain color information, and the texture displacement map only contains the texture coordinates of each key point of the target object, it also does not contain color information. Only the texture map contains the color information of each key point of the target object. Therefore, when superimposing the above three, the basic 3D model and the texture displacement map can be superimposed first to generate an intermediate 3D model. This intermediate 3D model can show the target object and its decorations, such as clothes and hats, but it cannot show the skin color of the target object or the color of the decorations. Therefore, the intermediate 3D model and the texture map can be superimposed to generate the target 3D model. The target 3D model can show the skin color or the color of the decorations, making the effect of the target 3D model more realistic.

[0121] If the SMPL estimation result (basic 3D model) obtained from the multi-view RGBD input is denoted as V_SMPL, and the texture displacement map (UV-displacement map) of the target object is denoted as M_pred, then the basic 3D model and the texture displacement map are superimposed, and the corresponding superposition method is shown in Formula 2.

[0122] V = VSMPL +N SMPL *M pred Formula 2

[0123] Where N_SMPL is the normal to the upsampled vertex of the SMPL model, and * indicates element-wise multiplication.

[0124] A schematic diagram of another three-dimensional model construction process provided in the embodiments of this specification is shown in 3d. Figure 3d First, a basic 3D model of the target human body is constructed based on multiple depth images of the target human body. Then, based on the 2D spatial coordinates and depth values ​​corresponding to multiple key points of the target human body in the multiple depth images, the texture coordinates of multiple key points in the corresponding 3D space of the basic 3D model are determined, and a texture displacement map corresponding to the target human body is generated based on the texture coordinates. Next, color images corresponding to the multiple depth images are obtained, and the texture map corresponding to the target human body is determined based on the color values ​​corresponding to multiple key points in the color images and the mapping relationship between texture coordinates and 2D spatial coordinates. Finally, the basic 3D model, texture displacement map, and texture map are superimposed to generate the target 3D model of the target human body.

[0125] In the embodiments of this specification, after constructing a basic 3D model using the SMPL model, the depth map can be converted into a UV-displacement map, where the displacement values ​​represent the geometric details of the key points, and the color map can be converted into a UV-texture map. After determining the SMPL fitting results, the UV-displacement map, and the UV-texture map, the three can be obtained by simple addition to obtain a human body model containing geometric details. Figure 3a The physical meaning of UV-displacement is illustrated using a magnified view of a complete human body model. For a specific viewpoint in multi-view input, only local depth is captured. The embodiments in this specification calculate the visibility of each pixel in the UV-displacement map based on the camera parameters of that viewpoint, and then perform a spatial transformation on the visible area of ​​the given viewpoint. Color information from a specific viewpoint in multi-view input is also obtained as a local UV-texture map using a similar method.

[0126] In addition, the embodiments of this specification construct a fusion network FusionNet to generate a complete UV map. The fusion network includes two sub-networks, which are used to integrate multi-view depth maps into a complete UV-displacement map and to integrate multi-view color maps into a complete UV-texture map.

[0127] This specification's embodiments use SMPL estimation as a shape prior and spatial medium, establishing a correlation between multi-view RGB images and depth images and the UV space corresponding to the SMPL topology. This process involves UV space transformation, resulting in UV-displacement maps and UV-texture maps. Specifically, multi-view information is projected onto a 2D UV-mapping space and fused. Compared to schemes such as constructing pose maps for optimization, this decouples human motion pose estimation from geometric detail estimation, thus making geometric reconstruction unaffected by the degrees of freedom of motion pose and exhibiting better robustness to complex motions.

[0128] In the geometry reconstruction stage, this embodiment transforms depth information based on SMPL priors, and the defined UV displacement better addresses geometric details. Furthermore, this embodiment performs overall inference on the surface shape of the human model during training and testing, resulting in higher operational efficiency compared to implicit function representations. In the texture reconstruction stage, this embodiment directly fuses color information in the UV-mapping space, i.e., the texture space. The output can directly establish a texture mapping relationship with the reconstructed geometry. By using the SMPL model, human geometry and texture are reconstructed from multi-view RGBD input and unified to the UV-mapping space. This 2D space is consistent with the 3D geometric topology of SMPL, reducing the topological dimension of the fusion process and eliminating the need for a time-consuming texture reconstruction optimization process. This reduces computational load while simultaneously obtaining high-quality geometry and texture, thus improving model construction efficiency.

[0129] Furthermore, the fusion within the UV mapping space provides robustness to sparse viewpoints, requiring only that the number of viewpoints cover the entire mapping space, without mandating sufficient overlap between viewpoints. The constructed target 3D model is obtained by overlaying the fused UV-displacement map and UV texture map onto the SMPL surface, thus directly sharing the same parameter properties as SMPL. This allows for direct integration with SMPL-based animation and special effects work without additional processing.

[0130] One embodiment of this specification constructs a basic 3D model of a target object based on multiple depth images of the target object. According to the 2D spatial coordinates and depth values ​​corresponding to multiple key points of the target object in the multiple depth images, the texture coordinates of the multiple key points in the 3D space corresponding to the basic 3D model are determined. A texture displacement map corresponding to the target object is generated based on the texture coordinates. Color images corresponding to the multiple depth images are obtained, and a texture map corresponding to the target object is determined based on the color values ​​corresponding to the multiple key points in the color images and the mapping relationship between the texture coordinates and the 2D spatial coordinates. The basic 3D model, the texture displacement map, and the texture map are superimposed to generate a target 3D model of the target object.

[0131] This specification's embodiments establish a correlation between multi-view color images and depth images and the UV space corresponding to the basic 3D model. Specifically, UV space transformation is performed on the multi-view color images and depth images based on the UV space, resulting in UV-displacement maps and UV-texture maps, respectively. Then, the basic 3D model, UV-displacement maps, and UV-texture maps can be superimposed to generate the target 3D model of the target object. Through this processing method, the realism of the constructed target 3D model can be guaranteed without the need for a time-consuming texture reconstruction and optimization process. This reduces the computational load of the model construction process and simultaneously obtains high-quality UV-displacement maps and UV-texture maps, thereby improving the accuracy and efficiency of the model construction results.

[0132] The following is in conjunction with the appendix Figure 4 Taking the application of the 3D model construction method provided in this specification in the scenario of constructing a 3D human body model as an example, the 3D model construction method will be further explained. Among them, Figure 4 The present specification shows a flowchart of a three-dimensional model construction method according to an embodiment, which includes the following steps.

[0133] Step 402: Obtain multiple depth images of the target human body and corresponding color images for each depth image.

[0134] Among them, multiple depth images and color images were obtained by an RGBD camera, and the depth images and color images obtained from the same viewpoint have a corresponding relationship.

[0135] Step 404: Input multiple depth images into the parameterized human body model for processing to generate a basic 3D model of the target human body.

[0136] Specifically, the basic 3D model does not include texture mapping.

[0137] Step 406: Determine the two-dimensional spatial coordinates and the first depth value corresponding to the first key point of the target human body in the target depth image.

[0138] The first key point is any one of the multiple key points of the target human body in the target depth image.

[0139] Step 408: Determine the second key point in the three-dimensional space corresponding to the first key point of the basic three-dimensional model, and determine the second depth value corresponding to the second key point.

[0140] In three-dimensional space, the first key point is located in the normal direction of the second key point.

[0141] Step 410: Determine the texture coordinates of the first keypoint in three-dimensional space based on the first depth value, the second depth value, and the three-dimensional spatial coordinates of the second keypoint.

[0142] Step 412: Generate a texture displacement map corresponding to the target depth image based on texture coordinates.

[0143] Step 414: Using a displacement fusion network, the texture displacement maps corresponding to multiple depth images are fused to generate the texture displacement map corresponding to the target human body.

[0144] Step 416: Determine the mapping relationship between two-dimensional spatial coordinates and texture coordinates.

[0145] Step 418: Determine the texture coordinates of multiple key points of the target object in three-dimensional space in the target color image according to the mapping relationship, wherein the target color image is any one of multiple color images.

[0146] Step 420: Determine the color value corresponding to the target key point among multiple key points, and establish the association between the color value and the texture coordinates of the target key point, wherein the target key point is any one of the multiple key points.

[0147] Step 422: Determine the texture map corresponding to the target human body based on texture coordinates, color values, and correlation relationships.

[0148] Step 424: The texture maps corresponding to multiple color images are fused using a color fusion network to generate the texture map corresponding to the target human body.

[0149] Step 426: Overlay the basic 3D model, texture displacement map, and texture map to generate the target 3D model of the target human body.

[0150] The target 3D model is one with texture mapping.

[0151] In addition, after constructing the basic 3D model of the target human body through the parametric human body model, a reference 3D model of the target human body can be obtained, and the first 3D spatial coordinates of multiple key points in the reference 3D model can be determined, as well as the second 3D spatial coordinates of multiple key points in the basic 3D model. Based on the first and second 3D spatial coordinates, the loss value corresponding to the parametric human body model can be determined, so as to optimize the shape parameters and posture parameters of the parametric human body model according to the loss value.

[0152] This specification's embodiments establish a correlation between multi-view color images and depth images and the UV space corresponding to the basic 3D model. Specifically, UV space transformation is performed on the multi-view color images and depth images based on the UV space, resulting in UV-displacement maps and UV-texture maps, respectively. Then, the basic 3D model, UV-displacement maps, and UV-texture maps can be superimposed to generate a target 3D model of the target human body. This processing method ensures the realism of the constructed target 3D model while eliminating the need for a time-consuming texture reconstruction and optimization process. This reduces the computational load of the model construction process and simultaneously obtains high-quality UV-displacement maps and UV-texture maps, thereby improving the accuracy and efficiency of the model construction results.

[0153] Corresponding to the above method embodiments, this specification also provides embodiments of a three-dimensional model construction apparatus. Figure 5 A schematic diagram of a three-dimensional model building apparatus according to one embodiment of this specification is shown. Figure 5 As shown, the device includes:

[0154] Construction module 502 is configured to construct a basic 3D model of the target object based on multiple depth images of the target object;

[0155] The determination module 504 is configured to determine the texture coordinates of the multiple key points in the three-dimensional space corresponding to the basic three-dimensional model based on the two-dimensional spatial coordinates and depth values ​​of the multiple key points of the target object in the multiple depth images, and generate a texture displacement map corresponding to the target object based on the texture coordinates.

[0156] The acquisition module 506 is configured to acquire color images corresponding to the multiple depth images respectively, and determine the texture map corresponding to the target object based on the color values ​​corresponding to the multiple key points in the color images and the mapping relationship between the texture coordinates and the two-dimensional spatial coordinates.

[0157] The generation module 508 is configured to superimpose the basic 3D model, the texture displacement map, and the texture map to generate a target 3D model of the target object.

[0158] Optionally, the determining module 504 is further configured to:

[0159] Based on the two-dimensional spatial coordinates and depth values ​​corresponding to multiple key points of the target object in the target depth image, the texture coordinates of the multiple key points in the three-dimensional space corresponding to the basic three-dimensional model are determined, wherein the target depth image is any one of the multiple depth images.

[0160] Optionally, the determining module 504 is further configured to:

[0161] In the target depth image, determine the two-dimensional spatial coordinates and the first depth value corresponding to the first key point of the target object, wherein the first key point is any one of a plurality of key points of the target object in the target depth image;

[0162] In the three-dimensional space corresponding to the basic three-dimensional model, a second key point is determined that corresponds to the first key point, and a second depth value is determined that corresponds to the second key point. In the three-dimensional space, the first key point is located in the normal direction of the second key point.

[0163] Based on the first depth value, the second depth value, and the three-dimensional spatial coordinates of the second key point in the three-dimensional space, the texture coordinates of the first key point in the three-dimensional space are determined.

[0164] Optionally, the determining module 504 is further configured to:

[0165] Generate a texture displacement map corresponding to the target depth image based on the texture coordinates;

[0166] The texture displacement maps corresponding to multiple depth images are fused to generate the texture displacement map corresponding to the target object.

[0167] Optionally, the acquisition module 506 is further configured to:

[0168] Determine the mapping relationship between the two-dimensional spatial coordinates and the texture coordinates;

[0169] Based on the mapping relationship, the texture coordinates of multiple key points of the target object in the three-dimensional space are determined in the target color image, wherein the target color image is any one of multiple color images;

[0170] Determine the color value corresponding to the target key point among the plurality of key points, and establish the association between the color value and the texture coordinates of the target key point, wherein the target key point is any one of the plurality of key points;

[0171] Based on the texture coordinates, the color values, and the correlation, the texture map corresponding to the target object is determined.

[0172] Optionally, the acquisition module 506 is further configured to:

[0173] Based on the texture coordinates, the color values, and the correlation, a texture map corresponding to the target color image is generated;

[0174] The texture maps corresponding to multiple color images are fused to generate the texture map corresponding to the target object.

[0175] Optionally, the building module 502 is further configured to:

[0176] Multiple depth images of a target object are acquired, along with color images corresponding to each of the multiple depth images. The target depth image and the color image corresponding to the target depth image are generated by shooting from the same shooting angle, and the target depth image is any one of the multiple depth images.

[0177] The multiple depth images are input into a 3D model construction network. The 3D model construction sub-network in the 3D model construction network processes the multiple depth images to generate a basic 3D model of the target object.

[0178] Optionally, the determining module 504 is further configured to:

[0179] Based on the texture coordinates of the multiple key points in the target depth image in the three-dimensional space, a texture displacement map corresponding to the target depth image is generated;

[0180] The displacement fusion subnetwork in the network constructed by the three-dimensional model is used to fuse the texture displacement maps corresponding to the multiple depth images to generate the texture displacement map corresponding to the target object.

[0181] Optionally, the acquisition module 506 is further configured to:

[0182] Based on the texture coordinates of the multiple key points in the three-dimensional space, the color values ​​corresponding to the multiple key points, and the correlation between the color values ​​and the texture coordinates in the target color image, a texture map corresponding to the target color image is generated.

[0183] The color fusion subnetwork in the network is constructed using the 3D model to fuse the texture maps corresponding to multiple color images to generate the texture map corresponding to the target object.

[0184] Optionally, the target object includes a target human body, and the 3D model construction subnetwork includes a parameterized human body subnetwork;

[0185] Accordingly, the device further includes a processing module configured to:

[0186] Obtain a reference 3D model of the target object, and determine the first 3D spatial coordinates of multiple key points in the reference 3D model;

[0187] Determine the second three-dimensional spatial coordinates of multiple key points in the basic three-dimensional model;

[0188] Based on the first three-dimensional spatial coordinates and the second three-dimensional spatial coordinates, the loss value corresponding to the parameterized human sub-network is determined;

[0189] The shape and pose parameters of the parameterized human subnetwork are optimized based on the loss value.

[0190] The above is a schematic scheme of a three-dimensional model building device according to this embodiment. It should be noted that the technical solution of this three-dimensional model building device and the technical solution of the three-dimensional model building method described above belong to the same concept. For details not described in detail in the technical solution of the three-dimensional model building device, please refer to the description of the technical solution of the three-dimensional model building method described above.

[0191] Figure 6 A structural block diagram of a computing device 600 according to one embodiment of this specification is shown. The components of the computing device 600 include, but are not limited to, a memory 610 and a processor 620. The processor 620 is connected to the memory 610 via a bus 630, and a database 650 is used to store data.

[0192] The computing device 600 also includes an access device 640, which enables the computing device 600 to communicate via one or more networks 660. Examples of these networks include a Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the Internet. The access device 640 may include one or more of any type of wired or wireless network interface (e.g., a Network Interface Card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) interface, a Wi-MAX interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0193] In one embodiment of this specification, the above-described components of the computing device 600 and Figure 6 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 6 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0194] The computing device 600 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or PCs. The computing device 600 can also be a mobile or stationary server.

[0195] The processor 620 is used to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above-described three-dimensional model construction method.

[0196] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the above-described three-dimensional model construction method belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the above-described three-dimensional model construction method.

[0197] An embodiment of this specification also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the above-described three-dimensional model construction method.

[0198] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium and the technical solution of the above-described three-dimensional model construction method belong to the same concept. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the above-described three-dimensional model construction method.

[0199] An embodiment of this specification also provides a computer program, wherein when the computer program is executed in a computer, it causes the computer to perform the steps of the above-described three-dimensional model construction method.

[0200] The above is an illustrative example of a computer program according to this embodiment. It should be noted that the technical solution of this computer program and the technical solution of the aforementioned three-dimensional model construction method belong to the same concept. Details not described in detail in the computer program's technical solution can be found in the description of the technical solution of the aforementioned three-dimensional model construction method.

[0201] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0202] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added to or subtracted according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0203] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0204] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0205] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A method for constructing a three-dimensional model, comprising: Based on multiple depth images of the target object, a basic 3D model of the target object is constructed. Based on the two-dimensional spatial coordinates and depth values ​​corresponding to multiple key points of the target object in the multiple depth images, the texture coordinates of the multiple key points in the three-dimensional space corresponding to the basic three-dimensional model are determined, and a texture displacement map corresponding to the target object is generated based on the texture coordinates. Obtain the color images corresponding to the multiple depth images respectively, and determine the texture map corresponding to the target object based on the color values ​​corresponding to the multiple key points in the color images and the mapping relationship between the texture coordinates and the two-dimensional spatial coordinates. The base 3D model, the texture displacement map, and the texture map are superimposed to generate the target 3D model of the target object.

2. The three-dimensional model construction method according to claim 1, wherein determining the texture coordinates of the multiple key points in the three-dimensional space corresponding to the basic three-dimensional model based on the two-dimensional spatial coordinates and depth values ​​corresponding to the multiple key points of the target object in the multiple depth images includes: Based on the two-dimensional spatial coordinates and depth values ​​corresponding to multiple key points of the target object in the target depth image, the texture coordinates of the multiple key points in the three-dimensional space corresponding to the basic three-dimensional model are determined, wherein the target depth image is any one of the multiple depth images.

3. The three-dimensional model construction method according to claim 2, wherein determining the texture coordinates of the multiple key points in the three-dimensional space corresponding to the basic three-dimensional model based on the two-dimensional spatial coordinates and depth values ​​corresponding to the multiple key points of the target object in the target depth image includes: Determine the two-dimensional spatial coordinates and the first depth value corresponding to the first key point of the target object in the target depth image, wherein the first key point is any one of a plurality of key points of the target object in the target depth image; In the three-dimensional space corresponding to the basic three-dimensional model, a second key point corresponding to the first key point is determined, and a second depth value corresponding to the second key point is determined, wherein in the three-dimensional space, the first key point is located in the normal direction of the second key point; Based on the first depth value, the second depth value, and the three-dimensional spatial coordinates of the second key point in the three-dimensional space, the texture coordinates of the first key point in the three-dimensional space are determined.

4. The three-dimensional model construction method according to claim 2 or 3, wherein generating the texture displacement map corresponding to the target object based on the texture coordinates includes: Generate a texture displacement map corresponding to the target depth image based on the texture coordinates; The texture displacement maps corresponding to multiple depth images are fused to generate the texture displacement map corresponding to the target object.

5. The three-dimensional model construction method according to claim 1, wherein determining the texture map corresponding to the target object based on the color values ​​corresponding to the plurality of key points in the color image and the mapping relationship between the texture coordinates and the two-dimensional spatial coordinates includes: Determine the mapping relationship between the two-dimensional spatial coordinates and the texture coordinates; Based on the mapping relationship, the texture coordinates of multiple key points of the target object in the three-dimensional space are determined in the target color image, wherein the target color image is any one of multiple color images; Determine the color value corresponding to the target key point among the plurality of key points, and establish the association between the color value and the texture coordinates of the target key point, wherein the target key point is any one of the plurality of key points; Based on the texture coordinates, the color values, and the correlation, the texture map corresponding to the target object is determined.

6. The three-dimensional model construction method according to claim 5, wherein determining the texture map corresponding to the target object based on the texture coordinates, the color value, and the correlation relationship includes: Based on the texture coordinates, the color values, and the correlation, a texture map corresponding to the target color image is generated; The texture maps corresponding to multiple color images are fused to generate the texture map corresponding to the target object.

7. The three-dimensional model construction method according to claim 1, wherein constructing a basic three-dimensional model of the target object based on multiple depth images of the target object comprises: Acquire multiple depth images of the target object; The multiple depth images are input into a 3D model construction network. The 3D model construction sub-network in the 3D model construction network processes the multiple depth images to generate a basic 3D model of the target object.

8. The three-dimensional model construction method according to claim 7, wherein generating the texture displacement map corresponding to the target object based on the texture coordinates comprises: Based on the texture coordinates of the multiple key points in the target depth image in the three-dimensional space, a texture displacement map corresponding to the target depth image is generated, wherein the target depth image is any one of the multiple depth images; The displacement fusion subnetwork in the network constructed by the three-dimensional model is used to fuse the texture displacement maps corresponding to the multiple depth images to generate the texture displacement map corresponding to the target object.

9. The three-dimensional model construction method according to claim 8, wherein determining the texture map corresponding to the target object based on the color values ​​corresponding to the plurality of key points in the color image and the mapping relationship between the texture coordinates and the two-dimensional spatial coordinates includes: Based on the texture coordinates of the multiple key points in the three-dimensional space, the color values ​​corresponding to the multiple key points, and the correlation between the color values ​​and the texture coordinates in the target color image, a texture map corresponding to the target color image is generated. The color fusion subnetwork in the network is constructed using the 3D model to fuse the texture maps corresponding to multiple color images to generate the texture map corresponding to the target object.

10. The three-dimensional model construction method according to claim 7, wherein the target object includes a target human body, and the three-dimensional model construction sub-network includes a parameterized human body sub-network; Accordingly, the method further includes: Obtain a reference 3D model of the target object, and determine the first 3D spatial coordinates of multiple key points in the reference 3D model; Determine the second three-dimensional spatial coordinates of multiple key points in the basic three-dimensional model; Based on the first three-dimensional spatial coordinates and the second three-dimensional spatial coordinates, the loss value corresponding to the parameterized human sub-network is determined; The shape and pose parameters of the parameterized human subnetwork are optimized based on the loss value.

11. A three-dimensional model building device, comprising: The construction module is configured to construct a basic 3D model of the target object based on multiple depth images of the target object; The determination module is configured to determine the texture coordinates of the multiple key points in the three-dimensional space corresponding to the basic three-dimensional model based on the two-dimensional spatial coordinates and depth values ​​of the multiple key points of the target object in the multiple depth images, and generate a texture displacement map corresponding to the target object based on the texture coordinates. The acquisition module is configured to acquire color images corresponding to the multiple depth images respectively, and determine the texture map corresponding to the target object based on the color values ​​corresponding to the multiple key points in the color images and the mapping relationship between the texture coordinates and the two-dimensional space coordinates. The generation module is configured to overlay the base 3D model, the texture displacement map, and the texture map to generate a target 3D model of the target object.

12. A computing device, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the three-dimensional model construction method according to any one of claims 1 to 10.

13. A computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the three-dimensional model construction method according to any one of claims 1 to 10.

Images