Training method and device of three-dimensional reconstruction model, equipment and storage medium

Abstract

The application discloses a three-dimensional reconstruction model training method and device, equipment and a storage medium, and relates to the technical field of artificial intelligence. The method comprises the following steps: obtaining a first depth image of a target object, the first depth image being a real depth image obtained when the target object is photographed, or a depth image obtained based on a real color image obtained by photographing the target object; obtaining three-dimensional reconstruction information of the target object according to the first depth image by a three-dimensional reconstruction model; generating a second depth image of the target object according to the three-dimensional reconstruction information; and training the three-dimensional reconstruction model based on the first depth image and the second depth image. The method trains the three-dimensional reconstruction model by using a real depth image obtained when the target object is photographed or a depth image obtained based on a real color image obtained by photographing the target object, and has stronger generalization and high fidelity.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a training method, apparatus, device, and storage medium for a three-dimensional reconstruction model. Background Technology

[0002] With the advancement of technology, reconstructing three-dimensional geometric configurations from images is becoming increasingly important.

[0003] In related technologies, the 3D reconstruction model is generally trained using synthetic data with real labels, which means that the model's output is trained and learned in a fully supervised manner with the real labels of the synthetic data.

[0004] Then, synthetic data with real labels is used to train the model. Compared with real data, synthetic data does not fully represent the details of objects, and the objects that synthetic data can represent are also relatively limited. Therefore, the fidelity is poor and the generalization is insufficient. Summary of the Invention

[0005] This application provides a method, apparatus, device, and storage medium for training a three-dimensional reconstruction model. The technical solution is as follows:

[0006] According to one aspect of the embodiments of this application, a method for training a three-dimensional reconstruction model is provided, the method comprising:

[0007] A first depth image of the target object is obtained, wherein the first depth image is a true depth image obtained when the target object is photographed, or a depth image obtained based on a true color image of the target object.

[0008] The three-dimensional reconstruction model obtains the three-dimensional reconstruction information of the target object based on the first depth image, and the three-dimensional reconstruction information is used to determine the three-dimensional geometric configuration of the target object in three-dimensional space;

[0009] Based on the three-dimensional reconstruction information, a second depth image of the target object is generated;

[0010] The 3D reconstruction model is trained based on the first depth image and the second depth image.

[0011] According to one aspect of the embodiments of this application, a training apparatus for a three-dimensional reconstruction model is provided, the apparatus comprising:

[0012] The image acquisition module is used to acquire a first depth image of the target object. The first depth image is a real depth image obtained when the target object is photographed, or a depth image obtained based on a real color image of the target object.

[0013] The information generation module is used to obtain the three-dimensional reconstruction information of the target object based on the first depth image through the three-dimensional reconstruction model. The three-dimensional reconstruction information is used to determine the three-dimensional geometric configuration of the target object in three-dimensional space.

[0014] An image generation module is used to generate a second depth image of the target object based on the three-dimensional reconstruction information;

[0015] The model training module is used to train the 3D reconstruction model based on the first depth image and the second depth image.

[0016] According to one aspect of the embodiments of this application, a computer device is provided, the computer device including a processor and a memory, the memory storing a computer program, the computer program being loaded and executed by the processor to implement the above-described method.

[0017] According to one aspect of the embodiments of this application, a computer-readable storage medium is provided, wherein a computer program is stored in the storage medium, the computer program being loaded and executed by a processor to implement the above-described method.

[0018] According to one aspect of the embodiments of this application, a computer program product is provided, the computer program product including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, causing the computer device to perform the method described above.

[0019] The technical solutions provided in this application embodiment may have the following beneficial effects:

[0020] First, by obtaining a true depth image of the target object captured in a photograph, or a depth image obtained from a true color image of the target object, and combining this with the depth image obtained through the 3D reconstruction model, the model parameters are trained. This achieves self-supervised model training, allowing the model to be trained using only the target object's data without the need for other data, thus reducing the model's training cost. Second, using real data of the target object (true depth images or depth images obtained from true color images) to train the 3D reconstruction model results in clearer depiction of the target object's texture details. Furthermore, real data covers a wider range, allowing the trained model to generalize more effectively. Additionally, because real data more fully represents the details of the target object, the trained model exhibits better fidelity. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the implementation environment of a solution provided in one embodiment of this application;

[0022] Figure 2 This is a schematic diagram illustrating an application scenario of a three-dimensional reconstruction model provided in one embodiment of this application;

[0023] Figure 3 This is a flowchart of a training method for a three-dimensional reconstruction model provided in one embodiment of this application;

[0024] Figure 4 This is a schematic diagram of the training results provided in one embodiment of this application;

[0025] Figure 5 This is a schematic diagram of the training results provided in another embodiment of this application;

[0026] Figure 6 This is a flowchart of a training method for a three-dimensional reconstruction model provided in another embodiment of this application;

[0027] Figure 7 This is a flowchart of a training method for a three-dimensional reconstruction model provided in another embodiment of this application;

[0028] Figure 8 This is a flowchart of a training method for a three-dimensional reconstruction model provided in another embodiment of this application;

[0029] Figure 9 This is a block diagram of a training device for a three-dimensional reconstruction model provided in one embodiment of this application;

[0030] Figure 10 This is a block diagram of a training device for a three-dimensional reconstruction model provided in another embodiment of this application;

[0031] Figure 11 This is a structural block diagram of a computer device provided in one embodiment of this application. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0033] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0034] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0035] Computer vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in recognizing and measuring targets, and further processes images to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR (optical character recognition), video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D (three-dimensional) technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0036] Natural Language Processing (NLP) is an important field within computer science and artificial intelligence. It studies the theories and methods for enabling effective communication between humans and computers using natural language. NLP is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field involves natural language—the language people use in daily life—and thus it has a close relationship with linguistic research. NLP techniques typically include text processing, semantic understanding, machine translation, question answering, and knowledge graphs.

[0037] Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, and inductive learning.

[0038] With the research and advancement of artificial intelligence (AI) technology, AI is being studied and applied in various fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, robots, smart healthcare, and smart customer service. It is believed that with the development of technology, AI will be applied in more fields and play an increasingly important role.

[0039] The solutions provided in this application involve technologies such as computer vision in artificial intelligence, and are specifically illustrated through the following embodiments.

[0040] Before introducing the embodiments of this application, the terms appearing in this solution are explained as follows for ease of understanding.

[0041] Synthetic data: The synthetic data described in this application embodiment can be considered as data not obtained based on real-world objects, but rather as simulated data from the real world. In scenarios where collecting real data would be dangerous, synthetic data can be used to train the model, thus significantly reducing the risk.

[0042] Real data: The real data mentioned in the embodiments of this application can be understood as data obtained based on real objects. For example, if you take a picture of a person in front of you, the resulting picture can be considered as real data.

[0043] Ground Truth Labeling: Labels for labeled data, rendered based on synthetic data.

[0044] Pseudo-labeling refers to labels that are not actually labeled by humans. For example, they may be the result predicted by another trained model, but are used as supervision signals during training.

[0045] Orthographic projection transformation: A cuboid is used to frame the scene, and the scene is projected onto the front of this cuboid. This projection does not have perspective shrinkage (objects that are farther away appear smaller on the image plane) because it ensures that parallel lines remain parallel after the transformation, thus keeping the relative distances between objects constant. Simply put, orthographic projection transformation ignores the scaling of objects at different distances, projecting objects onto a cross-section (such as a display screen) in their original proportions. Cameras that achieve this effect are called orthographic projection cameras, or simply orthographic cameras.

[0046] Perspective projection transformation: Similar to orthographic projection, it projects a spatial object (referring to a perspective pyramid with its projection center as the vertex) onto a two-dimensional image plane. However, it has a perspective contraction effect: distant objects appear smaller on the image plane than objects of the same size that are closer. Unlike orthographic projection, perspective projection does not maintain the relative magnitude of distances and angles, so parallel lines are not necessarily projected as parallel. In other words, perspective projection transformation allows an object to appear larger when close to the player and smaller when far away; a camera that achieves this effect is called a perspective camera. Perspective cameras are often used to develop 3D games; their working principle is to scale the projection ratio (i.e., the size of the cross-section) according to the distance between the camera and the object. Perspective projection is quite similar to the principle by which the human eye or camera lens produces images of the 3D world. The essential difference between the two projection methods is that the distance between the projection center and the projection plane in perspective projection is finite, while the distance between the projection center and the projection plane in parallel projection is infinite.

[0047] Depth image, also known as range image, refers to the distance (depth) from the image acquisition device to each point in the scene, expressed as pixel values. It directly reflects the geometry of the visible surface of the scene. Depth images can be converted into point cloud data through coordinate transformation, and point cloud data with regularity and necessary information can also be converted back into depth image data. Each pixel in a depth image represents the distance from the closest point on the camera plane to the depth sensor's field of view for that specific object at that coordinate location.

[0048] Voxel: Short for volume pixel, a solid containing voxels can be represented through stereo rendering or by extracting polygonal isosurfaces from a given threshold contour. As its name suggests, it is the smallest unit of digital data segmentation in three-dimensional space. Voxels are used in fields such as 3D imaging, scientific data, and medical imaging. Conceptually similar to the smallest unit in two-dimensional space—the pixel—which is used in image data of two-dimensional computer images. Some true 3D displays use voxels to describe their resolution; for example, a display that can show 512×512×512 voxels.

[0049] Feature Volume: Similar to feature vector, each cell (i.e., voxel) in three-dimensional space has its own corresponding feature vector. In this embodiment, the feature vector is obtained through a deep neural network.

[0050] Multilayer Perceptron (MLP): A feedforward artificial neural network that maps a set of input vectors to a set of output vectors.

[0051] Directed distance field: A signed distance function (or directed distance function) used to measure the distance between a set Ω in space and the boundary of Ω. The sign of the function depends on whether x is in Ω. The function has a positive value at a point x inside Ω, its value decreases as x approaches the boundary of Ω (where the signed distance function is zero), and it takes a negative value outside Ω.

[0052] Truncated Signed Distance Function (TSDF): A truncated directed distance field. Compared to a standard directed distance field, this function has a maximum and a minimum value. When the function value exceeds or falls below a certain value, the function value will be replaced.

[0053] 3D geometric configuration: also known as 3D human body geometry or 3D human body mesh. A 3D human body model without textures, containing only the geometric information of the human body surface, with the geometric topological information of the human body surface represented by points and triangular meshes.

[0054] Please refer to Figure 1 The diagram illustrates an implementation environment for a solution provided in one embodiment of this application. This implementation environment may include: a terminal device 10 and a server 20.

[0055] Terminal device 10 includes, but is not limited to, mobile phones, tablets, smart voice interaction devices, game consoles, wearable devices, multimedia playback devices, PCs (Personal Computers), in-vehicle terminals, smart home appliances, and other electronic devices. The client for the target application can be installed on terminal device 10.

[0056] In this embodiment, the target application can be any application capable of providing video streaming content services. Typically, the application is a video application. Of course, other types of applications besides video applications can also provide streaming content services. For example, news applications, social applications, interactive entertainment applications, browser applications, shopping applications, content sharing applications, virtual reality (VR) applications, augmented reality (AR) applications, etc., are not limited in this embodiment. In addition, the videos pushed by different applications will be different, and the corresponding functions will also be different. These can be pre-configured according to actual needs, and are not limited in this embodiment. Optionally, the terminal device 10 runs a client of the above-mentioned application. In some embodiments, the above-mentioned streaming content service covers many vertical content such as variety shows, movies, news, finance, sports, entertainment, and games, and users can enjoy a variety of content services such as articles, pictures, short videos, live broadcasts, special topics, and columns through the above-mentioned streaming content service.

[0057] Server 20 is used to provide backend services for the client of the target application in terminal device 10. For example, server 20 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms, but it is not limited to these.

[0058] Terminal device 10 and server 20 can communicate with each other via a network. This network can be a wired network or a wireless network.

[0059] The method provided in this application embodiment can be executed by a computer device in each step. The computer device can be any electronic device capable of data storage and processing. For example, the computer device can be... Figure 1 Server 20 in the middle can be Figure 1The terminal device 10 can also be another device other than the terminal device 10 and the server 20.

[0060] Please refer to Figure 2 The diagram illustrates an application scenario of a three-dimensional reconstruction model provided in one embodiment of this application.

[0061] like Figure 2 As shown, real human images are captured using a camera or webcam, and the model trained using this method quickly and efficiently reconstructs high-fidelity 3D human geometry with clothing. The reconstructed 3D human geometry has the following potential applications: one-click generation of metaverse digital humans, virtual clothing reconstruction, one-click dress-up, and film and animation production, etc.

[0062] In the production of 3D films and animations, modelers need to construct 3D human body models from scratch, which is a lengthy and costly process. According to the technical solution provided in this application, a simple photograph of a human body is all that's needed to quickly obtain 3D human geometry. Modelers then only need to make minor adjustments to this geometry to obtain a high-quality 3D human body model, significantly shortening the production cycle. With the development of related technologies, the graphics in many large-scale 3D games are gradually approaching those of the real world. This method can reconstruct high-fidelity 3D human geometry, increasing the feasibility of projecting real-life users into the virtual game world.

[0063] In this embodiment, a 3D human body mesh is generated using real data. Compared to synthetic data, synthetic data is insufficient in representing the details of the human body. For example, in some embodiments, the target object is a woman wearing a skirt. If synthetic data is used to simulate the image of this woman, it will be insufficient in representing details such as the folds of the skirt. This is because there are still differences in detail between the simulated human body and the real human body. Therefore, in this embodiment, real data captured by a camera is used to obtain the 3D human body mesh. Considering that real data can encompass objects that cannot be represented by synthetic data, and that real data can more accurately represent the target object, the network model trained with real data (referring to the 3D reconstruction model in this embodiment) has better generalization and accuracy. The 3D human body mesh obtained by the network model is certainly superior in detail compared to other network models trained with synthetic data.

[0064] Please refer to Figure 3 The diagram illustrates a flowchart of a training method for a 3D reconstruction model provided in one embodiment of this application. The execution entity for each step of this method can be... Figure 1 In the implementation environment of the scheme shown, the terminal device 10 can be either the client of the target application or the execution subject of each step. Figure 1 Server 20 in the implementation environment of the illustrated scheme. In the following method embodiments, for ease of description, only the execution subject of each step is described as a "computer device". The method may include at least one of the following steps (320-380):

[0065] Step 320: Obtain a first depth image of the target object. The first depth image is a true depth image obtained when the target object is photographed, or a depth image obtained based on a true color image of the target object.

[0066] The target object is any object in the real world, including but not limited to human bodies, faces, animals, scenery, etc.

[0067] In some embodiments, the first depth image is a true depth image obtained when the target object is photographed. That is, by using a depth acquisition device to directly photograph the target object, a depth image of the target object can be obtained, which is an image showing the distances of various parts of the target object to a set distance. The depth acquisition device is a device that can acquire the depth of the target object. Optionally, the depth acquisition device is a depth collector that can directly acquire the depth information of the target object. Optionally, the depth acquisition device is a camera plus a depth sensor, that is, the depth image is directly generated by the camera simultaneously capturing the image based on the depth information acquired by the depth sensor.

[0068] In some embodiments, the depth image is obtained based on a true-color image of the target object captured by a camera. That is, a true-color image of the target object is first acquired using a camera device, and then the depth image is obtained based on the true-color image.

[0069] In some embodiments, step 320 includes at least one of the following steps (322-324):

[0070] Step 322: Obtain a true color image of the target object.

[0071] Optionally, a true color image is an RGB image, which contains the position and color information of each pixel.

[0072] Step 324: Generate the first depth image of the target object based on the real color image using the depth map prediction model.

[0073] A depth map prediction model is a model used to predict depth images from color images. Optionally, the depth map prediction model is a neural network model. In some embodiments, the depth map prediction model is a pixel2pixel algorithm model. Obtaining a depth image based on the depth map prediction model makes the depth image more valuable for reference and can be used for the self-supervised training method of this application.

[0074] Step 340: Based on the first depth image, the three-dimensional reconstruction information of the target object is obtained through the three-dimensional reconstruction model. The three-dimensional reconstruction information is used to determine the three-dimensional geometric configuration of the target object in three-dimensional space.

[0075] The three-dimensional reconstruction information of the target object corresponding to the first depth image is obtained through the three-dimensional reconstruction model in the embodiments of this application. Here, the three-dimensional reconstruction information refers to the information in three-dimensional space, which is used to determine the three-dimensional geometric configuration of the target object in three-dimensional space.

[0076] Optionally, the 3D reconstruction information includes the coordinates of points in 3D space and the distance from the points to the object surface. Based on this distance information, the 3D geometric configuration of the target object in 3D space can be further determined.

[0077] In some embodiments, the target object has multiple first depth images, and each first depth image corresponds one-to-one with multiple shooting angles. Optionally, the target object is photographed from multiple shooting angles such as the front, side, and side views to obtain multiple depth images. In some embodiments, the depth image of the front view of the target object is used as input to the 3D reconstruction model. Optionally, if the target object is a human body, the image of the front view of the human body is used as input to obtain the 3D reconstruction information of the human body. This application does not limit the number of input depth images, the shooting angles of the target object corresponding to the input depth images, or the number of iterations for each model training.

[0078] In some embodiments, multiple first depth images are input into the same 3D reconstruction model, and the 3D reconstruction model outputs 3D reconstruction information of the target object based on the first depth images. Optionally, depth images from different shooting angles are used as input. Optionally, if the target object is a human body, depth images corresponding to the front, side, and back of the human body are used as input to the 3D reconstruction model. Optionally, multiple depth images from different angles are input at once, and the 3D reconstruction model outputs 3D reconstruction information of the target object based on the multiple depth images. Optionally, multiple depth images from different angles are sequentially input into the 3D reconstruction model, and the 3D reconstruction information of the target object is output.

[0079] In some embodiments, multiple first depth images are input into multiple 3D reconstruction models, each 3D reconstruction model processing a first depth image corresponding to a shooting angle. The multiple 3D reconstruction models output 3D reconstruction information corresponding to each of the multiple first depth images. Based on the 3D reconstruction information corresponding to the multiple first depth images, the 3D reconstruction information of the target object is obtained by fusing them. Optionally, depth images from different angles are input into different 3D reconstruction models, and the three output 3D reconstruction information are automatically or manually adjusted to form the final fused 3D reconstruction model.

[0080] In some embodiments, the surface of the target object is extracted based on the 3D reconstruction information using an isosurface extraction algorithm, thereby determining the 3D geometric configuration.

[0081] The technical solution provided in this application obtains depth images of the target object from different angles and then obtains fused 3D reconstruction information through a 3D reconstruction model. In other words, it can collect information about the target object from different angles to generate fused 3D reconstruction information. To a certain extent, this makes the 3D reconstruction information closer to the target object itself, that is, the output of the model is closer to the prototype, and the model has higher accuracy.

[0082] Step 360: Generate a second depth image of the target object based on the 3D reconstruction information.

[0083] In some embodiments, depth information of the target object in a two-dimensional coordinate system is obtained based on the three-dimensional reconstruction information; and a second depth image of the target object is generated based on the depth information of the target object in the two-dimensional coordinate system.

[0084] In some embodiments, a depth image of the target object is generated inversely based on 3D reconstruction information in three-dimensional space. In other embodiments, a second depth image of the target object is generated using an image rendering model based on the 3D reconstruction information. In some embodiments, the image rendering model is a renderer, which can be used to generate a depth image of the target object inversely based on the 3D reconstruction information in three-dimensional space. That is, based on the 3D reconstruction information, the depth information of the target object in a two-dimensional coordinate system is obtained, where the depth information includes the position coordinate data of a pixel and the depth data corresponding to that point. Based on this depth information, the depth image of the target object corresponding to the 3D reconstruction information is generated by the renderer, resulting in more accurate results and relatively lower costs. The technical solution provided in this application, which generates a depth image based on 3D reconstruction information, can improve the accuracy of the generated depth image.

[0085] In some embodiments, a differentiable rendering (DR) algorithm is used to generate a depth image of the target object based on the 3D reconstruction information. Specific algorithms will not be described in detail in this application.

[0086] Step 380: Train the 3D reconstruction model based on the first depth image and the second depth image.

[0087] After obtaining the first and second depth images of the target object, the 3D reconstruction model is trained. In some embodiments, the first depth image of the target object can be regarded as a pseudo-label, and the difference between the output of the 3D reconstruction model and the pseudo-label is judged to train the 3D reconstruction model.

[0088] In some embodiments, step 380 includes at least one of the following steps (382-384):

[0089] Step 382: Determine the training loss of the 3D reconstruction model based on the difference between the first depth image and the second depth image.

[0090] In some embodiments, the training loss is determined based on the difference between the first depth image and the second depth image. In some embodiments, regularization is used to constrain the first depth image and the second depth image. Using regularization to constrain the first depth image and the second depth image can effectively improve generalization ability while preventing overfitting. In some embodiments, the training loss of the model is determined based on the mean squared error or the absolute value of the difference between the first depth image and the second depth image.

[0091] In some embodiments, L1-norm regularization or L2-norm regularization is used. L1 and L2 regularization can be viewed as penalty terms in the loss function. The penalty refers to imposing restrictions on certain parameters in the loss function. Specific L1-norm or L2-norm regularization methods will not be elaborated upon here. Based on the technical solutions provided in the embodiments of this application, the L2-norm regularization method slightly outperforms the L1-norm regularization method in model training.

[0092] This application does not limit the function used to determine the training loss, nor does it limit the method used to adjust the model parameters based on the loss.

[0093] In some embodiments, the training loss of the model is determined based on a loss function.

[0094] Step 384: Adjust the parameters of the 3D reconstruction model based on the training loss.

[0095] In some embodiments, gradient descent is used to adjust the model parameters in reverse based on the model's training loss. In other embodiments, the backpropagation algorithm is used to adjust the parameters of the 3D reconstruction model.

[0096] In some embodiments, the trained 3D reconstruction model is used for at least one of the following: generating a 3D geometric configuration of a human body in a 3D virtual scene based on a real color image of a human body; generating a 3D geometric configuration of a digital human in a 3D virtual scene based on a real color image of a digital human; generating a 3D geometric configuration of clothing in a 3D virtual scene based on a real color image of clothing; and generating a 3D geometric configuration of a face in a 3D virtual scene based on a real color image of a face.

[0097] In some embodiments, such as in human body simulation games, it is necessary to project real human bodies into a virtual environment. Therefore, it is possible to first take pictures of real human bodies. After taking pictures of real human bodies and obtaining depth images of the human bodies, a three-dimensional geometric configuration of the human body can be generated based on the depth images through a three-dimensional reconstruction model. That is, a three-dimensional geometric configuration of the human body corresponding to the real human body is generated in the virtual environment, which can improve the user's gaming experience.

[0098] In some embodiments, such as a metaverse digital human scenario, multiple digital humans need to be generated. Similarly, the geometric configuration of the digital human can be generated in the virtual scene through a 3D reconstruction model based on the real image of the digital human.

[0099] In some embodiments, such as changing clothes, the three-dimensional geometric configuration of the clothes can be generated by a three-dimensional reconstruction model based on the real color image of the clothes. Different clothes correspond to different three-dimensional geometric configurations, thus enabling the changing of clothes.

[0100] In some embodiments, such as AI face-swapping technology, a real human face can be used to obtain the three-dimensional geometric configuration of the face through a three-dimensional reconstruction model, and then applied to the place where face swapping is needed.

[0101] The 3D reconstruction model trained in this application embodiment can be applied to many scenarios. It can generate the corresponding 3D geometric configuration of a real target object through the 3D reconstruction model. It can be widely used in various scenarios such as games and animation production. It can not only improve the accuracy of the generated geometric configuration, but also improve the user experience.

[0102] Figure 4This is a schematic diagram illustrating the training results provided in one embodiment of this application. Figure 400 shows a comparison between the results obtained by this method and those obtained by existing methods. The first column represents real-world images of the human body; the second and third columns represent the 3D human geometry reconstructed by this method, with the second column showing a frontal view and the third a side view; the fourth and fifth columns represent the 3D human geometry reconstructed by existing methods, with the fourth column showing a frontal view and the fifth a side view. The results show that the 3D human geometry reconstructed by this method has better completeness and generalization, and is closer to the real world.

[0103] Figure 5 This is a schematic diagram of the training results provided by another embodiment of this application. Figure 500 shows the results obtained using this method. The first column represents a real-world image of the human body, and the second, third, and fourth columns represent the reconstructed 3D human geometry, with the second column showing a frontal view and the latter two columns showing different side views. The results show that the reconstruction effect of this method is sufficiently ideal under different viewpoints.

[0104] The technical solution provided in this application firstly trains the model parameters by obtaining a real depth image of the target object taken by shooting, or a depth image obtained from a real color image of the target object, and a depth image obtained through a 3D reconstruction model. This achieves model training in a self-supervised manner, so that the model training does not need to rely on other data, but can be completed using only the data of the target object, thus reducing the training cost of the model.

[0105] Secondly, by using real data of the target object to train the 3D reconstruction model, the texture details of the target object can be more clearly depicted. At the same time, real data covers a wide range, and using real data to train the model can make the trained model more generalizable. In addition, because real data represents the details of the target object more fully, the trained model has high fidelity.

[0106] Please refer to Figure 6 The diagram illustrates a flowchart of a training method for a 3D reconstruction model provided in another embodiment of this application. The entity executing each step of this method may be... Figure 1 In the implementation environment of the scheme shown, the terminal device 10 can be either the client of the target application or the execution subject of each step. Figure 1 Server 20 in the implementation environment of the illustrated scheme. In the following method embodiments, for ease of description, only the execution subject of each step is described as a "computer device". The method may include at least one of the following steps (320-380):

[0107] Step 320: Obtain a first depth image of the target object. The first depth image is a true depth image obtained when the target object is photographed, or a depth image obtained based on a true color image of the target object.

[0108] In some embodiments, before step 320, the method further includes pre-training the 3D reconstruction model using pre-training samples to obtain a pre-trained 3D reconstruction model. The pre-training samples include sample images and target 3D reconstruction information corresponding to the sample images. The pre-trained 3D reconstruction model is used to obtain the 3D reconstruction information of the target object based on the first depth image, and to train the pre-trained 3D reconstruction model based on the first depth image and the second depth image.

[0109] In some embodiments, the sample image is a depth image or a depth image generated based on a color image, wherein the sample image is synthetic data.

[0110] In some embodiments, the depth image input to the 3D reconstruction model is a depth image of the front of the target object, thus potentially missing data from the back side. Therefore, a pre-trained 3D reconstruction model is considered to compensate for data gaps caused by a single depth image when generating 3D reconstruction information corresponding to the depth image. In some embodiments, the sample image is synthetic data, and the target 3D reconstruction information is the ground truth label of the synthetic data. Based on the 3D reconstruction information obtained from the synthetic data through the 3D reconstruction model, and based on the ground truth label of the synthetic data, the training loss of the 3D reconstruction model is determined, thereby adjusting the parameters of the 3D reconstruction model.

[0111] In some embodiments, inputting depth images from multiple shooting angles at once can avoid the loss of backplane information, thus eliminating the need for pre-training the 3D reconstruction model. In other embodiments, inputting depth images from only one shooting angle requires pre-training the model so that it can predict the missing data based on a single image.

[0112] Step 342: Based on the first depth image, the feature voxels of the target object are obtained through the 3D reconstruction model. The feature voxels include the feature information of the voxels corresponding to the target object.

[0113] In some embodiments, a feature voxel is a three-dimensional array that includes feature information of the voxel corresponding to the target object. In some embodiments, the size of the feature voxel array is determined by the size information of the depth image. For example, if the depth image size is 128*128, the corresponding three-dimensional feature voxel array can be a 128*128*a array, where 'a' is a positive number representing the data of the voxel in a dimension other than its length and width. Optionally, this other dimension is the height data of the voxel, and optionally, it corresponds to the depth information. In some embodiments, the size of the feature voxel can be adjusted and is not uniquely fixed. The feature voxel includes feature information of the voxel corresponding to the target object, where the feature information can be a feature vector. The feature voxel is an array used to characterize voxels, and a voxel is a three-dimensional geometric shape drawn based on the depth image. The feature voxel can be used to characterize the distance relationship between voxels, that is, to characterize the voxel itself.

[0114] The technical solution provided in this application obtains the feature voxels corresponding to the target object based on the depth image. These feature voxels can better characterize the voxels of the target object, that is, further characterize the target object, making it easier to determine the feature vectors of points in space based on the feature voxels, and then simulate the three-dimensional geometric configuration.

[0115] Step 344: Obtain the three-dimensional reconstruction information of the target object based on the feature voxels using the three-dimensional reconstruction model.

[0116] In some embodiments, the 3D reconstruction information includes a directed distance field used to determine the surface of a target object in the 3D space. In some embodiments, the distance from a point in space to the surface, and whether the point is inside or outside the surface, constitute the directed distance field. Therefore, using this directed distance field, the surface of the target object in the 3D space can be determined. In some embodiments, for cost and necessity considerations, the maximum and minimum values ​​of the function in the directed distance field are limited, that is, a directed truncated distance field is determined. Specifically, when the distance from a point in space to the surface is greater than or less than a threshold, the value of that point in the directed distance field is fixed. In some embodiments, the value of the directed truncated distance field is set to -0.8 to 0.8. This application does not limit the value of the directed truncated distance field.

[0117] In some embodiments, step 344 may further include at least one of the following steps (344-2 to 344-6):

[0118] Step 344-2: Sample the points in the three-dimensional space to obtain multiple sampling points.

[0119] In some embodiments, the sampling point is random and can be any point in space.

[0120] In some embodiments, the x-axis and y-axis in three-dimensional space are normalized according to the size of the feature voxels, that is, the space is normalized to the length and width range of the feature voxels, and points in this space are sampled.

[0121] Step 344-4: Use interpolation to determine the feature information corresponding to multiple sampling points from the feature voxels.

[0122] In some embodiments, feature information of the sampling point in space is determined based on feature voxels; in some embodiments, the feature information is a feature vector.

[0123] In some embodiments, linear interpolation is performed on the feature information of voxels in different directions to determine the feature information corresponding to multiple sampling points. In some embodiments, bilinear sampling is performed on the feature voxels along the x-axis and y-axis to obtain the feature vector of the sampling point. Optionally, the feature voxel is 128*128*256 three-dimensional data, that is, the feature voxel can be considered as 128*128 256-dimensional feature vectors. 128*128 can correspond to the position of the sampling point on the x-axis and y-axis in space. The two closest feature voxels (two 256-dimensional feature vectors) of the sampling point are determined on the x-axis, and the two closest feature voxels (two 256-dimensional feature vectors) of the sampling point are determined on the y-axis. The feature vector of the sampling point (256-dimensional) is determined according to the interpolation method. Optionally, the feature vector of the sampling point is increased to 257 dimensions based on the position information of the sampling point in the z-axis direction. That is, sampling is performed in the space enclosed by the x-axis and y-axis, and the resulting feature vector of the sampling point is 257-dimensional feature information including z-axis information. This application does not impose any limitation on the dimension of the feature vector. The dimension can be set manually or automatically adjusted by the model according to the actual situation. This application does not impose any limitation on this.

[0124] The interpolation method described in the embodiments of this application may be a spatial bilinear interpolation method. This application does not limit the specific interpolation method. Any method that determines the feature information of the sampling point in space based on the feature voxels is included in the protection scope of this application.

[0125] In some embodiments, the length of the third dimension of the voxel is set to 256. Optionally, the dimension of the feature vector of the sampling point is 257. In addition to the x and y coordinates of the sampling point itself, the z-axis coordinate is added to the 256-dimensional feature vector obtained based on the bilinear interpolation method, that is, the feature vector of the sampling point is 257-dimensional.

[0126] Step 344-6: Based on the feature information corresponding to multiple sampling points, the directional distance field of the target object in three-dimensional space is obtained through the three-dimensional reconstruction model.

[0127] In some embodiments, feature vectors of multiple sampling points are obtained, which in turn can yield a directed distance field in space, representing the distance from a point in space to the surface of a target object.

[0128] In some embodiments, the three-dimensional reconstruction model includes: a feature voxel extraction sub-model and a three-dimensional reconstruction sub-model; the feature voxel extraction sub-model is used to obtain feature voxels of the target object in three-dimensional space based on the first depth image; the three-dimensional reconstruction sub-model is used to obtain three-dimensional reconstruction information of the target object based on the feature voxels.

[0129] In some embodiments, the feature voxel extraction sub-model is an encoding model in a convolutional neural network, which can be used to extract feature voxels corresponding to the depth image. In some embodiments, the 3D reconstruction sub-model is a multilayer perceptron, which optionally includes fully connected layers that can map the feature vectors of sampling points in space obtained based on feature voxels to a directed distance field in space, i.e., 3D reconstruction information.

[0130] The technical solution provided in this application extracts feature voxels through a feature voxel extraction sub-model and obtains three-dimensional reconstruction information through a three-dimensional reconstruction sub-model. In other words, through these two models, the conversion from two-dimensional depth images to three-dimensional directed distance fields can be realized, which is beneficial to improving the training accuracy of the model.

[0131] In the embodiments of this application, the implicit field function can be considered to include a voxel extraction sub-model and a 3D reconstruction sub-model. In some embodiments, the implicit field function can be considered to include a voxel extraction sub-model and a multilayer perceptron.

[0132] Step 360: Generate a second depth image of the target object based on the 3D reconstruction information.

[0133] Step 380: Train the 3D reconstruction model based on the first depth image and the second depth image.

[0134] Figure 7 A flowchart illustrating a training method for a 3D reconstruction model provided in one embodiment of this application is shown. Figure 700 shows the input of a single human image, the prediction of the corresponding human depth image, where darker pixels indicate closer proximity to the camera, and lighter pixels indicate greater distance. An implicit field function is used to obtain the directed truncated distance field (containing 3D human geometry) across the entire space. Differentiable rendering based on orthogonal projection is used to render the depth image corresponding to the 3D human geometry. Self-supervised regularization constraints are applied to the predicted and rendered depth images to optimize the implicit field function and update the model parameters.

[0135] In addition, the technical solutions provided in this application are based on data transformation under orthographic projection. Compared with data transformation under perspective projection, the technical solutions provided in this application have better training effects on the model. However, this application does not limit the data transformation method. The data transformation methods that can be used in the technical solutions provided in this application include, but are not limited to, orthographic projection methods and perspective projection methods.

[0136] The technical solution provided in this application sample points in space and obtains the feature vectors corresponding to the sampled points based on feature voxels. That is, the two-dimensional planar image is converted into three-dimensional point data information through feature voxels. The three-dimensional point data information can be used to outline the three-dimensional geometric configuration of the target object.

[0137] Furthermore, the 3D reconstruction model is divided into a feature voxel extraction sub-model and a 3D reconstruction sub-model. This hierarchical design allows for timely and accurate identification of errors when they occur. Simultaneously, both the feature voxel extraction and 3D reconstruction sub-models are trained concurrently, further improving the final training accuracy.

[0138] Please refer to Figure 8 The diagram illustrates a flowchart of a training method for a three-dimensional reconstruction model according to an embodiment of this application. The method includes at least one of the following steps (S1-S5) (not shown in the diagram):

[0139] Step S1: First, the human body image is input into the deep learning algorithm model to obtain the human body depth image and feature voxels. The process is as follows:

[0140] M(I)→D, M1(D)→F V ;

[0141] Where M() represents the general pixel2pixel algorithm model, which is a deep learning algorithm model that generates another image from an input image. M1() represents the feature voxel extraction model, which obtains feature voxels from an input depth image. I represents the input human image, D represents the human depth image estimated by M(), and F... V This represents the feature voxels predicted by M1(). A typical image consists of three RGB channels and can be represented as a three-dimensional array, where the first and second dimensions represent the length and width of the image, respectively, and the third dimension represents the RGB values.

[0142] Step S2: Using spatial sampling interpolation, the feature vector of the entire space is obtained from the feature voxels obtained in Step 1. Each spatial sampling point will have a corresponding feature vector. Then, a multilayer perceptron is used to predict the directed cutoff distance field corresponding to each sampling point in the space, thus obtaining the directed cutoff distance field of the entire space. The process is as follows:

[0143] Interpolation(F V P i → F i ,M2(F s → F;

[0144] Where, P i F represents the sampling point in space. i F represents the feature vector corresponding to each sampling point in the space. S The feature vector of the entire space, M2() represents the multilayer perceptron, F represents the directed truncated distance field of the entire space, and Interpolation represents the spatial bilinear interpolation method.

[0145] Step S3: Based on the directed truncated distance field obtained in step S2, the human depth image D is obtained using a differentiable rendering algorithm based on orthogonal projection. R :

[0146] DR(F)→D R ;

[0147] Here, DR() represents the differentiable rendering depth image algorithm under orthogonal projection, D R This represents the rendered human depth image, and F represents the directed cutoff distance field across the entire space.

[0148] Step S4: Regularize the human depth image D predicted in step 1 and the human depth image D rendered in step 3. R This leads to the optimization of the parameter P to be optimized in the feature voxel extraction sub-model M1 in step 1 and the multilayer perceptron M2 in step 2. M1 and P M2 :

[0149] P M1 (L1(D,D R )),P M2 (L1(D,D R ));

[0150] Where L1 represents the L1 norm regularization constraint, and L2 norm or other norms can also be used for constraint, P M1 and P M2These represent the parameters to be optimized in the feature voxel extraction sub-model M1 and the multilayer perceptron M2, respectively. Through this step, our method can train the network model using real images.

[0151] Step S5: Using the optimized feature voxel extraction sub-model and multilayer perceptron, the directed truncated distance field is obtained, and the Marching Cube algorithm (isosurface extraction algorithm) is used to obtain the three-dimensional geometry of the human body.

[0152] MC(F)→S;

[0153] Here, S represents the reconstruction of the 3D human geometry from the input human image, and MC represents the Marching Cube (isosurface extraction) algorithm. The 3D human geometry is contained within the directed truncated distance field of the entire space, and the MC algorithm is needed to extract the 3D human geometry separately.

[0154] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0155] Please refer to Figure 9 This diagram illustrates a block diagram of a training apparatus for a three-dimensional reconstruction model according to an embodiment of this application. The apparatus has the functionality to implement the method example described above; this functionality can be implemented in hardware or by hardware executing corresponding software. The apparatus can be the computer device described above, or it can be located within a computer device. Figure 9 As shown, the device 900 may include: an image acquisition module 910, an information generation module 920, an image generation module 930, and a model training module 940.

[0156] The image acquisition module 910 is used to acquire a first depth image of the target object. The first depth image is a real depth image obtained when the target object is photographed, or a depth image obtained based on a real color image obtained by photographing the target object.

[0157] The information generation module 920 is used to obtain the three-dimensional reconstruction information of the target object based on the first depth image through the three-dimensional reconstruction model. The three-dimensional reconstruction information is used to determine the three-dimensional geometric configuration of the target object in three-dimensional space.

[0158] The image generation module 930 is used to generate a second depth image of the target object based on the three-dimensional reconstruction information.

[0159] The model training module 940 is used to train the three-dimensional reconstruction model based on the first depth image and the second depth image.

[0160] In some embodiments, such as Figure 10 As shown, the information generation module 920 further includes a feature voxel acquisition unit 922 and an information generation unit 924.

[0161] The feature voxel acquisition unit 922 is used to obtain the feature voxels of the target object based on the first depth image through the three-dimensional reconstruction model. The feature voxels include feature information of the voxels corresponding to the target object.

[0162] The information generation unit 924 is used to obtain the three-dimensional reconstruction information of the target object based on the feature voxels through the three-dimensional reconstruction model.

[0163] In some embodiments, the three-dimensional reconstruction information includes a directed distance field used to determine the surface of the target object in the three-dimensional space.

[0164] In some embodiments, the information generation unit 924 is used to sample points in the three-dimensional space to obtain multiple sampling points.

[0165] The information generation unit 924 is used to determine the feature information corresponding to the plurality of sampling points from the feature voxels by interpolation.

[0166] The information generation unit 924 is used to obtain the directed distance field of the target object in the three-dimensional space based on the feature information corresponding to the multiple sampling points through the three-dimensional reconstruction model.

[0167] In some embodiments, the three-dimensional reconstruction model includes: a feature voxel extraction sub-model and a three-dimensional reconstruction sub-model.

[0168] The feature voxel extraction sub-model is used to obtain the feature voxels of the target object in the three-dimensional space based on the first depth image.

[0169] The 3D reconstruction sub-model is used to obtain the 3D reconstruction information of the target object based on the feature voxels.

[0170] In some embodiments, the image generation module 930 is used to acquire a true color image of the target object.

[0171] The image generation module 930 is used to generate a first depth image of the target object based on the real color image using a depth map prediction model.

[0172] In some embodiments, such as Figure 10 As shown, the device also includes a pre-training module 950.

[0173] The pre-training module 950 is used to pre-train the 3D reconstruction model using pre-training samples to obtain a pre-trained 3D reconstruction model. The pre-training samples include sample images and target 3D reconstruction information corresponding to the sample images. The pre-trained 3D reconstruction model is used to obtain the 3D reconstruction information of the target object based on the first depth image, and to train the pre-trained 3D reconstruction model based on the first depth image and the second depth image.

[0174] In some embodiments, the target object has multiple first depth images, and the multiple first depth images correspond one-to-one with multiple shooting angles.

[0175] The information generation module 920 is used to input the multiple first depth images into the same three-dimensional reconstruction model, and output the three-dimensional reconstruction information of the target object based on the multiple first depth images; or, the multiple first depth images are input into multiple three-dimensional reconstruction models respectively, each three-dimensional reconstruction model is used to process the first depth image corresponding to a shooting angle, and output the three-dimensional reconstruction information corresponding to the multiple first depth images respectively through the multiple three-dimensional reconstruction models, and the three-dimensional reconstruction information of the target object is obtained by fusing the three-dimensional reconstruction information corresponding to the multiple first depth images respectively.

[0176] In some embodiments, the image generation module 930 is used to obtain the depth information of the target object in a two-dimensional coordinate system based on the three-dimensional reconstruction information.

[0177] The image generation module 930 is further configured to generate a second depth image of the target object based on the depth information of the target object in a two-dimensional coordinate system.

[0178] In some embodiments, the model training module 940 is used to determine the training loss of the 3D reconstruction model based on the difference between the first depth image and the second depth image.

[0179] The model training module 940 is used to adjust the parameters of the three-dimensional reconstruction model according to the training loss.

[0180] In some embodiments, the trained 3D reconstruction model is used for at least one of the following: generating a 3D geometric configuration of the human body in a 3D virtual scene based on a real color image of the human body; generating a 3D geometric configuration of the digital human in a 3D virtual scene based on a real color image of the digital human; generating a 3D geometric configuration of clothing in a 3D virtual scene based on a real color image of clothing; and generating a 3D geometric configuration of a face in a 3D virtual scene based on a real color image of a face.

[0181] It should be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0182] Please refer to Figure 11 The diagram shows a structural block diagram of a computer device 2100 provided in one embodiment of this application.

[0183] Typically, computer device 2100 includes a processor 2101 and a memory 2102.

[0184] Processor 2101 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 2101 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), and PLA (Programmable Logic Array). Processor 2101 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 2101 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 2101 may also include an AI processor for handling computational operations related to machine learning.

[0185] The memory 2102 may include one or more computer-readable storage media, which may be non-transitory. The memory 2102 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 2102 are used to store a computer program configured to be executed by one or more processors to implement the above-described method for training the 3D reconstruction model.

[0186] Those skilled in the art will understand that Figure 11 The structure shown does not constitute a limitation on computer device 2100 and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0187] In an exemplary embodiment, a computer-readable storage medium is also provided, wherein a computer program is stored therein, which, when executed by a processor, implements a method for training a three-dimensional reconstruction model.

[0188] Optionally, the computer-readable storage medium may include: ROM (Read-Only Memory), RAM (Random Access Memory), SSD (Solid State Drives), or optical disc, etc. The random access memory may include ReRAM (Resistance Random Access Memory) and DRAM (Dynamic Random Access Memory).

[0189] In an exemplary embodiment, a computer program product is also provided, the computer program product including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, causing the computer device to perform the above-described training method for the three-dimensional reconstruction model.

[0190] It should be understood that "multiple" as used herein refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. Furthermore, the step numbers described herein are merely illustrative of one possible execution order. In some other embodiments, the steps may not be executed in numerical order, such as two steps with different numbers being executed simultaneously, or two steps with different numbers being executed in the reverse order of the illustration. This application does not limit this.

[0191] The above description is merely an exemplary embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A training method for a three-dimensional reconstruction model, characterized in that, The 3D reconstruction model includes a feature voxel extraction sub-model and a multilayer perceptron, and the method includes: A first depth image of the target object is obtained, wherein the first depth image is a true depth image obtained when the target object is photographed, or a depth image obtained based on a true color image of the target object. The feature voxel extraction sub-model obtains the feature voxels of the target object based on the first depth image, and the feature voxels include the feature information of the voxels corresponding to the target object; The multilayer perceptron obtains the directed distance field of the target object in three-dimensional space based on the feature voxels, and the directed distance field is used to determine the three-dimensional geometric configuration of the target object in three-dimensional space. Based on the directed distance field, a second depth image of the target object is generated; Based on the first depth image and the second depth image, the three-dimensional reconstruction model is trained. The trained three-dimensional reconstruction model is used to determine the three-dimensional geometric configuration of the object based on the input real color image of the object. The object includes a human body, a digital human, clothing, and a human face. The target object has multiple first depth images, each corresponding to a multiple shooting angle. The multiple first depth images are respectively input into multiple three-dimensional reconstruction models to obtain the directed distance fields corresponding to the multiple first depth images. Each three-dimensional reconstruction model is used to process a first depth image corresponding to a shooting angle. The directed distance fields corresponding to the multiple first depth images are fused to obtain the directed distance field of the target object.

2. The method according to claim 1, characterized in that, The step of obtaining the directed distance field of the target object in the three-dimensional space using the multilayer perceptron based on the feature voxels includes: Multiple sampling points are obtained by sampling points in the three-dimensional space. The feature information corresponding to the plurality of sampling points is determined from the feature voxels by interpolation. Based on the feature information corresponding to the multiple sampling points, the directed distance field of the target object in the three-dimensional space is obtained.

3. The method according to claim 1, characterized in that, The acquisition of the first depth image of the target object includes: Obtain a true color image of the target object; A first depth image of the target object is generated based on the real color image using a depth map prediction model.

4. The method according to claim 1, characterized in that, The method further includes: The 3D reconstruction model is pre-trained using pre-training samples to obtain a pre-trained 3D reconstruction model. The pre-training samples include sample images and target 3D reconstruction information corresponding to the sample images. The pre-trained 3D reconstruction model is used to obtain the 3D reconstruction information of the target object based on the first depth image, and to train the pre-trained 3D reconstruction model based on the first depth image and the second depth image. The 3D reconstruction information includes the directed distance field.

5. The method according to claim 1, characterized in that, The step of generating a second depth image of the target object based on the directed distance field includes: Based on the directed distance field, the depth information of the target object in the two-dimensional coordinate system is obtained; Based on the depth information of the target object in a two-dimensional coordinate system, a second depth image of the target object is generated.

6. The method according to claim 1, characterized in that, The step of training the 3D reconstruction model based on the first depth image and the second depth image includes: The training loss of the 3D reconstruction model is determined based on the difference between the first depth image and the second depth image. The parameters of the 3D reconstruction model are adjusted based on the training loss.

7. The method according to any one of claims 1 to 6, characterized in that, The trained 3D reconstruction model is used for at least one of the following: Based on a real color image of the human body, the three-dimensional geometric configuration of the human body is generated in a three-dimensional virtual scene; Based on the real color image of the digital human, the three-dimensional geometric configuration of the digital human is generated in a three-dimensional virtual scene; Based on the real color image of the clothing, the three-dimensional geometric configuration of the clothing is generated in a three-dimensional virtual scene; Based on a real color image of a human face, the three-dimensional geometric configuration of the face is generated in a three-dimensional virtual scene.

8. A training device for a three-dimensional reconstruction model, characterized in that, The 3D reconstruction model includes: a feature voxel extraction sub-model and a multilayer perceptron; the device includes: The image acquisition module is used to acquire a first depth image of the target object. The first depth image is a real depth image obtained when the target object is photographed, or a depth image obtained based on a real color image of the target object. The information generation module is used to obtain the feature voxels of the target object based on the first depth image through the feature voxel extraction sub-model, wherein the feature voxels include feature information of the voxels corresponding to the target object; The information generation module is further configured to obtain a directed distance field of the target object in three-dimensional space based on the feature voxels using the multilayer perceptron, and the directed distance field is used to determine the three-dimensional geometric configuration of the target object in three-dimensional space; An image generation module is used to generate a second depth image of the target object based on the directional distance field; The model training module is used to train the 3D reconstruction model based on the first depth image and the second depth image. The trained 3D reconstruction model is used to determine the 3D geometric configuration of the object based on the input real color image of the object. The object includes a human body, a digital human, clothing, and a face. The target object has multiple first depth images, each corresponding to a multiple shooting angle. The multiple first depth images are respectively input into multiple three-dimensional reconstruction models to obtain the directed distance fields corresponding to the multiple first depth images. Each three-dimensional reconstruction model is used to process a first depth image corresponding to a shooting angle. The directed distance fields corresponding to the multiple first depth images are fused to obtain the directed distance field of the target object.

9. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing a computer program that is loaded and executed by the processor to implement the method as claimed in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that is loaded and executed by a processor to implement the method as described in any one of claims 1 to 7.

11. A computer program product, characterized in that, The computer program product includes computer instructions stored in a computer-readable storage medium, which a processor reads from and executes to implement the method as described in any one of claims 1 to 7.

Images