Texture generation method of three-dimensional model and training method of deep learning model
By converting the point cloud data of the three-dimensional model into a tree-like data structure and generating a Gaussian representation using an octree convolutional neural network, the problem of multi-view consistency and high-definition texture generation in the prior art is solved, and the problem of multi-view consistency and high-definition texture generation in the existing technology is solved. High-quality texture generation.
Patent Information
- Application Number
- CN202411274160.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-11
- Publication Date
- 2024-12-06
AI Technical Summary
The existing three-dimensional model texture generation methods are difficult to ensure the 3D consistency of multi-view images, and the texture generation model based on 3D networks is limited by 3D expression and resolution, making it difficult to generate high-definition textures.
Convert the point cloud data of the three-dimensional model into a tree-like data structure, store and process the partition block through the octree, generate a Gaussian representation using an octree-based convolutional neural network, and paste its rendering results on the 3D model. , combine guide information fusion features to generate high-quality textures.
The quality and consistency of three-dimensional model texture generation is improved, and the generated texture is more natural and high-definition, suitable for applications in multiple perspectives.
Smart Images

Figure CN119091024A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to computer vision, deep learning, large models and other technical fields, and can be applied to scenarios such as AIGC content generation based on artificial intelligence. More specifically, the present disclosure provides a texture generation method for a three-dimensional model, a training method for a deep learning model, a device, an electronic device, a storage medium and a computer program product. Background Art
[0002] With the rapid development of artificial intelligence, the application of image generation based on artificial intelligence is becoming more and more abundant. Texture generation refers to the use of trained neural networks to generate texture maps that match the guiding text or pictures for a given input 3D model that does not contain texture information, as well as pictures and text guidance, for subsequent use. Summary of the invention
[0003] The present disclosure provides a texture generation method for a three-dimensional model, a training method for a deep learning model, a device, an electronic device, a storage medium, and a computer program product.
[0004] According to a first aspect, a texture generation method for a three-dimensional model is provided, the method comprising: converting point cloud data of the three-dimensional model into a tree data structure, wherein the point cloud data partition blocks of the three-dimensional model are stored in multiple leaf nodes of the tree data structure; obtaining a Gaussian representation of the three-dimensional model based on the Gaussian representation of each block of the three-dimensional model, wherein the Gaussian representation of each block is generated based on the point cloud data stored in each leaf node; and fitting the rendering result of the Gaussian representation of the three-dimensional model to the three-dimensional model to obtain a three-dimensional model with texture information.
[0005] According to a second aspect, a training method for a deep learning model is provided, the method comprising: converting point cloud data of a sample model into a tree data structure, wherein the point cloud data partition blocks of the sample model are stored in multiple leaf nodes of the tree data structure; for each leaf node, using a deep learning model to process the point cloud data stored in the leaf node to obtain a Gaussian representation of the block to which the point cloud data of the leaf node belongs; obtaining a Gaussian representation of the sample model based on the Gaussian representation of each block of the sample model; rendering the Gaussian representation of the sample model to obtain an output texture image; determining the loss of the deep learning model based on the difference between the output texture image and the sample texture image, wherein the sample texture image is an image obtained by rendering the sample model; and adjusting the parameters of the deep learning model based on the loss.
[0006] According to a third aspect, a texture generation device for a three-dimensional model is provided, the device comprising: a first conversion module, used to convert point cloud data of the three-dimensional model into a tree data structure, wherein the point cloud data partition blocks of the three-dimensional model are stored in multiple leaf nodes of the tree data structure; a first Gaussian representation generation module, used to obtain a Gaussian representation of the three-dimensional model based on the Gaussian representation of each block of the three-dimensional model, wherein the Gaussian representation of each block is generated based on the point cloud data stored in each leaf node; and a texture generation module, used to fit the rendering result of the Gaussian representation of the three-dimensional model to the three-dimensional model to obtain a three-dimensional model with texture information.
[0007] According to a fourth aspect, a training device for a deep learning model is provided, the device comprising: a second conversion module, used to convert point cloud data of a sample model into a tree data structure, wherein the point cloud data partition blocks of the sample model are stored in multiple leaf nodes of the tree data structure; a leaf node processing module, used to process the point cloud data stored in the leaf node for each leaf node using a deep learning model, and obtain a Gaussian representation of the block to which the point cloud data of the leaf node belongs; a second Gaussian representation generation module, used to obtain a Gaussian representation of the sample model based on the Gaussian representation of each block of the sample model; a rendering module, used to render the Gaussian representation of the sample model to obtain an output texture image; a loss determination module, used to determine the loss of the deep learning model based on the difference between the output texture image and the sample texture image, wherein the sample texture image is an image obtained by rendering the sample model; and an adjustment module, used to adjust the parameters of the deep learning model based on the loss.
[0008] According to a fifth aspect, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method provided according to the present disclosure.
[0009] According to a sixth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the method provided according to the present disclosure.
[0010] According to a seventh aspect, a computer program product is provided, comprising a computer program, wherein the computer program is stored on at least one of a readable storage medium and an electronic device, and wherein the computer program implements the method provided according to the present disclosure when executed by a processor.
[0011] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.
[0013] Figure 1 is a schematic diagram of an exemplary system architecture to which a texture generation method for a three-dimensional model and a training method for a deep learning model can be applied according to an embodiment of the present disclosure;
[0014] Figure 2 is a flowchart of a texture generation method for a three-dimensional model according to an embodiment of the present disclosure;
[0015] Figure 3 is a schematic diagram of a texture generation method for a three-dimensional model according to an embodiment of the present disclosure;
[0016] Figure 4 is a flowchart of a method for training a deep learning model according to an embodiment of the present disclosure;
[0017] Figure 5 is a schematic diagram of a method for calculating the loss of a deep learning model according to an embodiment of the present disclosure;
[0018] Figure 6 is a schematic diagram of a training method for a deep learning model according to an embodiment of the present disclosure;
[0019] Figure 7 is a schematic diagram of a method for generating texture using a trained deep learning model according to an embodiment of the present disclosure;
[0020] Figure 8 is a block diagram of a texture generating apparatus for a three-dimensional model according to an embodiment of the present disclosure;
[0021] Fig. 9 is a block diagram of a training apparatus for a deep learning model according to an embodiment of the present disclosure;
[0022] Fig.10 It is a block diagram of an electronic device according to at least one of a texture generation method for a three-dimensional model and a training method for a deep learning model according to an embodiment of the present disclosure. DETAILED DESCRIPTION
[0023] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.
[0024] A 3D model is a three-dimensional, solid model. It can construct 3D models of various objects, such as people, animals, plants, machinery, etc. 3D models can be applied to scenarios such as artificial intelligence-based content generation, such as generating virtual digital humans.
[0025] Texture generation of a 3D model refers to adding color and details to the model to obtain a 3D model with texture information, making it look more realistic.
[0026] There are currently two main methods for generating textures for 3D models. One is to use a 2D diffusion model to generate multiple texture images that fit the input 3D model at multiple viewpoints, and then fine-tune them on a 2D plane. This method makes it difficult to ensure that the generated multi-view images have 3D consistency, resulting in a low success rate for generating usable textures.
[0027] The other is to directly use 3D networks to generate texture information on the surface of objects. Existing texture generation models based on 3D networks are limited by 3D expression and resolution, and it is difficult to generate high-definition textures.
[0028] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.
[0029] In the technical solution of the present disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.
[0030] Figure 1 is a schematic diagram of an exemplary system architecture to which a texture generation method for a three-dimensional model and a training method for a deep learning model can be applied according to an embodiment of the present disclosure. It should be noted that: Figure 1 What is shown is merely an example of a system architecture to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.
[0031] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.
[0032] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Terminal devices 101, 102, 103 can be various electronic devices, including but not limited to smart phones, tablet computers, laptops, etc.
[0033] At least one of the texture generation method of the three-dimensional model and the training method of the deep learning model provided in the embodiments of the present disclosure can generally be executed by the server 105. Accordingly, at least one of the texture generation device of the three-dimensional model and the training device of the deep learning model provided in the embodiments of the present disclosure can generally be set in the server 105. The texture generation method of the three-dimensional model and the training method of the deep learning model provided in the embodiments of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Accordingly, the texture generation device of the three-dimensional model and the training device of the deep learning model provided in the embodiments of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105.
[0034] Figure 2 The present invention is a flowchart of a method for generating texture of a three-dimensional model according to an embodiment of the present invention.
[0035] like Figure 2 As shown, the texture generation method 200 of the three-dimensional model includes operations S210 to S230.
[0036] In operation S210 , the point cloud data of the three-dimensional model is converted into a tree data structure, and the partition blocks of the point cloud data of the three-dimensional model are stored in a plurality of leaf nodes of the tree data structure.
[0037] The 3D model may be a geometric body without texture information, that is, the 3D model has no color. Point cloud data may be collected on the surface of the 3D model and stored in the form of a tree data structure.
[0038] The tree-like data structure can be an octree. An octree is a data structure that divides a three-dimensional space into cubes of equal size. Each cube can be further divided into eight sub-cubes, and this division level by level forms a tree structure.
[0039] The point cloud data of the 3D model can be partitioned and stored in multiple leaf nodes of the tree data structure. For example, the geometric information of the 3D model (which can be local coordinates or local normal vectors) can be stored in each leaf node of the octree, and each leaf node stores the geometric information of a local block of the 3D model. The geometric information stored in all leaf nodes on the octree constitutes the point cloud data of the entire 3D model.
[0040] In operation S220, a Gaussian representation of the three-dimensional model is obtained according to the Gaussian representation of each block of the three-dimensional model.
[0041] The Gaussian representation of each block is generated based on the point cloud data stored in each leaf node. For example, for each leaf node, the Gaussian representation of the block to which it belongs can be generated based on the stored point cloud data.
[0042] For example, after the point cloud data of the 3D model is partitioned and stored in multiple leaf nodes of the octree, the octree can be processed using an octree-based convolutional neural network (O-CNN). O-CNN performs convolution operations based on the octree data structure. Since the octree data structure can represent 3D shapes in a hierarchical manner, it can effectively reduce the demand for computing resources. By performing convolution operations on the octree, efficient 3D convolution operations are achieved, and point cloud data of different densities can be adaptively processed.
[0043] For example, a 3D UNet model (which can be called 3DOctree UNet) can be constructed using an octree-based convolutional neural network, and the 3D Octree UNet can be run on the octree to output a Gaussian representation of the point cloud at each leaf node of the octree. The Gaussian representation can be a Gaussian sphere with attributes such as color, size, and opacity.
[0044] For example, for each leaf node, the point cloud data stored in the leaf node can be input into the above 3D OctreeUNet, and the 3D Octree UNet performs feature extraction on the point cloud data stored in the leaf node, and obtains the Gaussian sphere of the block to which the point cloud data of the leaf node belongs based on the extracted point cloud features.
[0045] After all leaf nodes on the octree are processed by the above 3D Octree UNet, the Gaussian balls of the blocks to which each node belongs are obtained. The Gaussian balls of all blocks constitute the overall Gaussian representation of the 3D model.
[0046] The embodiment of the present disclosure utilizes an octree to spatially divide the point cloud data of a three-dimensional model, stores the point cloud data partition blocks in the leaf nodes of the octree, and performs point cloud processing in the partition blocks, thereby improving the processing efficiency of the point cloud processing.
[0047] In operation S230 , the rendering result of the Gaussian representation of the three-dimensional model is fitted to the three-dimensional model to obtain a three-dimensional model with texture information.
[0048] After obtaining the Gaussian representation of the three-dimensional model, the Gaussian representation can be rendered at any viewing angle. Three-dimensional rendering refers to the process of adding texture information to the three-dimensional model and converting the three-dimensional model into a two-dimensional image with texture information. The Gaussian representation of the three-dimensional model contains color information, so rendering the Gaussian representation of the three-dimensional model can refer to converting the Gaussian sphere into a two-dimensional image, and the converted two-dimensional image has texture information.
[0049] For example, a Gaussian sphere of a three-dimensional model can be projected onto a two-dimensional plane using Gaussian splashing to obtain an RGB image with texture information. The RGB image is the rendering result based on the Gaussian sphere. The RGB image is attached to the three-dimensional model to obtain a three-dimensional model with texture information.
[0050] The embodiment of the present disclosure processes the point cloud data of the three-dimensional model by partitioning the blocks into a tree data structure, and obtains the Gaussian representation of each block of the three-dimensional model, thereby improving the generation efficiency of the Gaussian representation. By fitting the rendering result of the Gaussian representation to the three-dimensional model, a three-dimensional model with texture information is obtained, which can improve the quality of texture generation of the three-dimensional model.
[0051] According to an embodiment of the present disclosure, a Gaussian representation of a three-dimensional model is rendered at each of multiple perspectives to obtain first texture images of multiple perspectives; and the first texture images of multiple perspectives are fitted to the three-dimensional model to obtain a three-dimensional model with texture information.
[0052] Observing a 3D model from different perspectives has different texture information, so the rendering of the 3D model includes rendering from multiple perspectives. The rendering results of the Gaussian representation of the 3D model at different perspectives can be fitted to the 3D model to obtain a complete 3D model with texture information.
[0053] The Gaussian sphere of the three-dimensional model is rendered at multiple viewing angles to obtain first texture images at multiple viewing angles. The first texture images at multiple viewing angles are attached to the three-dimensional model to obtain a three-dimensional model having texture information at multiple viewing angles.
[0054] In one example, the 3D model can be UV unfolded, which refers to the process of converting the 3D model into a 2D representation. The 2D plane after the 3D model is UV unfolded can be called a UV plane. By fine-tuning the first texture images of multiple perspectives on the UV plane, a 3D model with texture information can be obtained.
[0055] The 3D model and the UV plane are mapped. The first texture image is placed on the UV plane so that the UV plane has texture information. The pixel points on the UV plane have corresponding points on the 3D model. Therefore, the texture information of the pixel points on the UV plane can be synchronized to the corresponding points on the 3D model so that the 3D model also has texture information.
[0056] Adjusting the first texture images of multiple perspectives on the UV plane can make the first texture images of multiple perspectives more consistent. While adjusting the first texture image on the UV plane, the three-dimensional model is also adjusted accordingly according to the mapping relationship between the three-dimensional model and the UV plane, so that the texture information of the three-dimensional model is also more consistent. However, such direct adjustment on the UV plane may cause deformation of the three-dimensional model, texture distortion, etc., resulting in poor texture quality generated by the three-dimensional model.
[0057] In view of this, in order to more effectively fit the rendering result of Gaussian representation to the three-dimensional model, the texture generation method of the embodiment of the present disclosure provides a method for baking the rendering result of Gaussian representation to the three-dimensional model, which can obtain a more natural three-dimensional model with clearer texture.
[0058] Figure 3 is a schematic diagram of a texture generation method for a three-dimensional model according to an embodiment of the present disclosure.
[0059] According to an embodiment of the present disclosure, a Gaussian representation of a three-dimensional model is rendered to obtain a first texture image of multiple perspectives; the three-dimensional model is rendered to obtain a second texture image; and based on the color difference between the first texture image and the second texture image, the color of the three-dimensional model is adjusted to obtain a three-dimensional model with texture information.
[0060] It should be noted that Figure 3 The texture generation method shown is a schematic diagram of texture generation at any viewing angle, and the following is performed at multiple viewing angles: Figure 3 The texture generation method shown can obtain a complete three-dimensional model with texture information.
[0061] like Figure 3 As shown, at any viewing angle, the Gaussian representation 310 of the three-dimensional model and the three-dimensional model 320 itself are rendered respectively to obtain a first texture image 311 and a second texture image 321 .
[0062] Rendering the Gaussian representation 310 of the three-dimensional model may be projecting a Gaussian sphere onto a two-dimensional plane to obtain a first texture image. Rendering the three-dimensional model 320 may be projecting the three-dimensional model onto a two-dimensional plane to obtain a second texture image. If the rendering result of the Gaussian representation 310 is completely fitted to the three-dimensional model 320, the color information of the first texture image 311 and the second texture image 321 should be consistent.
[0063] In order to make the color information of the first texture image 311 and the second texture image 321 consistent, the color information of the three-dimensional model 320 can be used as an optimization parameter to optimize the color on the three-dimensional model 320 according to the color difference 330 between the first texture image 311 and the second texture image 321 .
[0064] For example, a differentiable rendering library may be used to perform differentiable rendering on the three-dimensional model 320. Differentiable rendering provides a differentiable function so that the rendering process can be derived, thereby allowing the use of gradient descent or other optimization algorithms to adjust parameters. In this embodiment, the parameter to be optimized may be color. During the rendering process, the color difference 330 between the first texture image 311 and the second texture image 321 is calculated to optimize the color of the three-dimensional model 320.
[0065] According to an embodiment of the present disclosure, the three-dimensional model is unfolded to obtain a two-dimensional image of the three-dimensional model; the color of the two-dimensional image is adjusted according to the color difference; and in response to the adjustment of the color on the two-dimensional image, the color on the three-dimensional model is adjusted.
[0066] The three-dimensional model 320 is UV-unwrapped to obtain a UV plane 322. The UV plane 322 and the three-dimensional model 320 are mapped to each other. The pixel points on the UV plane 322 have corresponding points on the three-dimensional model 320. By adjusting the color information of the pixel points on the UV plane 322, the color information of the points on the three-dimensional model 320 can be adjusted.
[0067] For example, the color of the corresponding pixel on the UV plane 320 can be adjusted according to the color difference 330 of each pixel between the first texture image 311 and the second texture image 321, and the color of the point on the three-dimensional model is adjusted as the color of the pixel on the UV plane 320 is adjusted.
[0068] The embodiments of the present disclosure calculate the difference between a first texture image and a second texture image, and adjust the color of a UV plane of a three-dimensional model according to the difference, so that the color of the three-dimensional model is adjusted accordingly until the colors of the first texture image and the second texture image are consistent. The obtained three-dimensional model with texture information is completely consistent with the rendering result of the Gaussian representation of the three-dimensional model, thereby improving the texture generation quality of the three-dimensional model.
[0069] According to an embodiment of the present disclosure, when processing a point cloud using the 3D UNet model (3D Octree UNet) constructed using the octree-based convolutional neural network, guidance information can be added to make the subsequently generated texture image more in line with the requirements. The guidance information can include at least one of a guidance image and a guidance text.
[0070] According to an embodiment of the present disclosure, for each leaf node, the guiding feature is fused with the point cloud feature of the point cloud data to obtain a fused feature, wherein the guiding feature includes at least one of a text feature and an image feature; and a Gaussian representation of the block to which the point cloud data of the leaf node belongs is generated based on the fused feature.
[0071] For example, the 3D Octree UNet model includes multiple processing layers, and the point cloud features and guide features can be fused in each processing layer. The fused features are then input to the next layer until the fused features are output by the last layer. Based on the fused features of the last layer, the Gaussian representation of the point cloud data can be obtained.
[0072] For example, a CrossAttention module can be connected to each layer of the 3D Octree UNet model. The point cloud features output by each layer of the 3D Octree UNet model are fused with the features of the text / image in the CrossAttention module to obtain the fused features, which are then input into the next layer of the 3D Octree UNet model.
[0073] In the process of processing point cloud data, the embodiments of the present disclosure can generate a Gaussian representation that better meets the needs by fusing the guiding features of text or pictures, and then generate texture information that better meets the needs.
[0074] According to an embodiment of the present disclosure, the present disclosure also provides a method for training a deep learning model. The deep learning model may be the 3D UNet model (3D Octree UNet) constructed by the octree-based convolutional neural network.
[0075] Figure 4 It is a flowchart of a method for training a deep learning model according to an embodiment of the present disclosure.
[0076] like Figure 4 As shown, the deep learning model training method 400 includes operations S410 to S460.
[0077] In operation S410 , point cloud data of a sample model is converted into a tree data structure, and partition blocks of the point cloud data of the sample model are stored in a plurality of leaf nodes of the tree data structure.
[0078] The sample model can be a three-dimensional geometric body with texture information. Point cloud data is collected on the surface of the sample model and converted into an octree data structure. The point cloud data of a local block of the sample model is stored on each leaf node of the octree. The point cloud data of all leaf nodes of the octree constitute the point cloud of the entire sample model.
[0079] In operation S420, for each leaf node, the point cloud data stored in the leaf node is processed using a deep learning model to obtain a Gaussian representation of the block to which the point cloud data of the leaf node belongs.
[0080] For example, the deep learning model builds a 3D UNet model (3D OctreeUNet) based on an octree-based convolutional neural network. Running the 3D Octree UNet on the octree can output a Gaussian representation of the point cloud at each leaf node of the octree. The Gaussian representation can be a Gaussian sphere with attributes such as color, size, and opacity. Since the position of the Gaussian sphere is fixed at the center of the leaf node of the octree, that is, the position of the Gaussian sphere has been known based on the data structure of the octree, the 3D Octree UNet model does not need to predict the position information, but only needs to predict other attributes of each Gaussian sphere, such as opacity, the amount of expansion on the three coordinate axes of the spatial coordinate system, the rotation quaternion, and the color RGB information.
[0081] For example, for each leaf node, the point cloud data stored in the leaf node can be input into the above 3D OctreeUNet, and the 3D Octree UNet performs feature extraction on the point cloud data stored in the leaf node, and obtains the Gaussian sphere of the block to which the point cloud data of the leaf node belongs based on the extracted point cloud features.
[0082] In operation S430 , a Gaussian representation of the sample model is obtained according to the Gaussian representation of each block of the sample model.
[0083] For each leaf node, the 3D Octree UNet model is used to process the point cloud data stored in the leaf node to obtain the Gaussian ball of the block to which the point cloud data of the leaf node belongs. All leaf nodes are processed by the deep learning model to obtain the Gaussian balls of each block of the sample model. The Gaussian balls of all blocks constitute the overall Gaussian representation of the sample model.
[0084] In operation S440, the Gaussian representation of the sample model is rendered to obtain an output texture image.
[0085] The overall Gaussian sphere of the sample model can be projected onto a two-dimensional plane to obtain an output texture image.
[0086] In operation S450, a loss of the deep learning model is determined according to a difference between the output texture image and the sample texture image.
[0087] The sample texture image may be a true value texture image obtained by rendering the sample model.
[0088] Based on the difference between the output texture image obtained by Gaussian sphere rendering and the true texture image, the loss of the deep learning model can be determined.
[0089] In operation S460, parameters of the deep learning model are adjusted according to the loss.
[0090] The parameters of the deep learning model can be adjusted according to the loss, such as adjusting the parameters of the convolutional neural network of each layer of the 3D Octree UNet to complete a round of training.
[0091] After multiple rounds of training, a trained deep learning model is obtained. The trained deep learning model can be run on the octree to predict the Gaussian sphere on the leaf node based on the point cloud in the leaf node of the octree.
[0092] The embodiments of the present disclosure store the point cloud data partition blocks of the sample model in a tree data structure, process the point cloud data by the partition blocks, obtain Gaussian representations of each block of the sample model, and determine the loss of the deep learning model based on the difference between the output texture image obtained by Gaussian sphere rendering and the true texture image obtained by rendering the sample model to adjust the parameters of the deep learning model, thereby improving the accuracy of the deep learning model in generating Gaussian representations.
[0093] According to an embodiment of the present disclosure, the sample texture image includes multiple sample texture images obtained by rendering the sample model at multiple viewing angles. For each sample texture image, a camera pose corresponding to the viewing angle of the sample texture image is determined; and according to the camera pose, a Gaussian representation of the sample model is rendered to obtain an output texture image with the same viewing angle as the sample texture image.
[0094] By rendering the sample model at different viewing angles, sample texture images at different viewing angles can be obtained. Therefore, the sample texture images in the training set may include sample texture images at multiple viewing angles. Correspondingly, when rendering the Gaussian representation of the sample model, it can also be rendered at different viewing angles. In order to calculate the accurate loss of the deep learning model, it is necessary to calculate the difference between the sample texture image and the output texture image at the same viewing angle.
[0095] Therefore, the camera poses of each multi-view sample texture image can be saved in the training set. After obtaining the Gaussian representation of the sample model, the Gaussian representation can be rendered in multiple perspectives according to the camera poses saved in the training set to obtain the output texture image in multiple perspectives.
[0096] Next, the loss of the deep learning model can be determined based on the difference between the sample texture image and the output texture image at the same viewpoint.
[0097] Figure 5 is a schematic diagram of a method for calculating the loss of a deep learning model according to an embodiment of the present disclosure.
[0098] like Figure 5 As shown, the sample texture image 510 may be obtained by rendering the sample model at multiple viewing angles, and may include sample texture images at multiple viewing angles, such as a first viewing angle sample texture image, a second viewing angle sample texture image, etc. Each viewing angle may be a viewing angle formed by rotating a shooting source at different angles at a certain position.
[0099] The output texture image 520 may be obtained by rendering the Gaussian representation of the sample model at the above-mentioned multiple perspectives, and may include output texture images at multiple perspectives, such as a first perspective output texture image, a second perspective output texture image, etc.
[0100] For example, after obtaining the Gaussian representation of the sample model, multiple pre-stored camera poses corresponding to each of the multiple perspectives may be obtained, such as the first perspective camera pose, the second perspective camera pose, etc. The Gaussian representation is rendered at multiple perspectives according to these camera poses to obtain the above-mentioned multi-perspective output texture image 520.
[0101] According to an embodiment of the present disclosure, for each viewing angle, the loss at the viewing angle is determined according to the difference between the sample texture image and the output texture image at the viewing angle; and the loss of the deep learning model is determined according to the losses at multiple viewing angles.
[0102] According to the difference between the sample texture image and the output texture image at the same viewing angle, the losses of multiple viewing angles 530 can be determined. For example, the difference between the first viewing angle sample texture image and the first viewing angle output texture image determines the first viewing angle loss. The difference between the second viewing angle sample texture image and the second viewing angle output texture image determines the second viewing angle loss. And so on, the losses of multiple viewing angles are obtained. According to the losses of multiple viewing angles, the loss of the deep learning model can be calculated.
[0103] According to an embodiment of the present disclosure, for each viewing angle, the pixel difference, style difference and object contour difference between the sample texture image and the output texture image at that viewing angle are determined; based on the pixel difference, style difference and object contour difference, the pixel loss, style loss and object contour loss are determined; based on the object contour difference, the pixel loss, style loss and object contour loss are determined to determine the overall loss under the viewing angle.
[0104] At each viewing angle, the difference between the sample texture image and the output texture image may include pixel difference, style difference, and object contour difference. The loss at this viewing angle may include pixel loss, style loss, and object contour loss. The overall loss at this viewing angle may be obtained by weighted summing the pixel loss, style loss, and object contour loss.
[0105] The overall losses of multiple perspectives are then added or weighted to obtain the overall loss of the deep learning model.
[0106] The embodiments of the present disclosure improve the accuracy of the loss by determining the loss based on the difference between the sample texture image and the output texture image under the same viewing angle. The overall loss is determined by calculating the loss of multiple aspects such as pixels, style, and object contours, so that the deep learning model can generate a higher quality Gaussian ball.
[0107] Figure 6 It is a schematic diagram of a training method for a deep learning model according to an embodiment of the present disclosure.
[0108] Figure 6 The training method of the deep learning model shown only shows the back propagation of the loss under one perspective. The actual training includes the back propagation of the loss of multiple perspectives to adjust the parameters of the deep learning model.
[0109] like Figure 6 As shown, the point cloud data 610 may be the point cloud of the sample model, and an octree is constructed based on the point cloud data 610 to obtain an octree 620. Each leaf node of the octree 620 stores the point cloud of a local block of the sample model.
[0110] The point cloud and guide text of each leaf node of the octree 620 are input into a 3D UNet model (3D Octree UNet) based on the convolutional neural network of the octree to obtain the Gaussian representation of the block to which the point cloud of the leaf node belongs. The guide text is, for example, "This is a colorful cow". In addition, a guide picture can also be input, and the guide picture can be one or more of the multi-view pictures of the sample model.
[0111] The Gaussian representation of each leaf node constitutes the Gaussian representation of the sample model 630. The Gaussian representation 630 is rendered to obtain an output texture image 640. The 3DOctree UNet is calculated based on the sample texture image 650 of the same viewing angle as the output texture image 640. The sample texture image 650 is obtained by rendering the sample model.
[0112] According to the calculated loss, back propagation can be performed to adjust the parameters of the 3D Octree UNet and complete a round of training. After multiple rounds of training, a trained 3D Octree UNet is obtained. The trained 3D Octree UNet can generate high-quality Gaussian spheres for point cloud data.
[0113] According to an embodiment of the present disclosure, for each leaf node, the guiding feature is fused with the point cloud feature of the point cloud data to obtain a fused feature, wherein the guiding feature includes at least one of a text feature and an image feature; and a Gaussian representation of the block to which the point cloud data of the leaf node belongs is generated based on the fused feature.
[0114] The deep learning model may include multiple processing layers, and the point cloud features and the guide features may be fused at each processing layer. The fused features are then input to the next layer until the fused features are output by the last layer. The Gaussian representation of the point cloud data may be obtained based on the fused features of the last layer.
[0115] For example, the deep learning model is a 3D Octree UNet model, which includes multiple layers of convolutional neural networks. A cross-attention module can be connected to each layer of the convolutional neural network. The point cloud features output by each layer of the convolutional neural network are fused with the features of the text / picture in the cross-attention module based on the cross-attention mechanism to obtain the fused features, which are then input into the next layer of the convolutional neural network.
[0116] In the process of processing point cloud data, the embodiments of the present disclosure can generate a Gaussian representation that better meets the needs by fusing the guiding features of text or pictures, and then generate texture information that better meets the needs.
[0117] Figure 7 It is a schematic diagram of a method for generating texture using a trained deep learning model according to an embodiment of the present disclosure.
[0118] like Figure 7 As shown, object 710 may be a three-dimensional model for which texture information is to be generated from a frontal perspective. Object 720 may be a texture image obtained by performing microscopic rendering on the three-dimensional model from a frontal perspective. Object 730 may be a texture image obtained by rendering a Gaussian representation of the three-dimensional model from a frontal perspective. The Gaussian representation of the three-dimensional model may be generated using a trained deep learning model (such as 3D Octree UNet).
[0119] If the rendering result of the Gaussian representation of the three-dimensional model is completely fitted to the three-dimensional model, the color information of the object 720 and the object 730 should be consistent.
[0120] In order to more effectively attach the rendering results of the Gaussian representation to the 3D model, a differentiable rendering library for the mesh of the 3D model can be used for differentiable rendering. First, the Mesh can be UV unfolded, and the RGB information on the UV plane can be used as an optimizable parameter. Then the camera pose (such as the pose of the front view of the 3D model) can be randomly selected, and the Mesh and the 3D Gaussian ball can be rendered at the same time. The rendering results are, for example, object 720 and object 730. The difference between the results of the two renderings is calculated, and the difference is used to adjust the RGB value of the UV plane. The color on the 3D model is adjusted as the color on the UV plane is adjusted. In this way, the rendering results of the Gaussian ball can be easily baked onto the 3D model.
[0121] According to an embodiment of the present disclosure, the present disclosure also provides a texture generation device for a three-dimensional model and a training device for a deep learning model.
[0122] Figure 8 The invention is a block diagram of a texture generating device for a three-dimensional model according to an embodiment of the present invention.
[0123] like Figure 8 As shown, the texture generation device 800 of the three-dimensional model includes a first conversion module 810 , a first Gaussian representation generation module 820 and a texture generation module 830 .
[0124] The first conversion module 810 is used to convert the point cloud data of the three-dimensional model into a tree data structure, wherein the partition blocks of the point cloud data of the three-dimensional model are stored in a plurality of leaf nodes of the tree data structure.
[0125] The first Gaussian representation generation module 820 is used to obtain the Gaussian representation of the three-dimensional model according to the Gaussian representation of each block of the three-dimensional model, wherein the Gaussian representation of each block is generated according to the point cloud data stored in each leaf node.
[0126] The texture generation module 830 is used to fit the rendering result of the Gaussian representation of the three-dimensional model to the three-dimensional model to obtain a three-dimensional model with texture information.
[0127] The texture generation module 830 includes a first texture generation unit and a second texture generation unit.
[0128] The first texture generation unit is used to render the Gaussian representation of the three-dimensional model at each of the multiple viewing angles to obtain first texture images of the multiple viewing angles.
[0129] The second texture generation unit is used to fit the first texture images of multiple perspectives to the three-dimensional model to obtain a three-dimensional model with texture information.
[0130] The second texture generating unit includes a texture generating subunit and a color adjusting subunit.
[0131] The texture generation subunit is used to render the three-dimensional model at each viewing angle to obtain a second texture image.
[0132] The color adjustment subunit is used to adjust the color of the three-dimensional model according to the color difference between the first texture image and the second texture image to obtain a three-dimensional model with texture information.
[0133] The color adjustment subunit is used to map the three-dimensional model into a two-dimensional image; adjust the color of the two-dimensional image according to the color difference; and in response to the adjustment of the color on the two-dimensional image, adjust the color on the three-dimensional model according to the mapping relationship between the three-dimensional model and the two-dimensional image.
[0134] The texture generation device 800 for a three-dimensional model further includes a feature fusion module and a feature processing module.
[0135] The feature fusion module is used to fuse the guide feature with the point cloud feature of the point cloud data for each leaf node to obtain a fused feature, wherein the guide feature includes at least one of a text feature and an image feature.
[0136] The feature fusion module is used to generate the Gaussian representation of the block to which the point cloud data of the leaf node belongs based on the fused features.
[0137] Fig. 9 is a block diagram of a training device for a deep learning model according to an embodiment of the present disclosure.
[0138] like Fig. 9 As shown, the training device 900 of the deep learning model includes a second conversion module 910, a leaf node processing module 920, a second Gaussian representation generation module 930, a rendering module 940, a loss determination module 950 and an adjustment module 960.
[0139] The second conversion module 910 is used to convert the point cloud data of the sample model into a tree data structure, wherein the partition blocks of the point cloud data of the sample model are stored in a plurality of leaf nodes of the tree data structure.
[0140] The leaf node processing module 920 is used to process the point cloud data stored in each leaf node using a deep learning model to obtain a Gaussian representation of the block to which the point cloud data of the leaf node belongs.
[0141] The second Gaussian representation generating module 930 is used to obtain the Gaussian representation of the sample model according to the Gaussian representation of each block of the sample model.
[0142] The rendering module 940 is used to render the Gaussian representation of the sample model to obtain an output texture image.
[0143] The loss determination module 950 is used to determine the loss of the deep learning model based on the difference between the output texture image and the sample texture image, wherein the sample texture image is an image obtained by rendering the sample model.
[0144] The adjustment module 960 is used to adjust the parameters of the deep learning model according to the loss.
[0145] The sample texture images include multiple sample texture images obtained by rendering the sample model at multiple viewing angles. The rendering module 940 includes a posture determination unit and a rendering unit.
[0146] The pose determination unit is used to determine, for each sample texture image, a camera pose corresponding to the viewing angle of the sample texture image.
[0147] The rendering unit is used to render the Gaussian representation of the sample model according to the camera posture to obtain an output texture image with the same viewing angle as the sample texture image.
[0148] The loss determination module 950 includes a first loss determination unit and a second loss determination unit.
[0149] The first loss determination unit is used to determine the loss under each viewing angle according to the difference between the sample texture image and the output texture image under the viewing angle.
[0150] The second loss determination unit is used to determine the loss of the deep learning model according to the losses under multiple perspectives.
[0151] The first loss determining unit includes a difference determining subunit, a first loss determining subunit, and a second loss determining subunit.
[0152] The difference determination subunit is used to determine the pixel difference, style difference and object contour difference between the sample texture image and the output texture image at the viewing angle.
[0153] The first loss determination subunit is used to determine pixel loss, style loss and object contour loss according to pixel difference, style difference and object contour difference.
[0154] The second loss determination subunit is used to determine the pixel loss, style loss and object contour loss according to the object contour difference, and determine the overall loss under the viewing angle.
[0155] The leaf node processing module 920 includes a feature fusion unit and a feature processing unit.
[0156] The feature fusion unit is used to fuse the guiding feature with the point cloud feature of the point cloud data to obtain a fused feature, wherein the guiding feature includes at least one of a text feature and an image feature.
[0157] The feature processing unit is used to generate a Gaussian representation of the block to which the point cloud data of the leaf node belongs according to the fused features.
[0158] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.
[0159] Fig.10 A schematic block diagram of an example electronic device 1000 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.
[0160] like Fig.10 As shown, the device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the device 1000 can also be stored. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.
[0161] A number of components in the device 1000 are connected to the I / O interface 1005, including: an input unit 1006, such as a keyboard, a mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a disk, an optical disk, etc.; and a communication unit 1009, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1009 allows the device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0162] The computing unit 1001 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1001 performs the various methods and processes described above, such as at least one of the texture generation method of the three-dimensional model and the training method of the deep learning model. For example, in some embodiments, at least one of the texture generation method of the three-dimensional model and the training method of the deep learning model may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into RAM 1003 and executed by the computing unit 1001, one or more steps of at least one of the texture generation method of the three-dimensional model and the training method of the deep learning model described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to execute at least one of the texture generation method of the three-dimensional model and the training method of the deep learning model by any other appropriate means (e.g., by means of firmware).
[0163] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0164] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0165] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0166] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0167] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.
[0168] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.
[0169] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.
[0170] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.
Claims
1. A texture generation method for a three-dimensional model, include: Converting point cloud data of the three-dimensional model into a tree data structure, wherein partition blocks of the point cloud data of the three-dimensional model are stored in a plurality of leaf nodes of the tree data structure; Obtaining a Gaussian representation of the three-dimensional model according to the Gaussian representation of each block of the three-dimensional model, wherein the Gaussian representation of each block is generated according to the point cloud data stored in each leaf node; and The rendering result of the Gaussian representation of the three-dimensional model is fitted to the three-dimensional model to obtain a three-dimensional model with texture information.
2. The method according to claim 1, in, The step of fitting the rendering result of the Gaussian representation of the three-dimensional model to the three-dimensional model to obtain the three-dimensional model with texture information comprises: Rendering the Gaussian representation of the three-dimensional model at each of the multiple viewing angles to obtain first texture images at the multiple viewing angles; and The first texture images of the multiple perspectives are fitted to the three-dimensional model to obtain the three-dimensional model with texture information.
3. The method according to claim 2, in, The step of fitting the first texture images of the multiple perspectives to the three-dimensional model to obtain the three-dimensional model with texture information comprises: at each perspective, Rendering the three-dimensional model to obtain a second texture image; and According to the color difference between the first texture image and the second texture image, the color of the three-dimensional model is adjusted to obtain a three-dimensional model with texture information.
4. The method according to claim 3, in, The adjusting the color of the three-dimensional model according to the color difference between the first texture image and the second texture image to obtain the three-dimensional model with color information comprises: Mapping the three-dimensional model into a two-dimensional image; adjusting the color of the two-dimensional image according to the color difference; and In response to the adjustment of the color on the two-dimensional image, the color on the three-dimensional model is adjusted according to the mapping relationship between the three-dimensional model and the two-dimensional image.
5. The method according to claim 1, further comprising: include: For each leaf node, fusing the guide feature with the point cloud feature of the point cloud data to obtain a fused feature, wherein the guide feature includes at least one of a text feature and an image feature; and A Gaussian representation of the block to which the point cloud data of the leaf node belongs is generated according to the fusion feature.
6. A training method for a deep learning model, include: Converting the point cloud data of the sample model into a tree data structure, wherein the partition blocks of the point cloud data of the sample model are stored in a plurality of leaf nodes of the tree data structure; For each leaf node, use a deep learning model to process the point cloud data stored in the leaf node to obtain a Gaussian representation of the block to which the point cloud data of the leaf node belongs; Obtaining a Gaussian representation of the sample model according to the Gaussian representation of each block of the sample model; Rendering the Gaussian representation of the sample model to obtain an output texture image; determining a loss of the deep learning model according to a difference between the output texture image and a sample texture image, wherein the sample texture image is an image rendered by the sample model; and Based on the loss, parameters of the deep learning model are adjusted.
7. The method according to claim 6, in, The sample texture image includes multiple sample texture images obtained by rendering the sample model at multiple viewing angles; the rendering of the Gaussian representation of the sample model to obtain the output texture image includes: For each sample texture image, determining a camera pose corresponding to the viewing angle of the sample texture image; and The Gaussian representation of the sample model is rendered according to the camera pose to obtain an output texture image with the same viewing angle as the sample texture image.
8. The method according to claim 7, in, Determining the loss of the deep learning model according to the difference between the output texture image and the sample texture image includes: For each viewing angle, determining the loss at the viewing angle according to the difference between the sample texture image and the output texture image at the viewing angle; and The loss of the deep learning model is determined according to the losses under the multiple perspectives.
9. The method according to claim 8, in, For each viewing angle, determining the loss under the viewing angle according to the difference between the sample texture image and the output texture image under the viewing angle comprises: for each viewing angle, Determine pixel differences, style differences, and object contour differences between the sample texture image and the output texture image at the viewing angle; Determining pixel loss, style loss and object contour loss according to the pixel difference, style difference and object contour difference; According to the object contour difference, pixel loss, style loss and object contour loss are determined to determine the overall loss under the viewing angle.
10. The method according to claim 6, in, For each leaf node, using a deep learning model to process the point cloud data stored in the leaf node to obtain a Gaussian representation of the block to which the point cloud data of the leaf node belongs includes: for each leaf node, fusing the guide feature with the point cloud feature of the point cloud data to obtain a fused feature, wherein the guide feature includes at least one of a text feature and an image feature; and A Gaussian representation of the block to which the point cloud data of the leaf node belongs is generated according to the fusion feature.
11. A texture generating device for a three-dimensional model, include: A first conversion module, configured to convert point cloud data of a three-dimensional model into a tree data structure, wherein partition blocks of the point cloud data of the three-dimensional model are stored in a plurality of leaf nodes of the tree data structure; a first Gaussian representation generating module, configured to obtain a Gaussian representation of the three-dimensional model according to the Gaussian representation of each block of the three-dimensional model, wherein the Gaussian representation of each block is generated according to the point cloud data stored in each leaf node; and The texture generation module is used to fit the rendering result of the Gaussian representation of the three-dimensional model to the three-dimensional model to obtain a three-dimensional model with texture information.
12. The device according to claim 11, in, The texture generation module comprises: A first texture generation unit is used to render the Gaussian representation of the three-dimensional model at each of a plurality of viewing angles to obtain first texture images at the plurality of viewing angles; and The second texture generation unit is used to fit the first texture images of the multiple perspectives to the three-dimensional model to obtain the three-dimensional model with texture information.
13. The device according to claim 12, in, The second texture generation unit comprises: a texture generation subunit, configured to render the three-dimensional model at each viewing angle to obtain a second texture image; and The color adjustment subunit is used to adjust the color of the three-dimensional model according to the color difference between the first texture image and the second texture image to obtain a three-dimensional model with texture information.
14. The device according to claim 13, in, The color adjustment subunit is used to map the three-dimensional model into a two-dimensional image; adjust the color of the two-dimensional image according to the color difference; and in response to the adjustment of the color on the two-dimensional image, adjust the color on the three-dimensional model according to the mapping relationship between the three-dimensional model and the two-dimensional image.
15. The device according to claim 11, further comprising: include: a feature fusion module, configured to fuse the guide feature with the point cloud feature of the point cloud data for each leaf node to obtain a fused feature, wherein the guide feature includes at least one of a text feature and an image feature; and The feature processing module is used to generate a Gaussian representation of the block to which the point cloud data of the leaf node belongs according to the fusion feature.
16. A training device for a deep learning model, include: A second conversion module, configured to convert the point cloud data of the sample model into a tree data structure, wherein the partition blocks of the point cloud data of the sample model are stored in a plurality of leaf nodes of the tree data structure; A leaf node processing module, for processing the point cloud data stored in each leaf node using a deep learning model to obtain a Gaussian representation of the block to which the point cloud data of the leaf node belongs; A second Gaussian representation generating module, configured to obtain the Gaussian representation of the sample model according to the Gaussian representation of each block of the sample model; A rendering module, used for rendering the Gaussian representation of the sample model to obtain an output texture image; a loss determination module, configured to determine the loss of the deep learning model according to a difference between the output texture image and a sample texture image, wherein the sample texture image is an image obtained by rendering the sample model; and An adjustment module is used to adjust the parameters of the deep learning model according to the loss.
17. The device according to claim 16, in, The sample texture image includes multiple sample texture images obtained by rendering the sample model at multiple viewing angles; the rendering module includes: A posture determination unit, configured to determine, for each sample texture image, a camera posture corresponding to a viewing angle of the sample texture image; and A rendering unit is used to render the Gaussian representation of the sample model according to the camera posture to obtain an output texture image with the same viewing angle as the sample texture image.
18. The device according to claim 17, in, The loss determination module comprises: a first loss determining unit, configured to determine, for each viewing angle, a loss at the viewing angle according to a difference between the sample texture image and the output texture image at the viewing angle; and The second loss determination unit is used to determine the loss of the deep learning model according to the losses under the multiple perspectives.
19. The device according to claim 18, in, The first loss determination unit includes: a difference determination subunit, for determining pixel differences, style differences, and object contour differences between the sample texture image and the output texture image at the viewing angle; A first loss determination subunit, configured to determine a pixel loss, a style loss, and an object contour loss according to the pixel difference, the style difference, and the object contour difference; The second loss determination subunit is used to determine the pixel loss, style loss and object contour loss according to the object contour difference, and determine the overall loss under the viewing angle.
20. The device according to claim 16, in, The leaf node processing module includes: a feature fusion unit, configured to fuse the guide feature with the point cloud feature of the point cloud data to obtain a fused feature, wherein the guide feature includes at least one of a text feature and an image feature; and A feature processing unit is used to generate a Gaussian representation of the block to which the point cloud data of the leaf node belongs according to the fusion feature.
21. An electronic device, include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 10.
22. A non-transitory computer-readable storage medium storing computer instructions, in, The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 10.
23. A computer program product, comprising a computer program, wherein the computer program is stored on at least one of a readable storage medium and an electronic device, and when the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.
Citation Information
Cited By
Human-computer interaction method based on expression recognition, interactive intelligent toy and robot
CN120088834A
