Texture generation method and device

Abstract

The present disclosure provides a texture generation method and device, which relates to the field of artificial intelligence technology, specifically computer vision, deep learning, augmented reality, virtual reality and other technical fields, and can be applied to scenes such as artificial intelligence content generation. A specific implementation of the method includes: amplifying the noise according to a preset time step as the noisy texture of the initial point cloud; constructing a cube by translating and quantizing the initial point cloud as the root node of the octree; recursively dividing the cube into eight equal parts according to a preset depth as the node of the octree; inputting the octree and the preset time step into the diffusion model based on the octree, denoising the noisy texture of the octree, and obtaining an octree with real texture; reconstructing the point cloud based on the octree with real texture to generate a reconstructed point cloud with real texture. This implementation directly optimizes texture generation in three-dimensional space.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, specifically computer vision, deep learning, augmented reality, virtual reality and other technical fields, and can be applied to scenarios such as artificial intelligence content generation. Background Art

[0002] 3D models are represented by polygonal meshes and are widely used in visual applications such as videos, games, virtual reality scenes, and augmented reality scenes. The quality of texture maps is crucial to the appearance of 3D models in various applications. However, creating high-quality textures usually requires a lot of effort from experienced designers, resulting in a large consumption of time and cost. Currently, progress in using neural networks to create content has brought new vitality and perspectives to the field of computer vision and graphics.

[0003] Currently, the commonly used method for processing texture maps is to use texture representation to convert the three-dimensional texture into a set of two-dimensional images through texture coordinate transformation. Summary of the invention

[0004] The embodiments of the present disclosure provide a texture generation method, apparatus, device, storage medium, and program product.

[0005] In a first aspect, an embodiment of the present disclosure proposes a texture generation method, comprising: amplifying noise according to a preset time step as a noisy texture of an initial point cloud; constructing a cube by translating and quantizing the initial point cloud as a root node of an octree; recursively dividing the cube into eight equal parts according to a preset depth as nodes of an octree, wherein the eight cubes obtained in each division serve as eight child nodes of the corresponding divided cubes; inputting the octree and the preset time step into an octree-based diffusion model, and obtaining an octree with real texture by denoising the noisy texture of the octree; and reconstructing the point cloud based on the octree with real texture to generate a reconstructed point cloud with real texture.

[0006] In a second aspect, an embodiment of the present disclosure proposes a training method for an octree-based diffusion model, including: constructing a sample cube by translating and quantizing a sample point cloud with real texture as the root node of the sample octree; recursively dividing the sample cube into eight equal parts according to a preset depth as the nodes of the sample octree, wherein the eight sample cubes obtained by each division serve as the eight child nodes of the corresponding divided sample cube; amplifying the random noise according to a random time step, and calculating the noisy texture of the sample octree based on the amplified random noise; inputting the sample octree with noisy texture and the random time step into the model to obtain the sample octree with predicted texture; adjusting the parameters of the model based on the difference between the real texture and the predicted texture to obtain the octree-based diffusion model.

[0007] In a third aspect, an embodiment of the present disclosure proposes a texture generation device, comprising: an amplification module, configured to amplify noise according to a preset time step as a noisy texture of an initial point cloud; a construction module, configured to construct a cube by translating and quantizing the initial point cloud, as a root node of an octree; a segmentation module, configured to recursively segment the cube into eight equal parts according to a preset depth, as a node of the octree, wherein the eight cubes obtained in each segmentation serve as eight child nodes of the corresponding segmented cube; a denoising module, configured to input the octree and the preset time step into an octree-based diffusion model, and obtain an octree with a real texture by denoising the noisy texture of the octree; and a reconstruction module, configured to reconstruct the point cloud based on the octree with the real texture, to generate a reconstructed point cloud with the real texture.

[0008] In a fourth aspect, an embodiment of the present disclosure proposes a training device for an octree-based diffusion model, comprising: a construction module, configured to construct a sample cube by translating and quantizing a sample point cloud with a real texture, as a root node of the sample octree; a segmentation module, configured to recursively segment the sample cube into eight equal parts according to a preset depth, as a node of the sample octree, wherein the eight sample cubes obtained by each segmentation serve as eight child nodes of the corresponding segmented sample cube; a calculation module, configured to amplify the random noise according to a random time step, and calculate the noisy texture of the sample octree based on the amplified random noise; a prediction module, configured to input the sample octree with the noisy texture and the random time step into the model, to obtain the sample octree with the predicted texture; an adjustment module, configured to adjust the parameters of the model based on the difference between the real texture and the predicted texture, to obtain a diffusion model based on the octree.

[0009] In a fifth aspect, an embodiment of the present disclosure proposes an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in the first aspect or the second aspect.

[0010] In a sixth aspect, an embodiment of the present disclosure proposes a non-transitory computer-readable storage medium storing computer instructions, where the computer instructions are used to cause a computer to execute the method described in the first aspect or the second aspect.

[0011] In a seventh aspect, an embodiment of the present disclosure proposes a computer program product, including a computer program, which implements the method described in the first aspect or the second aspect when executed by a processor.

[0012] The disclosed embodiment provides an octree-based diffusion method to directly optimize texture generation in three-dimensional space, effectively alleviating texture errors caused by self-occlusion and texture degradation caused by sparse sampling.

[0013] The key or important features of the embodiments of the present disclosure are not intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Other features, purposes and advantages of the present disclosure will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings. The drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure. Among them:

[0015] Figure 1 is a flow chart of an embodiment of a texture generation method according to the present disclosure;

[0016] Figure 2 This is the architecture diagram of the octree-based diffusion model;

[0017] Figure 3 It is the architecture diagram of the residual network block;

[0018] Figure 4 is a flowchart of another embodiment of a texture generation method according to the present disclosure;

[0019] Figure 5 is a flowchart of an embodiment of a training method of an octree-based diffusion model according to the present disclosure;

[0020] Figure 6 is a flowchart of another embodiment of a training method of an octree-based diffusion model according to the present disclosure;

[0021] Figure 7 is a schematic structural diagram of an embodiment of a texture generating device according to the present disclosure;

[0022] Figure 8 is a structural schematic diagram of an embodiment of a training device for an octree-based diffusion model according to the present disclosure;

[0023] Fig. 9 It is a block diagram of an electronic device used to implement the texture generation method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0024] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0025] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0026] Figure 1 A process 100 of an embodiment of a texture generation method according to the present disclosure is shown. The texture generation method comprises the following steps:

[0027] Step 101, amplify the noise according to a preset time step to serve as the noisy texture of the initial point cloud.

[0028] In this embodiment, the execution body of the texture generation method can amplify the noise according to a preset time step to serve as the noisy texture of the initial point cloud.

[0029] Typically, a set of points is sampled from the surface of the mesh of the 3D model using mesh sampling as the initial point cloud. Pure noise is sampled from a preset distribution (e.g., standard normal distribution N(0,1)). The pure noise is amplified using a preset time step (e.g., t=1000) and assigned to each point of the initial point cloud. The initial point cloud can be P=P i ,i=1,2,...,M. Each point can be defined by x, y, z, r, g, b. x, y, z are coordinates, r, g, b are noisy textures, i.e., RGB (Red Green Blue) colors.

[0030] Step 102 , constructing a cube by translating and quantizing the initial point cloud as the root node of the octree.

[0031] In this embodiment, the execution entity may construct a cube as the root node of the octree by translating and quantizing the initial point cloud.

[0032] Usually, the initial point cloud is first translated according to a preset offset, and then the translated initial point cloud is quantized according to a preset quantization step. The bounding box of the translated and quantized initial point cloud is a cube. The preset offset and the preset quantization step can be pre-set according to the requirements.

[0033] In some embodiments, first, the minimum coordinate of the initial point cloud is used as an offset to translate the initial point cloud; then the translated initial point cloud is quantized according to a preset quantization step size; and finally, the bounding box of the translated and quantized initial point cloud is determined as a cube. The difference between the maximum coordinate and the minimum coordinate of the initial point cloud, and the difference between 2 to the power of the preset depth and 1 are calculated. The preset quantization step size is not less than the quotient obtained by dividing the above two differences.

[0034] Among them, the translation and quantization formulas are as follows:

[0035]

[0036] offset=(minP x ,minP y ,minP z );

[0037]

[0038] Among them, P Q It is the initial point cloud after translation and quantization, P is the initial point cloud, offset is the offset, qs is the preset quantization step size, and L is the preset depth.

[0039] Step 103, recursively divide the cube into eight equal parts according to a preset depth to form nodes of an octree.

[0040] In this embodiment, the execution subject can recursively divide the cube into eight equal parts according to the preset depth as nodes of the octree. The cube divided each time can be used as the parent node of the eight cubes obtained by the corresponding division. Accordingly, the eight cubes obtained each time are used as the eight child nodes of the corresponding divided cube.

[0041] Among them, the octree is an efficient data structure that organizes point clouds in a tree format. The octree starts with the initial point cloud as the root node and recursively divides it into eight child nodes, merging points with similar coordinates. The subdivision continues until the preset depth is reached. For example, the octree uses the cube corresponding to the initial point cloud as the root node. In the first round of segmentation, the cube of the root node is divided into 8 equal sub-cubes, which serve as the 8 child nodes of the root node. In the second round of segmentation, the same subdivision is continued for each sub-cube, and the 8 equal sub-cubes are used as the 8 child nodes of the subdivided sub-cube. The segmentation is recursive in this way until the preset depth is reached.

[0042] Step 104 , input the octree and the preset time step into the octree-based diffusion model, and obtain an octree with real texture by denoising the noisy texture of the octree.

[0043] In this embodiment, the execution subject may input the octree and the preset time step into the octree-based diffusion model, and obtain an octree with a real texture by denoising the noisy texture of the octree. The octree-based diffusion model may synthesize a realistic texture from the noisy texture values ​​assigned to each node of the octree.

[0044] In some embodiments, the octree-based diffusion model can be a fully convolutional neural network with ResBlock (Residual Network Block). In combination with the octree-based operators in the octree-based convolutional neural network, a DDPM (Denoising Diffusion Probabilistic Model) process is performed on the octree structure. For example, the octree-based diffusion model can be a U-Net with ResBlock. In combination with the octree-based operators in OCNN (Octree-based Convolutional Neural Network), a DDPM process is performed on the octree structure.

[0045] In some embodiments, the octree-based diffusion model may include multiple stages. Each stage may process features of different tree depths. For example, the octree-based diffusion model may include four stages, and the tree depths of the four stages are 12, 10, 9, and 8, respectively. In this way, the octree-based diffusion model can learn information from different receptive fields of the octree, so that the model can generate details with global consistency.

[0046] In some embodiments, in an octree-based diffusion model, ResBlock may include an octree convolution layer, an octree normalization layer, and an MLP (Multilayer Perceptron). For example, in an octree-based diffusion model, ResBlock may include two octree convolution layers (Oct-conv), two octree normalization layers (Oct-norm), and one MLP.

[0047] In some embodiments, the octree-based diffusion model also supports texture generation under text conditions. For example, when generating the texture of a three-dimensional human body, the color of the three-dimensional human body's clothes, hat color, etc. can be controlled by inputting text. Specifically, in the octree-based diffusion model, ResBlock in at least one stage also includes an octree-based multi-head cross-attention module. The octree-based multi-head cross-attention module can integrate text conditions into the texture generation process, so that the texture generation is guided by the text conditions. In order to enhance memory efficiency, in the first stage, ResBlock can omit the octree-based multi-head cross-attention module.

[0048] Among them, the multi-head cross attention module based on octree is as input and divide it into Patch. Where X is the initial point cloud with noisy texture, N is the number of tree nodes in the batch, T is the patch size, and C is the number of channels. Then, a linear layer is used to calculate the query vector of H heads To compute the key vector K and value vector V from the text features extracted from the CLIP (Contrastive Language-Image Pre-Training) model, a set of two linear layers can be used for each vector. Then, for attention calculation, K and V can be reshaped to match the shape of Q.

[0049] It should be noted that the diffusion model based on the octree can be Figure 5 or Figure 6 The embodiment shown is obtained by training and will not be described in detail here.

[0050] Figure 2 FIG. 4 shows an architecture diagram of an octree-based diffusion model. Figure 2 As shown, a point cloud is sampled from the surface of the mesh of the three-dimensional model. The point cloud is combined with noise to construct an octree. The octree and time step t are input to the octree-based diffusion model to obtain a color texture by reducing the texture noise of the octree. The octree is converted back to the original point cloud using the "Reverse Octree" option to generate a colored point cloud. Finally, the colored point cloud is mapped to the mesh of the three-dimensional model to obtain a textured mesh. Among them, the octree-based diffusion model can be a U-net architecture with ResBlock.

[0051] Figure 3 The architecture diagram of the residual network block is shown in FIG. Figure 3As shown in the figure, each ResBlock consists of two Oct-conv, two Oct-norms and an MLP. In the case of the text conditional diffusion model, a multi-oct-head cross-attention module can also be included after each ResBlock.

[0052] Step 105 , performing point cloud reconstruction based on the octree with real texture to generate a reconstructed point cloud with real texture.

[0053] In this embodiment, the execution entity may perform point cloud reconstruction based on an octree with real textures to generate a reconstructed point cloud with real textures.

[0054] In order to reconstruct the point cloud, an inverse quantization operation can be performed on the octree with the real texture to obtain a reconstructed point cloud with the real texture. The reconstruction error is controlled as follows:

[0055]

[0056] Among them, error is the reconstruction error, is the reconstructed point cloud, P i (i=1,2,...,M) is the initial point cloud, and qs is the quantization step size.

[0057] The disclosed embodiment provides an octree-based diffusion method to directly optimize texture generation in three-dimensional space, effectively alleviating texture errors caused by self-occlusion and texture degradation caused by sparse sampling.

[0058] Continue to refer Figure 4 , which shows a process 400 of another embodiment of a texture generation method according to the present disclosure. The texture generation method comprises the following steps:

[0059] Step 401 : amplify the noise according to a preset time step to serve as the noisy texture of the initial point cloud.

[0060] In this embodiment, the execution body of the texture generation method can amplify the noise according to a preset time step to serve as the noisy texture of the initial point cloud.

[0061] Typically, a set of points is sampled from the surface of the mesh of the 3D model using mesh sampling as the initial point cloud. Pure noise is sampled from a preset distribution (e.g., standard normal distribution N(0,1)). The pure noise is amplified using a preset time step (e.g., t=1000) and assigned to each point of the initial point cloud. The initial point cloud can be P=P i,i=1,2,...,M. Each point can be defined by x, y, z, r, g, b. x, y, z are coordinates, r, g, b are noisy textures, i.e., RGB colors.

[0062] Step 402: Using the minimum coordinate of the initial point cloud as an offset, the initial point cloud is translated.

[0063] In this embodiment, the execution entity may use the minimum coordinate of the initial point cloud as an offset to translate the initial point cloud to obtain a translated initial point cloud.

[0064] Step 403: quantize the translated initial point cloud according to a preset quantization step size.

[0065] In this embodiment, the execution entity may quantize the translated initial point cloud according to a preset quantization step size to obtain the translated and quantized initial point cloud.

[0066] Step 404 , determine the bounding box of the translated and quantized initial point cloud as a cube.

[0067] In this embodiment, the execution entity may determine a bounding box of the initial point cloud after translation and quantization as a cube.

[0068] Among them, the translation and quantization formulas are as follows:

[0069]

[0070] offset=(minP x ,minP y ,minP z );

[0071]

[0072] Among them, P Q It is the initial point cloud after translation and quantization, P is the initial point cloud, offset is the offset, qs is the preset quantization step size, and L is the preset depth.

[0073] Step 405 , divide the cube into eight equal parts to obtain eight sub-cubes as eight sub-nodes of the cube.

[0074] In this embodiment, the execution entity can divide the cube into eight equal parts to obtain eight sub-cubes as eight child nodes of the cube. The cube divided each time can be used as the parent node of the eight sub-cubes obtained by the corresponding division. Accordingly, the eight sub-cubes obtained each time are used as the eight child nodes of the corresponding divided cube.

[0075] Step 406, determining the occupancy status of the eight sub-cubes.

[0076] In this embodiment, the execution entity can determine the occupation status of eight sub-cubes, wherein the sub-cube containing points in the initial point cloud is in an occupied state, and the sub-cube not containing points in the initial point cloud is in an unoccupied state.

[0077] The octree can use an 8-bit binary occupancy code to encode the occupancy status of the 8 sub-cubes, where non-empty sub-cubes are marked as 1 and empty sub-cubes are marked as 0.

[0078] Step 407, determining whether the octree reaches a preset depth.

[0079] In this embodiment, the execution subject can determine whether the octree reaches a preset depth. If the octree reaches the preset depth, step 408 is executed; if the octree does not reach the preset depth, step 409 is executed.

[0080] Step 408, determine whether the octree is established.

[0081] In this embodiment, if the octree reaches a preset depth, the execution entity may determine that the octree is established.

[0082] Step 409: The sub-cube in the occupied state is regarded as a cube.

[0083] In this embodiment, if the octree does not reach the preset depth, the execution subject may use the sub-cube in the occupied state as a cube and return to continue executing step 405 .

[0084] Since the sub-cube in the occupied state contains points in the initial point cloud, and the sub-cube in the unoccupied state does not contain points in the initial point cloud, subdividing only the sub-cube in the occupied state can avoid the workload caused by invalid segmentation.

[0085] The octree can use an 8-bit binary occupancy code to encode the occupancy status of 8 sub-cubes. Among them, non-empty sub-cubes are marked as 1 and further subdivided. Empty sub-cubes are marked as 0. The subdivision process continues until the preset depth is reached. At the leaf node of the octree, each 8-bit occupancy code can represent 8 sub-cubes with a side length equal to the quantization step size. The points contained in the sub-cube corresponding to each leaf node have similar coordinates and can be aligned and merged.

[0086] Step 410: Input the octree and the preset time step into the octree-based diffusion model, and obtain an octree with real texture by denoising the noisy texture of the octree.

[0087] Step 411, reconstructing the point cloud based on the octree with real texture to generate a reconstructed point cloud with real texture.

[0088] In this embodiment, the specific operations of steps 410-411 are already described in Figure 1 In the illustrated embodiment, steps 104-105 are described in detail and will not be repeated here.

[0089] The disclosed embodiment provides an octree-based diffusion method to directly optimize texture generation in three-dimensional space, effectively alleviating texture errors caused by self-occlusion and texture degradation caused by sparse sampling.

[0090] Further references Figure 5 , which shows a process 500 of an embodiment of a method for training a diffusion model based on an octree according to the present disclosure. The method for training a diffusion model based on an octree comprises the following steps:

[0091] Step 501 : construct a sample cube as the root node of the sample octree by translating and quantizing the sample point cloud with real texture.

[0092] In this embodiment, the execution subject of the training method of the octree-based diffusion model can construct a sample cube as the root node of the sample octree by translating and quantizing the sample point cloud with real texture.

[0093] Usually, a set of points is sampled from the surface of the mesh of the sample 3D model with real texture by mesh sampling as a sample point cloud. The sample point cloud can be P = P i ,i=1,2,...,M. Each point can be defined by x, y, z, r, g, b. x, y, z are coordinates, r, g, b are real textures, i.e., RGB colors.

[0094] Usually, the sample point cloud is first translated according to a preset offset, and then the translated sample point cloud is quantized according to a preset quantization step. The bounding box of the translated and quantized sample point cloud is a cube. The preset offset and the preset quantization step can be pre-set according to the requirements.

[0095] In some embodiments, first, the minimum coordinate of the sample point cloud is used as an offset to translate the sample point cloud; then the translated sample point cloud is quantized according to a preset quantization step; and finally, the bounding box of the translated and quantized sample point cloud is determined as a sample cube. The difference between the maximum coordinate and the minimum coordinate of the sample point cloud, and the difference between 2 to the power of the preset depth and 1 are calculated. The preset quantization step is not less than the quotient obtained by dividing the above two differences.

[0096] Among them, the translation and quantization formulas are as follows:

[0097]

[0098] offset=(minP x ,minP y ,minP z );

[0099]

[0100] Among them, P Q It is the sample point cloud after translation and quantization, P is the sample point cloud, offset is the offset, qs is the preset quantization step size, and L is the preset depth.

[0101] Step 502: recursively divide the sample cube into eight equal parts according to a preset depth to form nodes of the sample octree.

[0102] In this embodiment, the execution subject can recursively divide the sample cube into eight equal parts according to the preset depth as nodes of the sample octree. The sample cube divided each time can be used as the parent node of the eight sample cubes obtained by the corresponding division. Accordingly, the eight sample cubes obtained each time are used as the eight child nodes of the corresponding divided sample cube.

[0103] Among them, the octree is an efficient data structure that organizes point clouds in a tree format. The sample octree starts with the sample point cloud as the root node and recursively divides it into eight child nodes, merging points with similar coordinates. Continue to subdivide until the preset depth is reached. For example, the sample octree takes the sample cube corresponding to the sample point cloud as the root node. In the first round of segmentation, the sample cube of the root node is divided into 8 equal sample sub-cubes, which serve as the 8 child nodes of the root node respectively. In the second round of segmentation, the same subdivision is continued for each sample sub-cube, and the 8 equal sample sub-cubes are respectively used as the 8 child nodes of the segmented sample sub-cube. The segmentation is recursively performed in this way until the preset depth is reached.

[0104] Step 503: amplify the random noise according to the random time step, and calculate the noisy texture of the sample octree based on the amplified random noise.

[0105] In this embodiment, the execution subject may amplify the random noise according to a random time step, and calculate the noisy texture of the sample octree based on the amplified random noise.

[0106] Typically, a noisy texture is computed for a tree node by applying an amplified random noise padding to the node's assigned texture value. This process is repeated to complete the noisy texture computation for all nodes of the sample octree during the diffusion process of the training iteration.

[0107] Step 504: input the sample octree with noisy texture and the random time step into the model to obtain the sample octree with predicted texture.

[0108] In this embodiment, the execution subject may input a sample octree with noisy texture and a random time step into the model to obtain a sample octree with predicted texture, wherein the model may synthesize realistic textures from the noisy texture values ​​assigned to each node of the octree.

[0109] Step 505: Based on the difference between the real texture and the predicted texture, the parameters of the model are adjusted to obtain an octree-based diffusion model.

[0110] In this embodiment, the execution entity may adjust the parameters of the model based on the difference between the real texture and the predicted texture to obtain an octree-based diffusion model.

[0111] For example, the loss is calculated based on the sample octree with predicted texture and the sample octree with predicted texture. The parameters of the model are adjusted based on the loss until the loss is relatively small and the model converges, thereby obtaining an octree-based diffusion model.

[0112] In some embodiments, the octree-based diffusion model can be a fully convolutional neural network with ResBlock. In combination with the octree-based operators in the octree-based convolutional neural network, the DDPM process is performed on the octree structure. For example, the octree-based diffusion model can be a U-Net with ResBlock. In combination with the octree-based operators in the OCNN, the DDPM process is performed on the octree structure.

[0113] In some embodiments, the octree-based diffusion model may include multiple stages. Each stage may process features of different tree depths. For example, the octree-based diffusion model may include four stages, and the tree depths of the four stages are 12, 10, 9, and 8, respectively. In this way, the octree-based diffusion model can learn information from different receptive fields of the octree, so that the model can generate details with global consistency.

[0114] In some embodiments, in an octree-based diffusion model, ResBlock may include an octree convolution layer, an octree normalization layer, and an MLP. For example, in an octree-based diffusion model, ResBlock may include two octree convolution layers (Oct-conv), two octree normalization layers (Oct-norm), and one MLP.

[0115] In some embodiments, the octree-based diffusion model also supports texture generation under text conditions. For example, when generating the texture of a three-dimensional human body, the color of the three-dimensional human body's clothes, hat, etc. can be controlled by inputting text. Specifically, in the octree-based diffusion model, the ResBlock in at least one stage also includes an octree-based multi-head cross-attention module. The octree-based multi-head cross-attention module can integrate text conditions into the texture generation process, so that the texture generation is guided by the text conditions.

[0116] Among them, the multi-head cross attention module based on octree is as input and divide it into Patch. Where X is a sample point cloud with noisy texture, N is the number of tree nodes in the batch, T is the patch size, and C is the number of channels. Then, a linear layer is used to calculate the query vector of H heads To compute the key vector K and value vector V from the text features extracted by the CLIP model, a set of two linear layers can be used for each vector. Then, for attention calculation, K and V can be reshaped to match the shape of Q.

[0117] The disclosed embodiment provides an octree-based diffusion method to directly optimize texture generation in three-dimensional space, effectively alleviating texture errors caused by self-occlusion and texture degradation caused by sparse sampling.

[0118] Further references Figure 6 , which shows a process 600 of another embodiment of the training method of the diffusion model based on the octree according to the present disclosure. The training method of the diffusion model based on the octree includes the following steps:

[0119] Step 601: Using the minimum coordinate of the sample point cloud as an offset, the sample point cloud is translated.

[0120] In this embodiment, the training method of the diffusion model based on the octree can use the minimum coordinate of the sample point cloud as an offset to translate the sample point cloud to obtain the translated sample point cloud.

[0121] Step 602: quantize the translated sample point cloud according to a preset quantization step size.

[0122] In this embodiment, the execution subject may quantize the translated sample point cloud according to a preset quantization step size to obtain the translated and quantized sample point cloud.

[0123] Step 603 : determine the bounding box of the translated and quantized sample point cloud as a sample cube.

[0124] In this embodiment, the execution entity may determine a bounding box of the sample point cloud after translation and quantization as a sample cube.

[0125] Among them, the translation and quantization formulas are as follows:

[0126]

[0127] offset=(minP x ,minP y ,minP z );

[0128]

[0129] Among them, P Q It is the sample point cloud after translation and quantization, P is the sample point cloud, offset is the offset, qs is the preset quantization step size, and L is the preset depth.

[0130] Step 604 , divide the sample cube into eight equal parts to obtain eight sample sub-cubes as eight sub-nodes of the sample cube.

[0131] In this embodiment, the execution entity can divide the sample cube into eight equal parts to obtain eight sample sub-cubes as eight child nodes of the sample cube. The sample cube divided each time can be used as the parent node of the eight sample sub-cubes obtained by the corresponding division. Accordingly, the eight sample sub-cubes obtained each time are used as the eight child nodes of the corresponding divided sample cube.

[0132] Step 605: determine the occupancy status of eight sample sub-cubes.

[0133] In this embodiment, the execution subject can determine the occupation status of eight sample sub-cubes, wherein the sample sub-cube containing points in the sample point cloud is in an occupied state, and the sample sub-cube not containing points in the sample point cloud is in an unoccupied state.

[0134] The sample octree can use an 8-bit binary occupancy code to encode the occupancy status of the 8 sample sub-cubes, where a non-empty sample sub-cube is marked as 1 and an empty sample sub-cube is marked as 0.

[0135] Step 606, determining whether the sample octree reaches a preset depth.

[0136] In this embodiment, the execution subject may determine whether the sample octree reaches a preset depth. If the sample octree reaches the preset depth, step 607 is executed; if the sample octree does not reach the preset depth, step 608 is executed.

[0137] Step 607, determine whether the sample octree is established.

[0138] In this embodiment, if the sample octree reaches a preset depth, the execution entity may determine that the sample octree is established.

[0139] Step 608: Use the sample sub-cube in the occupied state as a sample cube.

[0140] In this embodiment, if the sample octree does not reach the preset depth, the execution subject may use the sample sub-cube in the occupied state as the sample cube and return to continue executing step 604 .

[0141] Since the sample sub-cube in the occupied state contains points in the sample point cloud, and the sample sub-cube in the unoccupied state does not contain points in the sample point cloud, subdividing only the sample sub-cube in the occupied state can avoid the workload caused by invalid segmentation.

[0142] The sample octree can use an 8-bit binary occupancy code to encode the occupancy status of 8 sample sub-cubes. Among them, non-empty sample sub-cubes are marked as 1 and further subdivided. Empty sample sub-cubes are marked as 0. The subdivision process continues until the preset depth is reached. At the leaf node of the sample octree, each 8-bit occupancy code can represent 8 sample sub-cubes with a side length equal to the quantization step size. The points contained in the sample sub-cube corresponding to each leaf node have similar coordinates and can be aligned and merged.

[0143] Step 609 , amplify the random noise according to the random time step, and calculate the noisy texture of the sample octree based on the amplified random noise.

[0144] Step 610: Input the sample octree with noisy texture and the random time step into the model to obtain the sample octree with predicted texture.

[0145] In this embodiment, the specific operations of steps 609-610 are already described in Figure 5 In the illustrated embodiment, steps 503 - 504 are described in detail and will not be repeated here.

[0146] Step 611, performing point cloud reconstruction based on the sample octree with predicted texture to generate a reconstructed point cloud with predicted texture.

[0147] In this embodiment, the execution entity may perform point cloud reconstruction based on the sample octree with predicted texture to generate a reconstructed point cloud with predicted texture.

[0148] In order to reconstruct the point cloud, an inverse quantization operation can be performed on the sample octree with the predicted texture to obtain a reconstructed point cloud with the predicted texture. The reconstruction error is controlled as follows:

[0149]

[0150] Among them, error is the reconstruction error, is the reconstructed point cloud, P i (i=1,2,...,M) is the sample point cloud, and qs is the quantization step size.

[0151] Step 612: Based on the difference between the sample point cloud with real texture and the reconstructed point cloud with predicted texture, the parameters of the model are adjusted to obtain an octree-based diffusion model.

[0152] In this embodiment, the execution entity may adjust the parameters of the model based on the difference between the sample point cloud with real texture and the reconstructed point cloud with predicted texture to obtain an octree-based diffusion model.

[0153] For example, based on the sample point cloud with real texture and the reconstructed point cloud with predicted texture, the loss is calculated. Based on the loss, the model parameters are adjusted until the loss is relatively small and the model converges, and the diffusion model based on the octree is obtained.

[0154] The disclosed embodiment provides an octree-based diffusion method to directly optimize texture generation in three-dimensional space, effectively alleviating texture errors caused by self-occlusion and texture degradation caused by sparse sampling.

[0155] Further references Figure 7 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a texture generation device. Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0156] like Figure 7As shown, the texture generation device 700 of this embodiment may include: an amplification module 701, a construction module 702, a segmentation module 703, a denoising module 704 and a reconstruction module 705. The amplification module 701 is configured to amplify the noise according to a preset time step as the noisy texture of the initial point cloud; the construction module 702 is configured to construct a cube by translating and quantizing the initial point cloud as the root node of the octree; the segmentation module 703 is configured to recursively segment the cube into eight equal parts according to a preset depth as the node of the octree, wherein the eight cubes obtained by each segmentation are eight child nodes of the corresponding segmented cube; the denoising module 704 is configured to input the octree and the preset time step into the diffusion model based on the octree, and obtain the octree with real texture by denoising the noisy texture of the octree; the reconstruction module 705 is configured to reconstruct the point cloud based on the octree with real texture to generate a reconstructed point cloud with real texture.

[0157] In this embodiment, the specific processing of the amplification module 701, the construction module 702, the segmentation module 703, the denoising module 704 and the reconstruction module 705 and the technical effects thereof can be referred to in detail. Figure 1 The relevant descriptions of steps 101-105 in the corresponding embodiment are not repeated here.

[0158] In some optional implementations of the present embodiment, the construction module 702 is further configured to: translate the initial point cloud using the minimum coordinate of the initial point cloud as an offset; quantize the translated initial point cloud according to a preset quantization step size, wherein the preset quantization step size is not less than the difference between the maximum coordinate and the minimum coordinate of the initial point cloud as the dividend, and the difference between 2 raised to the power of a preset depth and 1 as the divisor, and the calculated quotient; determine the bounding box of the translated and quantized initial point cloud as a cube.

[0159] In some optional implementations of the present embodiment, the segmentation module 703 is further configured to: perform the following segmentation steps: divide the cube into eight equal parts to obtain eight sub-cubes as eight sub-nodes of the cube; determine the occupancy status of the eight sub-cubes, wherein the sub-cube containing the points in the initial point cloud is in an occupied state, and the sub-cube not containing the points in the initial point cloud is in an unoccupied state; determine whether the octree reaches a preset depth; if the octree reaches the preset depth, determine that the octree is established; in response to the octree not reaching the preset depth, treat the sub-cube in the occupied state as a cube, and continue to perform the segmentation steps.

[0160] In some optional implementations of this embodiment, the octree-based diffusion model is a fully convolutional neural network with a residual network block, combined with an octree-based operator in the octree-based convolutional neural network, to perform a denoising diffusion probability model process on the octree structure.

[0161] In some optional implementations of this embodiment, the octree-based diffusion model includes multiple stages, each stage processing features of different tree depths.

[0162] In some optional implementations of this embodiment, the residual network block includes an octree convolution layer, an octree normalization layer and a multi-layer perceptron.

[0163] In some optional implementations of this embodiment, the residual network block in at least one stage also includes an octree-based multi-head cross-attention module, which integrates text conditions into the texture generation process.

[0164] Further references Figure 8 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a training device for a diffusion model based on an octree. Figure 5 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0165] like Figure 8 As shown, the octree-based diffusion model training device 800 of this embodiment may include: a construction module 801, a segmentation module 802, a calculation module 803, a prediction module 804 and an adjustment module 805. Among them, the construction module 801 is configured to construct a sample cube by translating and quantizing the sample point cloud with real texture, as the root node of the sample octree; the segmentation module 802 is configured to recursively segment the sample cube into eight equal parts according to a preset depth, as the node of the sample octree, wherein the eight sample cubes obtained by each segmentation serve as the eight child nodes of the corresponding segmented sample cube; the calculation module 803 is configured to amplify the random noise according to a random time step, and calculate the noisy texture of the sample octree based on the amplified random noise; the prediction module 804 is configured to input the sample octree with noisy texture and the random time step into the model to obtain the sample octree with predicted texture; the adjustment module 805 is configured to adjust the parameters of the model based on the difference between the real texture and the predicted texture to obtain an octree-based diffusion model.

[0166] In the present embodiment, in the training device 800 based on the diffusion model of the octree, the specific processing of the construction module 801, the segmentation module 802, the calculation module 803, the prediction module 804 and the adjustment module 805 and the technical effects thereof can be referred to respectively. Figure 5 The relevant descriptions of steps 501-505 in the corresponding embodiment are not repeated here.

[0167] In some optional implementations of this embodiment, the construction module 801 is further configured to: translate the sample point cloud using the minimum coordinate of the sample point cloud as an offset; quantize the translated sample point cloud according to a preset quantization step size, wherein the preset quantization step size is not less than the difference between the maximum coordinate and the minimum coordinate of the sample point cloud as the dividend, and the difference between 2 raised to the power of a preset depth and 1 as the divisor, and the quotient calculated; determine the bounding box of the translated and quantized sample point cloud as a sample cube.

[0168] In some optional implementations of the present embodiment, the segmentation module 802 is further configured to: perform the following segmentation steps: divide the sample cube into eight equal parts to obtain eight sample sub-cubes as eight child nodes of the sample cube; determine the occupancy status of the eight sample sub-cubes, wherein the sample sub-cube containing points in the sample point cloud is in an occupied state, and the sample sub-cube not containing points in the sample point cloud is in an unoccupied state; determine whether the sample octree reaches a preset depth; if the sample octree reaches the preset depth, determine that the sample octree is established; in response to the sample octree not reaching the preset depth, use the sample sub-cube in the occupied state as the sample cube, and continue to perform the segmentation steps.

[0169] In some optional implementations of this embodiment, the octree-based diffusion model is a fully convolutional neural network with a residual network block, combined with an octree-based operator in the octree-based convolutional neural network, to perform a denoising diffusion probability model process on the octree structure.

[0170] In some optional implementations of this embodiment, the octree-based diffusion model includes multiple stages, each stage processing features of different tree depths.

[0171] In some optional implementations of this embodiment, the residual network block includes an octree convolution layer, an octree normalization layer and a multi-layer perceptron.

[0172] In some optional implementations of this embodiment, the residual network block in at least one stage also includes an octree-based multi-head cross-attention module, which integrates text conditions into the texture generation process.

[0173] In some optional implementations of the present embodiment, the adjustment module 805 is further configured to: reconstruct the point cloud based on the sample octree with predicted texture to generate a reconstructed point cloud with predicted texture; adjust the model parameters based on the difference between the sample point cloud with real texture and the reconstructed point cloud with predicted texture to obtain an octree-based diffusion model.

[0174] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0175] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0176] Fig. 9 A schematic block diagram of an example electronic device 900 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0177] like Fig. 9 As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0178] A number of components in the device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0179] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as the texture generation method. For example, in some embodiments, the texture generation method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the texture generation method described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to perform the texture generation method in any other appropriate manner (e.g., by means of firmware).

[0180] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0181] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0182] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0183] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0184] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0185] A computer system may include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises through computer programs running on respective computers and having a client-server relationship to each other. The server may be a cloud server, a server in a distributed system, or a server combined with a blockchain.

[0186] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions provided by this disclosure can be achieved, and this document does not limit this.

[0187] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A texture generation method, comprising: Amplify the noise according to the preset time step as the noisy texture of the initial point cloud; Constructing a cube by translating and quantizing the initial point cloud as the root node of the octree; Recursively dividing the cube into eight equal parts according to a preset depth as nodes of the octree, wherein eight cubes obtained from each division serve as eight child nodes of the corresponding divided cube; The octree and the preset time step are input into an octree-based diffusion model, and an octree with a real texture is obtained by denoising the noisy texture of the octree, wherein the octree-based diffusion model is a fully convolutional neural network with a residual network block, and the denoising diffusion probability model process is performed on the octree structure in combination with the octree-based operator in the octree-based convolutional neural network, and the residual network block includes an octree-based multi-head cross-attention module, and the octree-based multi-head cross-attention module integrates text conditions into the texture generation process, and the octree-based multi-head cross-attention module divides the initial point cloud into patches, calculates a query vector using a linear layer, and calculates a key vector and a value vector from text features extracted from a contrastive language image pre-training model, and the key vector and the value vector are reshaped to match the shape of the query vector; Point cloud reconstruction is performed based on the octree with real texture to generate the reconstructed point cloud with real texture.

2. The method according to claim 1, wherein: The step of constructing a cube by translating and quantizing the initial point cloud comprises: Using the minimum coordinate of the initial point cloud as an offset, translating the initial point cloud; Quantize the initial point cloud after translation according to the preset quantization step size; Determine the bounding box of the translated and quantized initial point cloud as the cube.

3. The method according to claim 1, wherein: The recursive division of the cube into eight equal parts according to a preset depth as nodes of the octree includes: Perform the following segmentation steps: segment the cube into eight equal parts to obtain eight sub-cubes as eight sub-nodes of the cube; determine the occupation status of the eight sub-cubes, wherein the sub-cubes containing points in the initial point cloud are in an occupied state, and the sub-cubes not containing points in the initial point cloud are in an unoccupied state; determine whether the octree reaches the preset depth; if the octree reaches the preset depth, determine that the octree is established; In response to the octree not reaching the preset depth, the sub-cube in the occupied state is taken as a cube, and the segmentation step is continued.

4. The method according to claim 1, wherein: The octree-based diffusion model includes multiple stages, each stage processing features of different tree depths.

5. The method according to claim 4, wherein: The residual network block includes an octree convolution layer, an octree normalization layer and a multi-layer perceptron.

6. A method for training a diffusion model based on an octree, comprising: The sample point cloud with real texture is translated and quantized to construct a sample cube as the root node of the sample octree; Recursively dividing the sample cube into eight equal parts according to a preset depth to serve as nodes of the sample octree, wherein eight sample cubes obtained from each division serve as eight child nodes of the corresponding divided sample cube; Amplifying the random noise according to a random time step, and calculating a noisy texture of the sample octree based on the amplified random noise; Inputting the sample octree with noisy texture and the random time step into the model to obtain a sample octree with predicted texture; Based on the difference between the real texture and the predicted texture, the parameters of the model are adjusted to obtain an octree-based diffusion model, wherein the octree-based diffusion model is a fully convolutional neural network with a residual network block, combined with the octree-based operator in the octree-based convolutional neural network, a denoising diffusion probability model process is performed on the octree structure, the residual network block includes an octree-based multi-head cross-attention module, and the octree-based multi-head cross-attention module integrates text conditions into the texture generation process.

7. The method according to claim 6, wherein: The sample point cloud with real texture is translated and quantized to construct a sample cube as the root node of the sample octree, including: Taking the minimum coordinate of the sample point cloud as an offset, translating the sample point cloud; Quantize the translated sample point cloud according to the preset quantization step size; A bounding box of the translated and quantized sample point cloud is determined as the sample cube.

8. The method according to claim 6, wherein: The recursive division of the sample cube into eight equal parts according to a preset depth as nodes of the sample octree includes: Perform the following segmentation steps: segment the sample cube into eight equal parts to obtain eight sample sub-cubes as eight sub-nodes of the sample cube; determine the occupancy status of the eight sample sub-cubes, wherein the sample sub-cubes containing points in the sample point cloud are in an occupied state, and the sample sub-cubes not containing points in the sample point cloud are in an unoccupied state; determine whether the sample octree reaches the preset depth; if the sample octree reaches the preset depth, determine that the sample octree is established; In response to the sample octree not reaching the preset depth, the sample sub-cube in the occupied state is used as the sample cube, and the segmentation step is continued.

9. The method according to claim 6, wherein: The octree-based diffusion model includes multiple stages, each stage processing features of different tree depths.

10. The method according to claim 9, wherein: The residual network block includes an octree convolution layer, an octree normalization layer and a multi-layer perceptron.

11. The method according to claim 6, wherein: The adjusting the parameters of the model based on the difference between the real texture and the predicted texture to obtain the diffusion model based on the octree includes: Reconstructing a point cloud based on the sample octree with the predicted texture to generate the reconstructed point cloud with the predicted texture; Based on the difference between the sample point cloud with real texture and the reconstructed point cloud with predicted texture, the parameters of the model are adjusted to obtain the octree-based diffusion model.

12. A texture generating device, comprising: an amplification module, configured to amplify the noise according to a preset time step as a noisy texture of the initial point cloud; A construction module, configured to construct a cube as a root node of an octree by translating and quantizing the initial point cloud; A segmentation module is configured to recursively segment the cube into eight equal parts according to a preset depth as nodes of the octree, wherein eight cubes obtained from each segmentation serve as eight child nodes of the corresponding segmented cube; a denoising module configured to input the octree and the preset time step into an octree-based diffusion model, and obtain an octree with a real texture by denoising the noisy texture of the octree, wherein the octree-based diffusion model is a fully convolutional neural network with a residual network block, and the denoising diffusion probability model process is performed on the octree structure in combination with the octree-based operator in the octree-based convolutional neural network, the residual network block includes an octree-based multi-head cross-attention module, the octree-based multi-head cross-attention module integrates text conditions into the texture generation process, the octree-based multi-head cross-attention module divides the initial point cloud into patches, calculates a query vector using a linear layer, calculates a key vector and a value vector from text features extracted from a contrastive language image pre-training model, and the key vector and the value vector are reshaped to match the shape of the query vector; The reconstruction module is configured to perform point cloud reconstruction based on the octree with real texture to generate the reconstructed point cloud with real texture.

13. The device according to claim 12, wherein: The construction module is further configured to: Using the minimum coordinate of the initial point cloud as an offset, translating the initial point cloud; Quantize the initial point cloud after translation according to the preset quantization step size; Determine the bounding box of the translated and quantized initial point cloud as the cube.

14. The device according to claim 12, wherein: The segmentation module is further configured to: Perform the following segmentation steps: segment the cube into eight equal parts to obtain eight sub-cubes as eight sub-nodes of the cube; determine the occupation status of the eight sub-cubes, wherein the sub-cubes containing points in the initial point cloud are in an occupied state, and the sub-cubes not containing points in the initial point cloud are in an unoccupied state; determine whether the octree reaches the preset depth; if the octree reaches the preset depth, determine that the octree is established; In response to the octree not reaching the preset depth, the sub-cube in the occupied state is taken as a cube, and the segmentation step is continued.

15. The device according to claim 12, wherein: The octree-based diffusion model includes multiple stages, each stage processing features of different tree depths.

16. The device according to claim 15, wherein: The residual network block includes an octree convolution layer, an octree normalization layer and a multi-layer perceptron.

17. A training device for a diffusion model based on an octree, comprising: A construction module, configured to construct a sample cube as a root node of a sample octree by translating and quantizing a sample point cloud with a real texture; a segmentation module configured to recursively segment the sample cube into eight equal parts according to a preset depth as nodes of the sample octree, wherein eight sample cubes obtained from each segmentation serve as eight child nodes corresponding to the segmented sample cube; A calculation module is configured to amplify the random noise according to a random time step, and calculate the noisy texture of the sample octree based on the amplified random noise; A prediction module, configured to input the sample octree with noisy texture and the random time step into a model to obtain a sample octree with predicted texture; An adjustment module is configured to adjust the parameters of the model based on the difference between the real texture and the predicted texture to obtain an octree-based diffusion model, wherein the octree-based diffusion model is a fully convolutional neural network with a residual network block, combined with an octree-based operator in the octree-based convolutional neural network, and a denoising diffusion probability model process is performed on the octree structure, the residual network block includes an octree-based multi-head cross-attention module, and the octree-based multi-head cross-attention module integrates text conditions into the texture generation process.

18. The device according to claim 17, wherein: The construction module is further configured to: Taking the minimum coordinate of the sample point cloud as an offset, translating the sample point cloud; Quantize the translated sample point cloud according to the preset quantization step size; A bounding box of the translated and quantized sample point cloud is determined as the sample cube.

19. The device according to claim 17, wherein: The segmentation module is further configured to: Perform the following segmentation steps: segment the sample cube into eight equal parts to obtain eight sample sub-cubes as eight sub-nodes of the sample cube; determine the occupancy status of the eight sample sub-cubes, wherein the sample sub-cubes containing points in the sample point cloud are in an occupied state, and the sample sub-cubes not containing points in the sample point cloud are in an unoccupied state; determine whether the sample octree reaches the preset depth; if the sample octree reaches the preset depth, determine that the sample octree is established; In response to the sample octree not reaching the preset depth, the sample sub-cube in the occupied state is used as the sample cube, and the segmentation step is continued.

20. The device according to claim 17, wherein: The octree-based diffusion model includes multiple stages, each stage processing features of different tree depths.

21. The device according to claim 20, wherein: The residual network block includes an octree convolution layer, an octree normalization layer and a multi-layer perceptron.

22. The device according to claim 17, wherein: The adjustment module is further configured to: Reconstructing a point cloud based on the sample octree with the predicted texture to generate the reconstructed point cloud with the predicted texture; Based on the difference between the sample point cloud with real texture and the reconstructed point cloud with predicted texture, the parameters of the model are adjusted to obtain the octree-based diffusion model.

23. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-5 or 6-11.

24. A non-transitory computer-readable storage medium storing computer instructions, the computer instructions being used to cause the computer to execute the method of any one of claims 1-5 or 6-11.

25. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1-5 or 6-11.

Images