A 3D object generation method based on diffusion model and semantic guidance

Abstract

This invention discloses a method for generating three-dimensional objects based on a diffusion model and semantic guidance. The method comprises the following steps: generating a semantic vector from text data using a text encoder based on a CLIP model; generating a shape vector from the semantic vector and a first random noise using a conditional flow model, and concatenating the shape vector with a time step vector as a guiding condition; using a second random noise as an initial inverse diffusion vector, generating a low-dimensional point cloud vector through inverse diffusion based on the guiding condition and the initial inverse diffusion vector using a diffusion model; decoding the low-dimensional point cloud vector using a point cloud decoder to obtain a high-dimensional point cloud, and generating a three-dimensional object based on the high-dimensional point cloud. This method can generate higher-quality three-dimensional objects, but still suffers from drawbacks such as slow training and inference times, and the ability to generate only a single type of object.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to a three-dimensional point cloud generation method based on a diffusion model and semantic guidance. Background Art

[0002] With the development of computer technology and graphics technology, three-dimensional objects have been widely used in various fields. For example, games, film and television, animation and other fields use three-dimensional models as their basic resources to build virtual worlds; industry, architecture and other fields use three-dimensional objects to accelerate or optimize their product design; medicine, biochemistry and other fields use three-dimensional objects to represent precise models; social, media and other fields use three-dimensional objects to create richer entertainment products and information dissemination tools, etc.

[0003] At present, the generation of three-dimensional objects mainly relies on professional modelers or designers to manually produce them using professional modeling tools such as 3dsMax and Maya.

[0004] With the rapid development of deep learning technology, people can use deep learning models to obtain the desired three-dimensional objects. Three-dimensional object generation methods based on deep learning can be roughly divided into two categories. One is the three-dimensional reconstruction method for specific objects and the method based on generative models. The three-dimensional reconstruction method for specific objects takes the information of a single object or scene (such as RGB-D images) as input and outputs the corresponding three-dimensional representation object, which is a one-to-one mapping; the other is based on a generative model. The model learns the data distribution from a large number of three-dimensional object samples to generate new sample instances. This method is often used to generate a large number of diverse three-dimensional objects.

[0005] The flow model is a generative model. Generative networks based on the flow model are widely used in various fields such as image generation and audio and video synthesis. Other generative models, such as variational autoencoders and generative adversarial networks, use parameterized models to learn the latent space distribution of data, so it is impossible to calculate an exact likelihood function to optimize model training. The flow model uses a series of reversible transformation functions to learn the transformation process between two distributions to fit specific distribution changes, which can achieve good generation results. The flow model starts with a standard normal distribution and uses a series of reversible transformation functions to convert the original normal distribution into the probability distribution of the data we want.

[0006] The diffusion model is a generative model widely used in various generative tasks. Its core idea is to continuously add Gaussian noise to real samples during the training phase and use a noise prediction network to predict this noise. During the inference phase, random noise is input to the noise prediction network, which then converts it into a sample.

[0007] At present, some works apply diffusion models to the task of three-dimensional object generation. Compared with non-diffusion model methods, they can generate higher-quality three-dimensional objects, but they still have defects such as slow training, slow inference, and only being able to generate objects of a single category. Summary of the Invention

[0008] In view of the above, the object of the present invention is to provide a three-dimensional object generation method based on a diffusion model and semantic guidance, so as to improve the efficiency and quality of three-dimensional object generation and enrich the diversity of generated objects.

[0009] To achieve the above-mentioned object of the invention, an embodiment provides a method for generating a three-dimensional object based on a diffusion model and semantic guidance, comprising the following steps:

[0010] Use the text encoder of the CLIP model to generate semantic vectors based on text data;

[0011] A conditional flow model is used to generate a shape vector based on the semantic vector and the first random noise, and the shape vector is concatenated with the time step vector as a guiding condition;

[0012] The second random noise is used as the initial reverse diffusion vector, and a low-dimensional point cloud vector is generated by reverse diffusion based on the guidance condition and the initial reverse diffusion vector using a diffusion model;

[0013] The point cloud decoder is used to decode the low-dimensional point cloud vector to obtain a high-dimensional point cloud, and a three-dimensional object is generated based on the high-dimensional point cloud.

[0014] In one embodiment, the conditional flow model and the diffusion model need to undergo parameter optimization before being applied, including the following steps:

[0015] Build samples: prepare high-dimensional point clouds and render them to obtain object images;

[0016] Constructing a training system: The training system includes a shape encoder, a CLIP model, a point cloud encoder, a conditional flow model, and a diffusion model. The high-dimensional point cloud is encoded by the shape encoder and the point cloud encoder at the same time to obtain a shape vector and a low-dimensional point cloud vector. The object image is encoded by the image encoder of the CLIP model to obtain a semantic vector. The shape vector and the semantic vector are generated by the conditional flow model to generate a third random noise. The low-dimensional point cloud vector is used as the initial forward diffusion vector of the forward diffusion process of the diffusion model. The forward diffusion is achieved by adding actual noise at each diffusion step and the forward diffusion vector of each diffusion step is obtained. The fourth random noise randomly sampled from the Gaussian distribution is used as the initial reverse diffusion vector of the reverse diffusion process of the diffusion model. The concatenation of the shape vector and the time step vector is used as the guiding condition. For each step of reverse diffusion, the cumulative noise is calculated based on the guiding condition and the reverse diffusion vector of the previous diffusion step, and the reverse diffusion vector of each diffusion step is calculated based on the cumulative noise.

[0017] Constructing the loss function: The difference between the third random noise and the preset noise label is used as the supervised loss function of the conditional flow model, and the difference between the actual noise added by the forward diffusion process and the cumulative noise calculated by the reverse diffusion process in the same diffusion step is used as the supervised loss function of the diffusion model;

[0018] System training: The training system is trained using loss functions and samples to optimize the parameters of the conditional flow model and the diffusion model.

[0019] In one embodiment, the point cloud encoder and point cloud decoder need to undergo parameter optimization before being used, including the following steps:

[0020] The real high-dimensional point cloud is encoded by the point cloud encoder to obtain a low-dimensional point cloud vector. The low-dimensional point cloud vector and the random sphere point cloud are decoded by the point cloud decoder to obtain a reconstructed high-dimensional point cloud. The parameters of the point cloud encoder and point cloud decoder are optimized by calculating the difference between the real high-dimensional point cloud and the reconstructed high-dimensional point cloud.

[0021] In one embodiment, during the inverse diffusion process of the diffusion model, the cumulative noise is predicted based on the guide condition and the inverse diffusion vector of the previous diffusion step through a noise prediction network. The noise prediction network includes at least two noise prediction units, each noise prediction unit includes a feature fusion module, a feature extraction module, and a feature propagation module. The guide condition and the inverse diffusion vector of the previous diffusion step are fused into a splicing feature through the feature fusion module. The splicing feature is extracted by the point cloud branch, sampling branch, and voxel branch of the feature extraction module to extract three modal features respectively. The features calculated by the feature propagation module from the three modal features are used as the cumulative noise.

[0022] In one embodiment, the feature fusion module includes three linear layers. The inverse diffusion vector of the previous diffusion step passes through the first linear layer and is dot-multiplied with the result of the guide condition passing through the second linear layer to obtain a fused feature. The results of the guide condition passing through the second linear layer and the third linear layer in sequence are spliced ​​with the fusion vector to obtain the spliced ​​feature.

[0023] In one embodiment, in the feature extraction module, the splicing feature is subjected to the first MLP in the point cloud branch to obtain the second modal feature, the splicing feature is subjected to the voxelization operation, the second MLP and the devoxelization operation in the voxel branch to obtain the second modal feature, and the splicing feature is sampled in the sampling branch and spliced ​​with the second modal feature to obtain the third modal feature.

[0024] In one embodiment, the feature propagation module includes an upsampling layer, a splicing layer, a third MLP and a PVConv layer. After being sampled in the upsampling layer, the second modal feature and the third modal feature are spliced ​​with the second modal feature and the first modal feature in the splicing layer. The splicing result is sequentially calculated through the third MLP and PVConv layer to obtain a feature as the cumulative noise.

[0025] In one embodiment, the PVConv layer includes a point cloud branch and a voxel branch. The feature output by the third MLP is subjected to a first MLP in the point cloud branch to obtain a feature. The feature output by the third MLP is subjected to a voxelization operation, a second MLP, and a devoxelization operation in the voxel branch to obtain another feature. The features output by the two branches are spliced ​​to obtain the accumulated noise.

[0026] In one embodiment, the time step vector is obtained by embedding a uniform parameter in [0, 1].

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] Taking text data as the basis, the text encoder of the CLIP model is used to generate semantic vectors corresponding to the text data, and the conditional flow model is used to generate shape vectors based on the semantic vectors to construct guidance conditions. Then, based on the guidance conditions, the inverse diffusion of the diffusion model is used to generate low-dimensional point cloud vectors. Finally, based on the decoding of the low-dimensional point cloud vectors, a high-dimensional point cloud for generating three-dimensional objects is constructed. This process has a fast inference speed, and since the guidance conditions are constructed based on semantic text data, multiple types of three-dimensional objects can be generated, enriching the diversity of three-dimensional object generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0030] Figure 1 is a flow chart of a method for generating a three-dimensional object based on a diffusion model and semantic guidance provided by an embodiment;

[0031] Figure 2 is a schematic structural diagram of a training system provided in an embodiment;

[0032] Figure 3 is a schematic structural diagram of a shape encoder provided in an embodiment;

[0033] Figure 4Schematic diagram of the conditional flow model structure and forward calculation process provided by the embodiment;

[0034] Figure 5 is a schematic structural diagram of a point cloud encoder provided in an embodiment;

[0035] Figure 6 is a schematic structural diagram of a point cloud decoder provided in an embodiment;

[0036] Figure 7 is a schematic diagram of the structure of a noise prediction network provided in an embodiment;

[0037] Figure 8 is a structural diagram of a feature fusion module provided in an embodiment;

[0038] Figure 9 is a structural diagram of a feature extraction module provided in an embodiment;

[0039] Figure 10 is a schematic structural diagram of a feature propagation module provided in an embodiment;

[0040] Figure 11 Schematic diagram of the structure of the PVConv layer provided in the embodiment;

[0041] Figure 12 This is a flowchart of generating a three-dimensional object using a conditional flow model and a diffusion model provided by an embodiment. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0043] The embodiment provides a three-dimensional object generation method based on diffusion model and semantic guidance, such as Figure 1 As shown, the following steps are included:

[0044] Step 1: Construct training samples.

[0045] In an embodiment, a high-dimensional point cloud representing shape and appearance is prepared, and the high-dimensional point cloud is rendered to obtain an object image, wherein the high-dimensional point cloud and the corresponding object image serve as a training sample.

[0046] Step 2: Build a training system that includes a shape encoder, a CLIP model, a point cloud encoder, a conditional flow model, and a diffusion model.

[0047] In the embodiment, Figure 2As shown, the training system includes a shape encoder, a CLIP model, a point cloud encoder, a conditional flow model, and a diffusion model training system. The shape encoder is used to encode high-dimensional point clouds into shape vectors. The shape encoder adopts a multi-layer perceptron structure. Figure 3 As shown in the figure, the input high-dimensional point cloud is encoded to output features μ and σ, and the formula z=μ+∈*exp(0.5*log(σ 2 ))Get the shape vector z, where ∈ represents a randomly generated offset value, Figure 3 In the figure, Conv1d represents a one-dimensional convolutional layer, BatchNorm1d represents a one-dimensional batch processing, ReLU represents a ReLU activation function, MaxPooling represents a maximum pooling operation, and Linear represents a linear layer.

[0048] The CLIP model is trained on hundreds of millions of text-image pairs through contrastive learning. The model can map images and texts describing the same thing into similar feature vectors, including image encoders and text encoders, such as Figure 3 and 12 As shown, in the training phase, the semantic vector c of the object image is extracted through the image encoder of the CLIP model. In the inference phase, the semantic vector c of the text data is extracted through the text encoder of the CLIP model. The semantic vector is used as the semantic conditional vector of the guided diffusion model.

[0049] The conditional flow model is used to find a reversible nonlinear transformation. During the training phase, it can estimate the third random Gaussian noise g based on the semantic vector c as a condition and the shape vector z with the number of channels D. During the inference phase, it generates the shape vector z based on the input first random Gaussian noise and the semantic vector c as a condition. Figure 4 An example of a conditional flow model is given, and the forward calculation process of the conditional flow model is also as follows Figure 4 As shown, z:d represents the first d dimensions of the shape vector z, zd+1: represents the Dd dimensions starting from d+1 in the shape vector z, and scale and shift are both a 3-layer MLP that projects the concatenation of the input z and c back to d dimensions. The forward calculation process is a reversible process. By calculating its inverse process, we can use the first random Gaussian noise g in the inference phase to pass through the reversible process f. -1 Get the shape vector z, that is, g = f -1 (z).

[0050] The diffusion model is very slow during inference, and this phenomenon will be very obvious for three-dimensional point cloud data, especially dense point cloud data. Therefore, the embodiment uses a diffusion model based on a hidden layer, and thus introduces a point cloud encoder and a point cloud decoder to obtain a low-dimensional point cloud vector in the latent space as the input of the diffusion model. At the same time, a point cloud decoder is introduced to diffuse the low-dimensional point cloud vector output by the diffusion model into a high-dimensional point cloud.

[0051] The point cloud encoder is used to encode high-dimensional point clouds into low-dimensional point cloud vectors. Figure 12 The point cloud decoder in is used to convert the low-dimensional point cloud vector (the low-dimensional point cloud vector is the low-dimensional point cloud vector output by the point cloud encoder or the inverse diffusion vector X′ generated by the last diffusion step of the inverse diffusion process of the diffusion model) (t) ) and the spherical point cloud are decoded into a high-dimensional point cloud, that is, the spherical point cloud is gradually expanded into the shape described by the low-dimensional point cloud vector according to the low-dimensional point cloud vector. Figure 5 and Figure 6 The exemplary structures of point cloud encoder and point cloud decoder are given. Figure 5 In the point cloud encoder shown in the figure, Cov represents the calculation of the covariance matrix of the high-dimensional point cloud, KNN represents the calculation of the K nearest neighbors of each point in the high-dimensional point cloud, Graph Layer represents the module that processes data information based on the graph structure, which includes a series of maximum pooling layers, fully connected layers, activation functions and convolutional layers, and GlobalMaxPooling represents the global maximum pooling operation. Figure 6 In the point cloud decoder shown, Concat represents the concatenation operation, and Folding represents the folding operation on the point cloud data, which consists of a series of convolutional layers and activation functions.

[0052] Before use, the point cloud encoder and decoder must undergo parameter optimization. This involves the following steps: The point cloud encoder encodes the real high-dimensional point cloud into a low-dimensional point cloud vector. This low-dimensional point cloud vector and a random sphere point cloud are then decoded with the point cloud decoder to reconstruct a high-dimensional point cloud. The parameters of the point cloud encoder and decoder are optimized by calculating the difference between the real high-dimensional point cloud and the reconstructed high-dimensional point cloud. The difference between the real high-dimensional point cloud and the reconstructed high-dimensional point cloud can be calculated as the mean squared error.

[0053] The diffusion model based on latent space also includes forward diffusion and reverse diffusion processes. In the forward diffusion process, the low-dimensional point cloud vector output by the point cloud encoder is used as the initial forward diffusion vector of the forward diffusion process. By adding actual noise in each diffusion step, forward diffusion is achieved and the forward diffusion vector of each diffusion step is obtained, which can be expressed as follows:

[0054]

[0055] Among them, X (t) and X(t-1) denote the positive diffusion vectors corresponding to diffusion steps t and t-1, β (t) It is a parameter obtained by uniform sampling in [0,1] according to the total number of steps T, N (t-1) The actual noise added in step t-1 can be Gaussian noise.

[0056] In the reverse diffusion process, the fourth random noise randomly sampled from the Gaussian distribution is used as the initial reverse diffusion vector X′ of the reverse diffusion process (T) , and the concatenation of the shape vector and the time step vector is used as the guiding condition. For each step of reverse diffusion, the cumulative noise is calculated based on the guiding condition and the reverse diffusion vector of the previous diffusion step, and the reverse diffusion vector of each diffusion step is calculated based on the cumulative noise, which can be expressed as follows:

[0057]

[0058] Among them, X′ (t-1) and X′ (t) represents the inverse diffusion vector corresponding to diffusion step t-1 and t, N′ (t-1) Indicates that the noise prediction network net() in step t-1 is based on the guidance condition (z, s) and the inverse diffusion vector X′ of the previous diffusion step (t) The predicted cumulative noise is expressed as:

[0059] N′ (t-1) =net(X′ (t) ,z,s)

[0060] s=embedding(β (t) )

[0061] Among them, S represents the time step vector, which is determined by the parameter β (t) It is obtained through embedding().

[0062] The noise prediction network provided by the embodiment is as follows Figure 7 As shown, it can enhance local details and includes at least two noise prediction units. Each noise prediction unit includes a feature fusion module, a feature extraction module, and a feature propagation module. The guide condition (z, s) and the inverse diffusion vector X′ of the previous diffusion step (t) The feature fusion module fuses the splicing features, and the splicing features are extracted by the point cloud branch, sampling branch, and voxel branch of the feature extraction module respectively to extract three modal features. The features calculated by the feature propagation module are used as the cumulative noise N′ (t-1) .

[0063] like Figure 8 As shown, the feature fusion module provided by the embodiment includes three linear layers Linear, the inverse diffusion vector X′ of the previous diffusion step(t) After passing through the first linear layer, the result of the guide condition (z, s) passing through the second linear layer is multiplied to obtain the fusion feature. The result of the guide condition passing through the second linear layer and the third linear layer in turn is spliced ​​with the fusion vector to obtain the splicing feature M.

[0064] like Figure 9 As shown, the feature extraction module provided by the embodiment includes a point cloud branch (Point-branch), a voxel branch (Voxel-branch) and a sampling branch (Sampling). The splicing feature M output by the feature fusion module is subjected to the first MLP in the point cloud branch to obtain the second modal feature P. The splicing feature is subjected to the voxelization operation, the second MLP and the devoxelization operation in the voxel branch to obtain the second modal feature Y. The splicing feature is sampled in the sampling branch and spliced ​​with the second modal feature to obtain the third modal feature C. It should be noted that the point cloud branch and voxel branch structures in the feature extraction module are the same as those in FIG. Figure 11 The point cloud branch and voxel branch structures in the PVConv layer shown are the same.

[0065] like Figure 10 As shown, the feature propagation module provided in the embodiment includes an upsampling layer (UpSampling), a concatenation layer (Concat), a third MLP and a PVConv layer, wherein the second modal feature Y and the third modal feature C are sampled in the upsampling layer and then concatenated with the second modal feature Y and the first modal feature P in the concatenation layer. The concatenation result is sequentially calculated by the third MLP and PVConv layers to obtain a feature as the cumulative noise N′ (t-1) .

[0066] like Figure 11 As shown, the PVConv layer provided in the embodiment includes a point cloud branch (Point-branch) and a voxel branch (Voxel-branch). The feature Q output by the third MLP is passed through the first MLP in the point cloud branch to obtain a feature. The feature Q output by the third MLP is sequentially subjected to a voxelization operation (Voxellization), a second MLP, and a trilinear devoxellization operation (TrillinearDevoxellization) in the voxel branch to obtain another feature. The features output by the two branches are spliced ​​to obtain the cumulative noise N′ (t-1) , where GroupNorm represents the group normalization function and Swish represents the Swish activation function.

[0067] Step 3: Construct the loss function.

[0068] In this embodiment, the difference between the third random noise and the preset noise label is used as the supervised loss function of the conditional flow model, and the difference between the actual noise added by the forward diffusion process and the cumulative noise calculated by the backward diffusion process in the same diffusion step is used as the supervised loss function of the diffusion model. It should be noted that the difference in noise can be measured using the mean square error.

[0069] Step 4: Use the loss function and training samples to optimize the parameters of the training system.

[0070] In this embodiment, when optimizing the parameters of the training system using a loss function and training samples, the parameters of the point cloud encoder and decoder, shape encoder, and ClIP are fixed, and the parameters of the conditional flow model and diffusion model are optimized with the goal of minimizing the loss function. This embodiment uses the encoded implicit low-dimensional point cloud vectors to train the diffusion model, thereby accelerating training, reducing network parameters, and saving video memory overhead.

[0071] Step 5: Generate three-dimensional objects using the parameter-optimized conditional flow model and diffusion model.

[0072] In the embodiment, after the parameter optimization is completed, the three-dimensional object is generated using the conditional flow model and diffusion model of the parameter optimization, such as Figure 12 As shown, the following steps are included:

[0073] Step 5-1, using the text encoder of the CLIP model to generate a semantic vector based on the text data;

[0074] Step 5-2: Generate a shape vector based on the semantic vector and the first random noise using the parameter-optimized conditional flow model, and concatenate the shape vector with the time step vector as a guiding condition;

[0075] Step 5-3: Using the second random noise as the initial reverse diffusion vector, a parameter-optimized diffusion model is used to generate a low-dimensional point cloud vector through reverse diffusion based on the guidance condition and the initial reverse diffusion vector;

[0076] In step 5-4, a point cloud decoder is used to decode the low-dimensional point cloud vector to obtain a high-dimensional point cloud, and a three-dimensional object is generated based on the high-dimensional point cloud.

[0077] Existing diffusion model-based 3D point cloud generation solutions can only generate a single category of 3D point cloud. The method provided in this embodiment introduces object category as conditional semantic information and uses the CLIP model to match semantic information with 3D point clouds, thus enabling the generation of multi-category point cloud data using a single diffusion model.

[0078] The method provided in this embodiment introduces a point cloud encoder and decoder. The input to the diffusion model is no longer 3D point cloud data, but rather hidden low-dimensional point cloud vectors calculated using the point cloud encoder. The low-dimensional point cloud vectors inferred by the diffusion model are then restored to a 3D point cloud using the point cloud decoder. This significantly improves the training speed of the 3D point cloud diffusion model and reduces computational overhead.

[0079] The method provided in this embodiment introduces a conditional flow model. During the training phase, the conditional flow model estimates noise based on a conditional vector. During the inference phase, the conditional flow model generates a conditional vector for the guided diffusion model based on the input noise. The conditional vector during training includes the point cloud shape vector extracted by the point cloud encoder and the semantic vector extracted by the CLIP model from the sample appearance image.

[0080] The method provided in the embodiment targets the noise prediction network in the diffusion model, introduces the PVConv (point cloud-voxel convolution module) operator and a multi-level structure, so that the network can extract finer local detail features.

[0081] In summary, the method provided in the embodiment can generate more detailed and higher quality three-dimensional objects while significantly reducing training time and memory consumption.

[0082] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A three-dimensional object generation method based on diffusion model and semantic guidance, characterized in that: The following steps are involved: Use the text encoder of the CLIP model to generate semantic vectors based on text data; A conditional flow model is used to generate a shape vector based on the semantic vector and the first random noise, and the shape vector is concatenated with the time step vector as a guiding condition; The second random noise is used as the initial reverse diffusion vector, and a low-dimensional point cloud vector is generated by reverse diffusion based on the guidance condition and the initial reverse diffusion vector using a diffusion model; Use the point cloud decoder to decode the low-dimensional point cloud vector to obtain a high-dimensional point cloud, and generate a three-dimensional object based on the high-dimensional point cloud; The conditional flow model and diffusion model need to undergo parameter optimization before being applied, including the following steps: Build samples: prepare high-dimensional point clouds and render them to obtain object images; Constructing a training system: The training system includes a shape encoder, a CLIP model, a point cloud encoder, a conditional flow model, and a diffusion model. The high-dimensional point cloud is encoded by the shape encoder and the point cloud encoder at the same time to obtain a shape vector and a low-dimensional point cloud vector. The object image is encoded by the image encoder of the CLIP model to obtain a semantic vector. The shape vector and the semantic vector are generated by the conditional flow model to generate a third random noise. The low-dimensional point cloud vector is used as the initial forward diffusion vector of the forward diffusion process of the diffusion model. The forward diffusion is achieved by adding actual noise at each diffusion step and the forward diffusion vector of each diffusion step is obtained. The fourth random noise randomly sampled from the Gaussian distribution is used as the initial reverse diffusion vector of the reverse diffusion process of the diffusion model. The concatenation of the shape vector and the time step vector is used as the guiding condition. For each step of reverse diffusion, the cumulative noise is calculated based on the guiding condition and the reverse diffusion vector of the previous diffusion step, and the reverse diffusion vector of each diffusion step is calculated based on the cumulative noise. Constructing the loss function: The difference between the third random noise and the preset noise label is used as the supervised loss function of the conditional flow model, and the difference between the actual noise added by the forward diffusion process and the cumulative noise calculated by the reverse diffusion process in the same diffusion step is used as the supervised loss function of the diffusion model; System training: Use loss functions and samples to train the training system to optimize the parameters of the conditional flow model and diffusion model; During the inverse diffusion process of the diffusion model, cumulative noise is predicted based on a guide condition and an inverse diffusion vector of a previous diffusion step through a noise prediction network, wherein the noise prediction network includes at least two noise prediction units, each of which includes a feature fusion module, a feature extraction module, and a feature propagation module. The guide condition and the inverse diffusion vector of the previous diffusion step are fused into a splicing feature through the feature fusion module, and the splicing feature is respectively extracted from the point cloud branch, the sampling branch, and the voxel branch of the feature extraction module. The features obtained by calculating the three modal features through the feature propagation module are used as the cumulative noise; The feature fusion module includes three linear layers. The inverse diffusion vector of the previous diffusion step passes through the first linear layer and is dot-multiplied with the result of the guide condition passing through the second linear layer to obtain a fused feature. The results of the guide condition passing through the second and third linear layers in sequence are spliced ​​with the fusion vector to obtain the spliced ​​feature.

2. The method for generating three-dimensional objects based on diffusion model and semantic guidance according to claim 1, characterized in that: The point cloud encoder and point cloud decoder need to undergo parameter optimization before being used, including the following steps: The real high-dimensional point cloud is encoded by the point cloud encoder to obtain a low-dimensional point cloud vector. The low-dimensional point cloud vector and the random sphere point cloud are decoded by the point cloud decoder to obtain a reconstructed high-dimensional point cloud. The parameters of the point cloud encoder and point cloud decoder are optimized by calculating the difference between the real high-dimensional point cloud and the reconstructed high-dimensional point cloud.

3. The method for generating three-dimensional objects based on diffusion model and semantic guidance according to claim 1, characterized in that: In the feature extraction module, the splicing feature is subjected to the first MLP in the point cloud branch to obtain the second modal feature, the splicing feature is subjected to voxelization operation, the second MLP and devoxelization operation in the voxel branch to obtain the second modal feature, and the splicing feature is sampled in the sampling branch and spliced ​​with the second modal feature to obtain the third modal feature.

4. The method for generating three-dimensional objects based on diffusion model and semantic guidance according to claim 3, characterized in that: The feature propagation module includes an upsampling layer, a splicing layer, a third MLP and a PVConv layer. After being sampled in the upsampling layer, the second modal feature and the third modal feature are spliced ​​with the second modal feature and the first modal feature in the splicing layer. The splicing result is sequentially calculated through the third MLP and PVConv layer to obtain a feature as the cumulative noise.

5. The method for generating three-dimensional objects based on diffusion model and semantic guidance according to claim 4, characterized in that: The PVConv layer includes a point cloud branch and a voxel branch. The feature output by the third MLP is subjected to the first MLP in the point cloud branch to obtain a feature. The feature output by the third MLP is subjected to voxelization operation, second MLP and devoxelization operation in the voxel branch to obtain another feature. The features output by the two branches are spliced ​​to obtain the accumulated noise.

6. The method for generating three-dimensional objects based on diffusion model and semantic guidance according to claim 1 or 2, characterized in that: The time step vector is obtained by embedding parameters uniformly sampled from [0, 1].

Images