A controllable point cloud generation method, medium and device without text training

Abstract

This invention provides a controllable point cloud generation method, medium, and device that does not require text training. The method includes a training phase and an inference phase. The training phase involves extracting features from a cross-modal skeleton point generator to obtain image features, and using a sparse point decoder to generate a sparse skeleton point cloud from the image features. The dense point cloud generator uses a conditional diffusion model to generate a dense point cloud. The sparse point decoder and conditional diffusion model are trained to obtain the sparse point decoder and conditional diffusion model. The inference phase involves using a text encoder to retrieve the image features that best match the embedded features of the text input. The final image features are automatically modulated by combining the proportions. These features are then input into the sparse point decoder and conditional diffusion model obtained in the training phase to generate a dense point cloud corresponding to the text. This method aims to accurately generate semantically consistent 3D point clouds using text input as a condition, without relying on text-shape pairing data for training, to optimize practicality in application scenarios such as industrial design.

Description

Technical Field

[0001] The present invention relates to the field of computer data processing technology, and more particularly to a controllable point cloud generation method, medium and device that do not require text training. Background Art

[0002] While text-guided image generation has made significant progress in terms of quality, diversity, and creativity, text-driven 3D shape generation technology remains in its early stages. A key challenge lies in the lack of large-scale paired text-3D datasets for model training. Building a large, well-annotated 3D dataset is both tedious and potentially impractical, posing a significant obstacle to the development of this technology.

[0003] Therefore, existing mainstream methods rely primarily on pre-trained visual language models (such as CLIP) to circumvent the need for text-3D paired data. These methods can be broadly categorized into two categories: 1) CLIP-based display generation methods, and 2) CLIP-guided neural radiance field (Nerf) optimization methods. However, both methods currently suffer from inevitable drawbacks. CLIP-based display generation methods primarily utilize autoencoders combined with normalized flow networks to generate target point clouds. While effective, their relatively simple text inputs prevent them from generating complex shapes. While CLIP-guided neural radiance field optimization methods offer advantages in generating novel views, they suffer from the following drawbacks: a) Neural radiance field-based iterative optimization is computationally expensive, typically requiring hours to generate a single instance of an object. b) Neural radiance field-based generation methods do not provide readily usable 3D display objects, requiring additional post-processing techniques (primarily the marching cubes algorithm) to extract objects that are usable by standard software. c) Due to the lack of strong geometric prior guidance, these methods struggle to generate accurate and expected shapes from given text input. Summary of the Invention

[0004] The purpose of the present invention is to overcome the shortcomings and deficiencies in the prior art and to provide a controllable point cloud generation method, medium, and device that do not require text training. This method aims to accurately generate semantically consistent three-dimensional point clouds using text input as a condition without relying on text-shape pairing data for training, thereby optimizing the practicality of application scenarios such as industrial design.

[0005] In order to achieve the above object, the present invention is implemented by the following technical solutions: a controllable point cloud generation method without text training, including a training stage and an inference stage;

[0006] The training phase includes: constructing a cross-modal skeleton point generator and a dense point cloud generator; the cross-modal skeleton point generator includes a dual encoder and a sparse point decoder; the dual encoder includes a visual image feature extractor of a CLIP pre-trained model and a convolutional feature extractor; the dense point cloud generator includes a conditional diffusion model;

[0007] In the cross-modal skeleton point generator, the visual image feature extractor and convolutional feature extractor of the CLIP pre-trained model are used to extract features and then merge them to obtain the image feature e cx ; Sparse point decoder for image feature e cx Predict the skeleton point cloud and generate a sparse skeleton point cloud. In the dense point cloud generator, the sparse skeleton point cloud is used as the geometric guide condition and the conditional diffusion model is used to generate a dense point cloud. The convolutional feature extractor, sparse point decoder, and conditional diffusion model are trained. The trained sparse point decoder and conditional diffusion model are used to construct the mapping relationship between image features and dense point clouds.

[0008] The inference stage is to use the trained CLIP text encoder to retrieve the m image features that best match the text input embedding features according to cosine similarity; and to automatically modulate each image feature e using a self-modulation combination function. ri The combination ratio of the final image feature e is obtained * , thereby building a mapping relationship between text and image; image feature e * The sparse point decoder and conditional diffusion model obtained in the training phase are input, and the mapping relationship between image features and dense point clouds constructed in the training phase is used to obtain the mapping between text, image features and dense point clouds, thereby generating the dense point cloud corresponding to the text.

[0009] Preferably, the training phase comprises the following steps:

[0010] S1, input image I x Input to the cross-modal skeleton point generator: Use the visual image feature extractor and convolutional feature extractor of the CLIP pre-trained model to extract the input image I x Visual semantic features of e x1 and local image features e x2 , and merge and connect to get the image feature e cx ; Use multi-layer perceptron to calculate image features e cx The mean u and variance σ of 2 :

[0011] (u,σ 2 )=MLP(Concat(e x1 , e x2 ))

[0012] Among them, Concat represents the merging operation, and MLP represents the multi-layer perceptron;

[0013] The input f1 of the sparse point decoder consists of the visual semantic features e x1 and the sampled eigenvector z; the sampled eigenvector z is obtained by reparameterization from the mean u and variance σ 2 The latent variables are sampled from the Gaussian distribution; the full connection operation is performed on the input f1 to obtain three consecutive feature layers f pi (i=1,2,3); by f pi Perform expansion, deformation and addition operations to obtain point clouds P with different resolutions i (i=1, 2, 3); point cloud P3 is:

[0014] P3=pθ(I x |z,e x1 )

[0015] Among them, p θ represents a sparse point decoder;

[0016] Train the convolutional feature extractor and sparse point decoder;

[0017] Set point cloud P3 as sparse skeleton point cloud P s ;

[0018] S2, generate the sparse skeleton point cloud P s As a geometric guidance condition, the dense point cloud P is generated by refining the conditional diffusion model. f , so that the final generated dense point cloud is semantically consistent with the input image; the conditional forward process of the conditional diffusion model and generation process for:

[0019]

[0020]

[0021] Where N represents the normal distribution, β t represents the attenuation coefficient at time t, Indicates the point cloud state at time t-1, Indicates the point cloud state at time t, P t represents the state of the free point at time t, μ θ (P t , P s , t) represents the mean value at time t, P s represents the constant sparse skeleton point cloud, σ 2 represents variance, I is the covariance matrix;

[0022] The following desired objective is trained by maximizing the log-likelihood of the point cloud:

[0023]

[0024] Among them, E (.) represents the expected function, represents the initial noise point, P0 represents the initial noise state including the skeleton point cloud, T represents the step size, Indicates the point cloud state at time t-1, Indicates the point cloud state at time t, P s represents the constant sparse skeleton point cloud, p θ represents the probability function, log represents the logarithmic function;

[0025] Train the conditional diffusion model.

[0026] Preferably, in step S1, the chamfer distance function is used as the loss function L c Train the convolutional feature extractor and sparse point decoder:

[0027]

[0028] Among them, P dr Represents the real skeleton point cloud sampled from the real point cloud.

[0029] Preferably, in step S2, the loss function L is used d Train the conditional diffusion model:

[0030]

[0031] Among them, ∈ represents the added noise, ∈ θ represents the model prediction function, represents the noise value predicted by the model, P s represents a constant sparse skeleton point cloud, Represents the point cloud state at time t.

[0032] Preferably, the reasoning stage includes the following steps:

[0033] Y1. Using the trained CLIP text encoder, retrieve the m image features e that best match the text input embedding features based on cosine similarity ri (i=1, 2, 3, ..., m);

[0034] Y2. Use self-modulation combination function to automatically modulate each image feature e ri The combination ratio of the final image feature e is obtained *, thereby building a mapping relationship between text and image; image feature e * for:

[0035]

[0036]

[0037] Among them, c ci Represents the i-th image feature e ri The weight coefficient, S ci represents the i-th cross-modal text-image cosine similarity, S ni represents the image-image cosine similarity of the i-th modality;

[0038] Y3. Use the sparse point decoder obtained in the training phase to predict the skeleton point cloud of the image features to generate a sparse skeleton point cloud; use the sparse skeleton point cloud as the geometric guidance condition and use the conditional diffusion model obtained in the training phase to generate a dense point cloud.

[0039] A readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the controllable point cloud generation method without text training.

[0040] A computer device includes a processor and a memory for storing a program executable by the processor. When the processor executes the program stored in the memory, the controllable point cloud generation method without text training is implemented.

[0041] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0042] 1. Existing text-to-3D generation methods are either limited to relatively simple text input, resulting in the inability to generate complex shapes, or require cumbersome optimization processes and post-processing techniques, which makes them difficult to use in a friendly manner. The present invention uses images as a bridge between text and three-dimensional shapes to establish their text-to-3D mapping relationship. Considering the discrete nature of point clouds, the present invention innovatively proposes a sparse-guided point cloud diffusion model to generate dense point clouds that are semantically consistent with the input image, ensuring the accuracy of point cloud generation and a high degree of matching of the image input. Considering the modal gap between CLIP's image features and text features, the present invention does not directly use text embedding features instead of image embedding features to generate the target point cloud, but instead proposes a non-parametric migration engine that directly performs text-guided 3D point cloud generation without the need for separate training. The present invention has made important progress in solving the data dependency and generation accuracy problems in the prior art, and has opened up a new path for the synthesis of three-dimensional shapes and the intelligent processing of text descriptions.

[0043] 2. The cross-modal skeleton point generator proposed in this paper can directly predict a 3D point cloud skeleton that matches the text description from an image, which is unusual in the prior art. Traditional methods typically rely on large datasets of text-3D shape pairings for training. However, this paper significantly reduces this reliance on paired data by innovatively utilizing images to bridge the transition between text and 3D shapes.

[0044] 3. In terms of generating detailed and accurate 3D point cloud shapes, the present invention uses a conditional diffusion model that uses skeleton point clouds as conditional input, which is a major improvement over existing technologies. It can generate point clouds with higher quality and more semantic details, which is superior to conventional generation methods in existing technologies.

[0045] 4. The present invention adopts a non-parametric migration engine without text training, allowing point clouds consistent with text prompts to be generated directly in the inference stage without text-to-3D data training; existing technologies usually require special training for text-to-3D shape mapping, while the method of the present invention provides a more flexible and rapid way to convert text to three-dimensional shapes. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 1 is a flow chart of a controllable point cloud generation method without text training according to the present invention;

[0047] Figure 2 Schematic diagram of the distribution of inter-modal and intra-modal text embedding and image embedding distances. DETAILED DESCRIPTION

[0048] The present invention will be described in further detail below with reference to the accompanying drawings and specific embodiments.

[0049] Example 1

[0050] Although large-scale text-shape datasets are difficult to obtain, using images as a bridge between text and three-dimensional shapes to establish their mapping relationship is an effective method. In addition, considering the discrete nature of point clouds, the present invention does not use the traditional folded decoder to generate point clouds, but innovatively uses a sparse-guided point cloud diffusion model to generate dense point clouds that are semantically consistent with the input image, ensuring the accuracy of point cloud generation and a high degree of matching of the image input. To this end, the present invention innovatively proposes three key technologies: 1) constructing an empirically robust cross-modal skeleton point generator for predicting specific shape skeleton points from a single image input; 2) developing a diffusion-based dense point cloud generator that performs geometric guidance based on skeleton points to ensure a high degree of consistency between the generated point cloud and the input image; 3) proposing a non-parametric migration engine that can directly realize text-guided 3D point cloud generation in the absence of text-3D data training.

[0051] This embodiment provides a controllable point cloud generation method that does not require text training, such as Figure 1 As shown, it includes the training phase and the inference phase.

[0052] The training phase refers to: building a cross-modal skeleton point generator and a dense point cloud generator; the cross-modal skeleton point generator includes a dual encoder and a sparse point decoder; the dual encoder includes a visual image feature extractor of a CLIP pre-trained model and a convolutional feature extractor; the dense point cloud generator includes a conditional diffusion model.

[0053] In the cross-modal skeleton point generator, the visual image feature extractor and convolutional feature extractor of the CLIP pre-trained model are used to extract features and then merge them to obtain the image feature e cx ; Sparse point decoder for image feature e cx A skeleton point cloud is predicted to generate a sparse skeleton point cloud; a cross-modal skeleton point generator is used to directly predict a skeleton point cloud describing the geometric structure of the object from the input image. The cross-modal skeleton point generator combines a dual encoder architecture and sparse point decoder technology to improve the robustness of the generated point cloud. Considering that the image features extracted by the CLIP pre-training model are mainly used to align text features and do not have detailed image feature information. Therefore, it is not enough to rely solely on the image feature extractor of the CLIP pre-training model to predict the skeleton point cloud. To this end, the dual encoder structure designed in the present invention mainly includes a visual image feature extractor of the CLIP pre-training model and a learnable convolutional feature extractor. The sparse point decoder is mainly implemented based on a modified point cloud pyramid structure. The reason for this design is mainly due to two ideal characteristics: a) better perception of the spatial structure of the target point cloud, and b) progressive propagation of local details.

[0054] In the dense point cloud generator, taking into account the discrete nature of the point cloud, the sparse skeleton point cloud is used as a geometric guidance condition, and the conditional diffusion model is used to generate a dense point cloud that is highly consistent with the original image, ensuring that the final generated dense point cloud is semantically consistent with the input image; the convolutional feature extractor, sparse point decoder and conditional diffusion model are trained; the sparse point decoder and conditional diffusion model obtained by training are used to construct a mapping relationship between image features and dense point clouds. The conditional diffusion model includes a conditional forward process and a generation process. Since it is expected that the generated dense point cloud maintains semantic consistency with the image, the present invention sets the geometric prior skeleton point cloud to remain unchanged during the generation step. Taking into account the computational efficiency and the effectiveness of local feature extraction, the noise prediction model used in the present invention is based on a modified PVCNN network.

[0055] Specifically, the training phase includes the following steps:

[0056] S1, input image I x Input to the cross-modal skeleton point generator: Use the visual image feature extractor and convolutional feature extractor of the CLIP pre-trained model to extract the input image I x Visual semantic features of e x1 and local image features e x2 , and merge and connect to get the image feature e cx ; Use a two-layer multilayer perceptron to calculate image features e cx The mean u and variance σ of 2 :

[0057] (u,σ 2 )=MLP(Concat(e x1 , c x2 ))

[0058] Among them, Concat represents a merging operation, and MLP represents a two-layer multi-layer perceptron.

[0059] Although, we can get the approximate posterior distribution q φ (z|I x , e x1 ) samples the potential feature vector z, but does not have z and (u, σ 2 ) between the two, which also leads to the calculation graph being truncated during the forward propagation process, and the gradient cannot be back-propagated back to the encoder. To this end, the present invention uses a re-parameterization strategy to solve this problem. The sparse decoder is represented as: p0(I x |z,e x1 ), the sparse point decoder adopts a modified pyramid structure. Specifically, the input f1 of the sparse point decoder consists of the visual semantic features e x1 and the sampled feature vector z; then, a full connection operation is performed on the input f1 to obtain three consecutive feature layers f pi (i=1,2,3); Finally, by pi Perform expansion, deformation and addition operations to obtain point clouds P with different resolutions i (i=1, 2, 3). Considering the computational efficiency and optimization goals, the present invention only uses the chamfer distance loss function to supervise the generated P3; the pyramid structure is used to generate the point cloud P3 (the resolution of P3 and the skeleton point cloud P s Keep the resolution consistent).

[0060] P3=p θ (I x |z,e x1 )

[0061] Among them, p θrepresents the probability density function of the model parameterized by the parameter θ, which is called the sparse point decoder here; I x Represents the input image; z represents the image obtained from the parameters u and σ 2 Specifically, the input data is mapped to these two parameters through a multi-layer perceptron, and then samples are drawn from this distribution using a reparameterization strategy. The reparameterization strategy allows the model to generate z through random sampling while maintaining the differentiability of the entire network, so that gradient optimization can be used;

[0062] Use the chamfer distance function as the loss function L c Train the convolutional feature extractor and sparse point decoder:

[0063]

[0064] Among them, P dr represents the real skeleton point cloud sampled from the real point cloud;

[0065] Set point cloud P3 as sparse skeleton point cloud P s ;

[0066] S2. Considering the discrete nature of the point cloud, the generated sparse skeleton point cloud P s As a geometric guidance condition, the conditional diffusion model is used to refine and generate a dense point cloud P that is highly consistent with the original image. f (The resolution of the dense point cloud is 2025×3), so that the final generated dense point cloud is semantically consistent with the input image; the conditional forward process of the conditional diffusion model and generation process for:

[0067]

[0068]

[0069] Where N represents the normal distribution, β t represents the attenuation coefficient at time t, Indicates the point cloud state at time t-1, Indicates the point cloud state at time t, P t represents the state of the free point at time t, μ θ (P t , P s，t ) represents the mean value at time t, P s represents the constant sparse skeleton point cloud, σ 2 represents variance, I is the covariance matrix;

[0070] Since the dense point cloud generated in this paper is expected to maintain semantic consistency with the image, the geometric prior skeleton point cloud is set to remain unchanged during the generation step; to this end, the following desired objectives are trained by maximizing the log-likelihood of the point cloud:

[0071]

[0072] Among them, E (.) represents the expected function, represents the initial noise point, P0 represents the initial noise state including the skeleton point cloud, T represents the step size, Indicates the point cloud state at time t-1, Indicates the point cloud state at time t, P s represents the constant sparse skeleton point cloud, p θ represents the probability function, log represents the logarithmic function;

[0073] Use the loss function L d Train the conditional diffusion model:

[0074]

[0075] Among them, ∈ represents the added noise, ∈ θ Represents the model prediction function, ∈ θ (P t , P s , t) represents the noise value predicted by the model, P s represents a constant sparse skeleton point cloud, Represents the point cloud state at time t.

[0076] During the training phase, the present invention establishes a mapping relationship between image features and point clouds. During the inference phase, only the mapping between text and image features needs to be established, which will complete the mapping between text, image features, and point clouds. To this end, the present invention designs an efficient non-parametric transfer engine that does not require text training. This engine directly synthesizes the target point cloud based on text input without a separate training step.

[0077] The inference phase is as follows: first, the trained CLIP text encoder is used to retrieve the m image features that best match the text input embedding features based on cosine similarity; second, the self-modulation combination function is used to automatically modulate each image feature e ri The combination ratio of the final image feature e is obtained * , thereby building a mapping relationship between text and image; then, the image feature e *The sparse point decoder and conditional diffusion model generated during the training phase are fed into the training phase. The mapping between image features and dense point clouds established during the training phase is then used to map text to image features and dense point clouds, thereby generating a dense point cloud corresponding to the text. During the inference phase, the network weights for both the sparse point cloud and dense point cloud generation processes are frozen and no longer trained. This avoids the need for large-scale text-point cloud matching pairs required for model training and offers the following advantages: a) The image embedding with the highest cosine similarity is given the highest weight and controls the final shape generated. b) When the same text input is used repeatedly to generate the target point cloud, the results are consistent or similar.

[0078] Through this method, a new approach is opened up for text-driven 3D point cloud generation without the need for tedious dataset preparation, greatly simplifying the generation process from text description to 3D point cloud.

[0079] Specifically, the inference phase includes the following steps:

[0080] Y1. Using the trained CLIP text encoder, retrieve the m image features e that best match the text input embedding features based on cosine similarity ri (i=1,2,3,…,m);

[0081] Y2. Considering that the CLIP model maximizes the similarity of text-image matching pairs without considering the similarity within the modality, the spatial distance between the m image features may be far or close, such as Figure 2 As shown; To this end, the present invention further imposes constraints between other image features (m-1) and the image feature with the highest matching degree with the text input, reducing the proportion weight of other non-matching image features; To this end, the present invention uses a self-modulation combination function to automatically modulate each image feature e ri The combination ratio of the final image feature e is obtained * , thereby building a mapping relationship between text and image; image feature e * for:

[0082]

[0083]

[0084] Among them, c ci Represents the i-th image feature e ri The weight coefficient, S ci represents the i-th cross-modal text-image cosine similarity, S ni represents the image-image cosine similarity of the i-th modality;

[0085] Y3. Use the sparse point decoder obtained in the training phase to predict the skeleton point cloud of the image features to generate a sparse skeleton point cloud; use the sparse skeleton point cloud as the geometric guidance condition and use the conditional diffusion model obtained in the training phase to generate a dense point cloud.

[0086] Example 2

[0087] This embodiment provides a readable storage medium, wherein the readable storage medium stores a computer program. When the computer program is executed by a processor, the processor executes the controllable point cloud generation method without text training described in the first embodiment.

[0088] Example 3

[0089] This embodiment provides a computer device, including a processor and a memory for storing a program executable by the processor. When the processor executes the program stored in the memory, the controllable point cloud generation method without text training described in the first embodiment is implemented.

[0090] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A controllable point cloud generation method without text training, characterized by: Includes training phase and inference phase; The training phase includes: constructing a cross-modal skeleton point generator and a dense point cloud generator; The cross-modal skeleton point generator includes a dual encoder and a sparse point decoder; the dual encoder includes a visual image feature extractor of a CLIP pre-trained model and a convolutional feature extractor; the dense point cloud generator includes a conditional diffusion model; In the cross-modal skeleton point generator, the visual image feature extractor and convolutional feature extractor of the CLIP pre-trained model are used to extract features and then merge them to obtain the image feature e cx ; Sparse point decoder for image features e cx Predict the skeleton point cloud and generate sparse skeleton point cloud; In the dense point cloud generator, a sparse skeleton point cloud is used as a geometric guide condition and a conditional diffusion model is used to generate a dense point cloud. The convolutional feature extractor, sparse point decoder, and conditional diffusion model are trained. The trained sparse point decoder and conditional diffusion model are used to construct a mapping relationship between image features and dense point clouds. The inference stage is to use the trained CLIP text encoder to retrieve m image features that best match the text input embedding features based on cosine similarity; Use self-modulation combination function to automatically modulate each image feature ri The combination ratio of the final image feature e is obtained * , thereby building a mapping relationship between text and image; image feature e * The sparse point decoder and conditional diffusion model obtained in the training phase are input, and the mapping relationship between image features and dense point clouds constructed in the training phase is used to obtain the mapping between text, image features and dense point clouds, thereby generating the dense point cloud corresponding to the text.

2. The method for generating a controllable point cloud without text training according to claim 1, characterized in that: The training phase includes the following steps: S1, input image I x Input to the cross-modal skeleton point generator: Use the visual image feature extractor and convolutional feature extractor of the CLIP pre-trained model to extract the input image I x Visual semantic features of e x1 and local image features e x2 , and merge and connect to get the image feature e cx ; Use multi-layer perceptron to calculate image features e cx The mean u and variance σ of 2 : (u,σ 2 )=MLP(Concat(e x1 ,and x2 )) Among them, Concat represents the merging operation, and MLP represents the multi-layer perceptron; The input f1 of the sparse point decoder consists of the visual semantic features e x1 and the sampled eigenvector z; the sampled eigenvector z is obtained by reparameterization from the mean u and variance σ 2 The latent variables are sampled from the Gaussian distribution; the full connection operation is performed on the input f1 to obtain three consecutive feature layers f pi (i=1,2,3); by f pi Perform expansion, deformation and addition operations to obtain point clouds P with different resolutions i (i=1, 2, 3); point cloud P3 is: P3=p θ (I x |z,e x1 ) Among them, p θ represents a sparse point decoder; Train the convolutional feature extractor and sparse point decoder; Set point cloud P3 as sparse skeleton point cloud P s ; S2, generate the sparse skeleton point cloud P s As a geometric guidance condition, the dense point cloud P is generated by refining the conditional diffusion model. f , so that the final generated dense point cloud is semantically consistent with the input image; the conditional forward process of the conditional diffusion model and generation process for: Where N represents the normal distribution, β t represents the attenuation coefficient at time t, Indicates the point cloud state at time t-1, Represents the point cloud state at time t, P t represents the state of the free point at time t, μ θ (P t , P s , t) represents the mean value at time t, P s represents the constant sparse skeleton point cloud, σ 2 represents variance, I is the covariance matrix; The following desired objective is trained by maximizing the log-likelihood of the point cloud: Among them, E(.) represents the expectation function, represents the initial noise point, P0 represents the initial noise state including the skeleton point cloud, T represents the step size, Indicates the point cloud state at time t-1, Represents the point cloud state at time t, P s represents the constant sparse skeleton point cloud, p θ represents the probability function, log represents the logarithmic function; Train the conditional diffusion model.

3. The method for generating a controllable point cloud without text training according to claim 2, characterized in that: In step S1, the chamfer distance function is used as the loss function L c Train the convolutional feature extractor and sparse point decoder: Among them, P dr Represents the real skeleton point cloud sampled from the real point cloud.

4. The method for generating a controllable point cloud without text training according to claim 2, characterized in that: In step S2, the loss function L is used d Train the conditional diffusion model: Among them, ∈ represents the added noise, ∈ θ represents the model prediction function, represents the noise value predicted by the model, P s represents a constant sparse skeleton point cloud, Represents the point cloud state at time t.

5. The method for generating controllable point clouds without text training according to claim 1, characterized in that: The reasoning phase includes the following steps: Y1. Using the trained CLIP text encoder, retrieve the m image features e that best match the text input embedding features based on cosine similarity ri (i=1, 2, 3, ..., m); Y2. Use self-modulation combination function to automatically modulate each image feature e ri The combination ratio of the final image feature e is obtained * , thereby building a mapping relationship between text and image; image feature e * for: Among them, c ci Represents the i-th image feature e ri The weight coefficient, S ci represents the i-th cross-modal text-image cosine similarity, S ni represents the image-image cosine similarity of the i-th modality; Y3. Use the sparse point decoder obtained in the training phase to predict the skeleton point cloud of the image features to generate a sparse skeleton point cloud; use the sparse skeleton point cloud as the geometric guidance condition and use the conditional diffusion model obtained in the training phase to generate a dense point cloud.

6. A readable storage medium, characterized in that: The storage medium stores a computer program, which, when executed by a processor, enables the processor to execute the controllable point cloud generation method without text training according to any one of claims 1 to 5.

7. A computer device comprising a processor and a memory for storing a program executable by the processor, characterized in that: When the processor executes the program stored in the memory, the controllable point cloud generation method without text training according to any one of claims 1 to 5 is implemented.

Images