Controllable three-dimensional model generation method and system

Abstract

The application discloses a controllable three-dimensional model generation method and system. The method includes data processing, model training and model application. The data processing includes multi-scale point cloud sampling of the three-dimensional model, which is divided into three levels from high to low to capture the contour features of the model; meanwhile, the length, width and height of the model are counted and quantified as a cuboid and sampled as an 8-dimensional sparse point cloud, which is used as a conditional input to control the size of the generated model. The model training includes replacing the traditional UNET architecture with the Transformer architecture and introducing the cross-attention mechanism to fuse the point cloud set and the cuboid sparse point cloud information and accurately control the size of the generated model. The model application includes controlling the appearance of the generated model by using the object picture and combining the cuboid to control the model size. The application can automatically adjust the object layout, accurately control the length, width and height, meet the demand of automatic building in three-dimensional scene modeling, improve the modeling efficiency and quality, and reduce the cost of manual adjustment.

Description

Technical Field

[0001] This invention relates to the fields of pattern recognition and computer vision technology, and in particular to a controllable method and system for generating 3D models. Background Technology

[0002] In today's rapidly evolving field of 3D model generation, designers face unprecedented challenges. Traditional 3D model generation processes often rely on the individual experience and creativity of designers, which not only limits the improvement of generation efficiency but also makes it difficult for designers to achieve optimal generation results when faced with complex model requirements, as they often struggle to work alone.

[0003] With the rapid development of technologies such as artificial intelligence and big data, human-machine collaborative 3D model generation is gradually becoming possible. In this context, combining the creativity of designers with the efficient computing power of machines holds promise for significantly improving the efficiency and quality of 3D model generation. However, a mature human-machine collaborative 3D model generation system is currently lacking in the market, especially in the field of 3D model generation, where this problem is particularly prominent. Summary of the Invention

[0004] Based on this, the embodiments of this application provide a controllable three-dimensional model generation method and system. This application achieves efficient and accurate three-dimensional model generation through innovative data processing, model training and application processes.

[0005] Firstly, a controllable method for generating 3D models is provided, the method comprising:

[0006] S1 Data Processing: Rendering and cropping the original 3D model, rendering object images from different angles, and sampling point clouds at different scales from the original 3D model to obtain the model's contour information as input for training and learning; then defining a cuboid sparse point cloud based on the length, width, and height of the original 3D model, and using the sparse point cloud as a conditional input to control the volume of the generated model.

[0007] S2 Model Training: An image encoder maps generated object images from different angles into a latent space, generating latent feature vectors. The resulting backbone network, Transformer, is then trained to obtain the target network. Specifically, a 3D model point cloud is used as input, with the latent feature vectors from the image and a cuboid sparse point cloud as conditional inputs to a cross-attention module. The 3D model point cloud is mapped into the latent space via the encoder, and random noise is added as input to the backbone network. The latent feature vectors from the image and the cuboid sparse point cloud are then used as conditional inputs to the backbone network for training. The backbone network outputs predicted noise, which, along with the previously generated random noise, is input into the mean squared error loss function. Gradient descent is used to guide the network's training.

[0008] S3 model application: Acquire target images, use image encoders to map target images into target feature variables, and input the target feature variables and the target cuboid sparse point cloud as conditional control inputs into the backbone network. Use samplers to denoise the input random noise. Input the denoised feature variables into the decoder to generate the final target 3D model.

[0009] Optionally, S1 data processing specifically includes:

[0010] The original 3D model is rendered to obtain a 2D image; different angles are used during rendering, and the sampled objects are mainly based on the x and y axes, while the z axis remains consistent.

[0011] The original 3D models were processed into point clouds using 3D tools, and each model was uniformly processed into 100,000 point clouds.

[0012] The farthest point sampling algorithm was used to sample sets of 8192, 4096, and 2048 points respectively to obtain point cloud data at different scales;

[0013] Calculate the length, width, and height of the original 3D model, and then calculate the coordinates of the eight vertices of the cuboid.

[0014] The coordinates of these 8 vertices are used as a sparse point cloud and then normalized.

[0015] Optionally, the S2 model training specifically includes:

[0016] The image encoder is used to encode the rendered object images in the data processing step into feature vectors. The feature map size is 768*1024, and the RGB image is converted into implicit encoding.

[0017] The 3D model point cloud is mapped to latent space feature vectors through an encoder, and noise is added as input. The image encoding of the image encoder unit and the sparse point cloud of 8-dimensional vertices are used as conditional inputs. The backbone network predicts the noise and updates the network parameters using the gradient descent method.

[0018] The loss function is composed of the mean squared error loss function, and the inputs are the noise added to the point cloud of the 3D model and the noise predicted by the backbone network.

[0019] Optionally, an image encoder is used to encode the rendered object image in the data processing step into a feature vector, specifically including:

[0020] Establish an image encoding network and load the trained image encoder parameters into the network;

[0021] The parameters of the established image encoding network are frozen. During training, a 512*512 image is encoded into a feature vector, which is used as an implicit vector. The feature map size of the implicit vector is 768*1024.

[0022] Optionally, noise is added as input. The image encoding from the image encoder unit and the sparse point cloud of 8-dimensional vertices are used as conditional inputs. The backbone network predicts the noise, and the network parameters are updated using the gradient descent method. Specifically, this includes:

[0023] Random noise is gradually added to the latent space feature vectors; specifically, the original feature vectors are gradually added to a set of pure noise feature vectors.

[0024] The generated pure noise feature vector is added to the backbone network as input, and the image encoding and cuboid point cloud are used as conditional inputs. The backbone network structure adopts a full Transformer module, each module contains one self-attention module and two cross-attention modules. The self-attention module is used for feature mapping and information interaction of the point cloud input. One of the two cross-attention modules is used for information interaction between the point cloud input and the image encoding, and the other is used for global and local interaction between the point cloud input and the cuboid point cloud input. Weights are added to balance the information interaction ratio between them.

[0025] In the initial training phase, the conditional control between point cloud input and image encoding is trained first. A model that can stably control the generation of 3D model is first trained. The overall learning rate is 1e-4 for training, and the training process is 500 loops.

[0026] During the fine-tuning phase of training, the cross-attention module that interacts with image encoding does not participate in the update. Instead, sparse points of a cuboid are added as conditional input to train another cross-attention module. The second phase of training mainly trains the interaction between the sparse point cloud of the cuboid and the overall point cloud, so that the vertex information of the custom cuboid can control the size of the generated model. The overall learning rate is 1e-5 for training, and the training process consists of 200 loops.

[0027] Optionally, the loss function is composed of the mean squared error loss function, and the inputs are the noise added to the 3D model point cloud and the noise predicted by the backbone network, specifically including:

[0028] The loss function adopted is the mean squared error loss function, and the inputs are the noise predicted by the backbone network and the random noise added to the feature vector of the latent space of the point cloud of the 3D model, respectively.

[0029] Update the model parameters using gradient descent.

[0030] Optionally, S3 model applications also include:

[0031] Edit the images and custom cuboids needed to generate the 3D model;

[0032] We construct random noise and use an autoencoder to map the random noise into a feature vector. The images needed to generate the 3D model are mapped into image feature vectors through an image encoder. The image feature vectors are used as input to the backbone network. The image feature vectors and the cuboid sparse point cloud are used as conditional feature vectors. The backbone network outputs predicted noise. We use a sampler to denoise the image encoding to obtain the denoised implicit vector.

[0033] The final 3D model is obtained by decoding the denoised implicit vectors.

[0034] Secondly, a controllable 3D model generation system is provided, the system comprising:

[0035] The data processing module is used to render and crop the original 3D model, render object images from different angles, and sample point clouds of the original 3D model at different scales to obtain the model's contour information as input for training and learning. Then, based on the length, width, and height of the original 3D model, a cuboid sparse point cloud is defined, and the sparse point cloud is used as a conditional input to control the volume of the generated model.

[0036] The model training module uses an image encoder to map generated object images from different angles into a latent space, generating latent feature vectors for the images. This latent feature vectors are then used to train the backbone network, Transformer, to obtain the trained target network. Specifically, a cross-attention module takes the point cloud of a 3D model as input and the latent feature vectors of the images and the sparse point cloud of a cuboid as conditional control inputs. The module maps the 3D model point cloud into the latent space through the encoder and adds random noise as input to the backbone network. The latent feature vectors of the images and the sparse point cloud of the cuboid are then used as conditional inputs to the backbone network for training. The backbone network outputs predicted noise, and this predicted noise, along with the previous random noise, is input into the mean squared error loss function. Gradient descent is used to guide the network's training.

[0037] The model is applied to acquire target images, which are then mapped to target feature variables using an image encoder. The target feature variables and the target cuboid sparse point cloud are input as conditional control inputs to the backbone network, and a sampler is used to denoise the random noise in the input. The denoised feature variables are then input to the decoder to generate the final target 3D model.

[0038] Thirdly, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the three-dimensional model generation method described in any of the first aspects above.

[0039] Fourthly, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the three-dimensional model generation method described in any of the first aspects above.

[0040] The beneficial effects of the technical solutions provided in this application include at least the following:

[0041] (1) By quantizing the length, width, and height of the 3D model into a sparse point cloud of cuboids and using it as a conditional input, combined with the feature vector of the rendered image, this invention can precisely control the size and appearance of the generated 3D model. This control method enables the generated 3D model to better meet design requirements, and is especially suitable for scene design and concept verification that require strict size and appearance requirements.

[0042] (2) The use of a Transformer architecture backbone network instead of the traditional UNET network enables more efficient utilization of spatial and temporal information. Combined with the cross-attention module, the model can better integrate point cloud input, image features, and conditional information of sparse point clouds, thereby improving the efficiency and quality of generating 3D models. In addition, the training effect of the model is further optimized through a phased training strategy, reducing the number of parameters that need to be updated and improving training efficiency.

[0043] (3) By sampling point clouds of the 3D model at different scales and using the farthest point sampling algorithm to ensure the diversity and representativeness of the sampling, this invention can enhance the robustness of the model to 3D models of different complexities and shapes. At the same time, by rendering object images from different angles as conditional inputs, the model can better adapt to various appearance features, thereby improving its generalization ability and enabling it to generate high-quality 3D models, maintaining good performance even when facing complex input conditions. Attached Figure Description

[0044] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely exemplary, and those skilled in the art can derive other embodiments based on the provided drawings without creative effort.

[0045] Figure 1 A flowchart illustrating the steps of a controllable 3D model generation method provided in this application embodiment;

[0046] Figure 2 This is a schematic diagram of the model training process provided in the embodiments of this application;

[0047] Figure 3 This is a schematic diagram of the model generation process provided in the embodiments of this application;

[0048] Figure 4 This is a schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0050] In the description of this invention, the terms "comprising," "having," and any variations are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units that are expressly listed, but may also include other steps or units that are not expressly listed but are inherent to these processes, methods, products, or apparatuses, or steps or units added based on further optimizations of the inventive concept.

[0051] To facilitate understanding of this embodiment, a controllable three-dimensional model generation method disclosed in this application embodiment will first be described in detail.

[0052] Please refer to Figure 1 The diagram illustrates a flowchart of a controllable 3D model generation method provided in an embodiment of this application. The method may include the following steps:

[0053] S1 Data Processing: The original 3D model is rendered and cropped, and object images from different angles are rendered. Point cloud sampling at different scales is performed on the original 3D model to sample the model's contour information as input for training and learning. Then, a cuboid sparse point cloud is defined based on the length, width, and height of the original 3D model, and the sparse point cloud is used as a conditional input to control the volume of the generated model.

[0054] Specifically, in data processing, 3D tools are used to convert 3D models into point clouds. Each model is uniformly processed into 100,000 point clouds. The farthest point sampling algorithm is used to sample 8,192, 4,096, and 2,048 point sets respectively. Different sampling scales can make the model more robust. Normalizing the point cloud sets can enable the training model to converge quickly.

[0055] The length, width, and height of the 3D model are statistically analyzed, and the eight vertices of the cuboid are calculated using the length, width, and height. These eight vertices are used as input conditions and are normalized in the same way as the model's electric cloud.

[0056] The 3D model is rendered to obtain a 2D image. Different angles are used during rendering, and images from different angles are used as input during training. The sampled objects are mainly based on the x and y axes, while the z axis remains consistent. In addition, the lighting needs to be closer to the actual image.

[0057] S2 Model Training: An image encoder maps generated object images from different angles into a latent space, generating latent feature vectors. The resulting backbone network, Transformer, is then trained to obtain the target network. Specifically, a 3D model point cloud is used as input, with the latent feature vectors from the image and the cuboid sparse point cloud as conditional control inputs to a cross-attention module. The 3D model point cloud is mapped into the latent space via the encoder, and random noise is added as input to the backbone network. The latent feature vectors from the image and the cuboid sparse point cloud are then used as conditional inputs to the backbone network for training. The backbone network outputs predicted noise, which, along with the previously generated random noise, is input into the mean squared error loss function. Gradient descent is used to guide the network's training.

[0058] Specifically, model training includes an image encoder unit, which encodes the rendered image into a feature vector. The feature map size is 768*1024. The RGB image is converted into implicit encoding. Subsequent operations are performed on the hidden encoding, which can achieve better generation results.

[0059] The image encoder unit includes:

[0060] Step (1): Establish an image encoding network and load the trained image encoder parameters into the network.

[0061] Step (2): Freeze the image encoder parameters obtained in step (1). During the training process, encode the 512*512 image into a feature vector. The feature vector, as an implicit vector, can better handle potential feature information. The feature map size of the implicit vector is 768*1024.

[0062] The noise prediction unit maps the 3D model point cloud to latent space feature vectors by the encoder and adds noise as input. The image encoding from the image encoder unit and the sparse point cloud of 8-dimensional vertices are used as conditional inputs. The backbone network predicts noise and updates network parameters using the gradient descent method.

[0063] The noise prediction unit includes:

[0064] Step (1): Random noise is gradually added to the latent space feature vector of . Let the number of steps for adding noise be T, and let the coefficients for adding noise be a1, a2, ... a tThe coefficients are all predetermined, and they gradually increase as the steps increase. Through this sampling method, the original feature vector is gradually added to a set of pure noise feature vectors.

[0065] Step (2): The pure noise feature vector generated in step (1) is added to the backbone network as input, and the image encoding and cuboid point cloud are used as conditional inputs. The backbone network structure adopts a full Transformer module. Each module contains a self-attention module and two cross-attention modules. The self-attention module is used for feature mapping and information interaction of the point cloud input. One of the two cross-attention modules is used for information interaction between the point cloud input and the image encoding, and the other is used for global and local interaction between the point cloud input and the cuboid point cloud input. Weights are added to balance the information interaction ratio between them.

[0066] Step (3): In the initial training phase, the conditional control between the point cloud input and the image encoding is trained first. A model that can stably control the generation of the 3D model is trained first. The overall learning rate is 1e-4, and the training process is 500 loops. In the fine-tuning phase of training, the cross-attention module that interacts with the image encoding does not participate in the update. A cuboid sparse point is added as a conditional input, and another cross-attention module is trained. The second phase of training mainly trains the interaction between the cuboid sparse point cloud and the overall point cloud, so that the vertex information of the custom cuboid can control the size of the generated model. The overall learning rate is 1e-5, and the training process is 200 loops.

[0067] Step (4): In the third stage, alpha is used to balance the ratio of the two cross-attention modules. CrossAtten is used to represent the cross-attention module, as shown in Equation 1. The training in the third stage is holistic, training the entire backbone network with a learning rate of 1e-5 and a training process of 200 cycles. That is:

[0068] alpha*CrossAtten(point cloud, image) + (1-alpha)*CrossAtten(point cloud, cuboid).

[0069] The loss function unit consists of a mean squared error loss function, and the inputs are the noise added to the point cloud of the 3D model and the noise predicted by the backbone network.

[0070] The loss function unit includes:

[0071] Step (1): The loss function adopted is the mean squared error loss function, and the inputs are the noise predicted by the backbone network and the random noise added to the feature vector of the latent space of the point cloud of the 3D model.

[0072] Step (2): Update the model parameters using gradient descent.

[0073] S3 model application: Acquire target images, use image encoders to map target images into target feature variables, and input the target feature variables and the target cuboid sparse point cloud as conditional control inputs into the backbone network. Use samplers to denoise the input random noise. Input the denoised feature variables into the decoder to generate the final target 3D model.

[0074] The model applications include:

[0075] Step (1): Edit the images needed to generate the 3D model and the custom cuboid.

[0076] Step (2): Create random noise and use an autoencoder to map the random noise into a feature vector. The image in step (1) is mapped into an image feature vector by the image encoder. The image feature vector is used as the input of the backbone network. The image feature vector and the cuboid sparse point cloud are used as conditional feature vectors. The backbone network outputs predicted noise. The sampler is used to denoise the image encoding to obtain the denoised implicit vector.

[0077] Step (3): Decode the implicit vectors obtained in step (2) to obtain the final three-dimensional model.

[0078] In one optional embodiment of this application:

[0079] This invention includes a data processing module, the main function of which is to preprocess the input raw image. The specific steps are as follows:

[0080] Image rendering: Rendering and cropping 3D models to render images of objects from different angles.

[0081] Model sampling: Point cloud sampling at different scales is performed on the 3D model as input. The main sampling is the contour information of the model for training and learning, to ensure that the generated 3D model has an accurate appearance.

[0082] Based on the length, width, and height of the model, a cuboid sparse point cloud is defined. The sparse point cloud can be used as a conditional input to control the volume of the generated model.

[0083] Next, the present invention includes a model training module, which performs the following steps based on the generative capability of the diffusion model:

[0084] Model structure adjustment: In the pre-trained diffusion model, a full Transformer architecture model was selected to replace the previous UNET network, which can better utilize spatial and temporal information, such as... Figure 2 As shown.

[0085] Parameter optimization: After the point cloud input undergoes a self-attention module, it is then added to the cross-attention modules of the image and the cuboid point cloud. This process is divided into three stages. This strategy effectively reduces the number of parameters that need to be updated, improves training efficiency, and adjusts the balance of various conditions.

[0086] Finally, the present invention includes a model application module, the steps of which are as follows:

[0087] Model loading: Load the trained model into the system, ready for the generation of the 3D model.

[0088] Model Generation: Using a trained model, accurate 3D models are generated based on the input image and a custom cuboid. These models can be used for early-stage concept verification and design demonstration in 3D scene design, such as... Figure 3 As shown.

[0089] In summary, this invention provides a controllable method for generating 3D models. The invention comprises a data processing module, a model training module, and a model application module. The data processing module primarily samples the 3D model, sampling point cloud sets at different scales (from high to low), using these sampled point cloud sets as model input. It statistically analyzes the length, width, and height of the 3D model, quantizes them into a cuboid, and samples them as an 8-dimensional sparse point cloud as conditional input, controlling the size of the generated 3D model. The model training module replaces the traditional UNET architecture with a Transformer architecture and adds a cross-attention model. This involves fusing information from the point cloud set and the sparse point cloud of the cuboid to precisely control the size of the generated 3D model. The final model size can be controlled by the size of the cuboid. The application module uses object images as conditions to control the appearance of the generated 3D model and uses the cuboid to control its size. This invention addresses the issue of manually adjusting the proportions of 3D models to achieve the overall stereoscopic requirements of a 3D scene during modeling. This method automatically adjusts the object layout, precisely controls the length, width, and height of objects, and appropriately matches them with the scene, meeting the needs of automated scene construction.

[0090] This application provides a block diagram of a 3D model generation system. The system may include:

[0091] The data processing module is used to render and crop the original 3D model, render object images from different angles, and sample point clouds of the original 3D model at different scales to obtain the model's contour information as input for training and learning. Then, based on the length, width, and height of the original 3D model, a cuboid sparse point cloud is defined, and the sparse point cloud is used as a conditional input to control the volume of the generated model.

[0092] The model training module uses an image encoder to map generated object images from different angles into a latent space, generating latent feature vectors for the images. This latent feature vectors are then used to train the backbone network, Transformer, to obtain the trained target network. Specifically, a cross-attention module takes the point cloud of a 3D model as input and the latent feature vectors of the images and the sparse point cloud of a cuboid as conditional control inputs. The module maps the 3D model point cloud into the latent space through the encoder and adds random noise as input to the backbone network. The latent feature vectors of the images and the sparse point cloud of the cuboid are then used as conditional inputs to the backbone network for training. The backbone network outputs predicted noise, and this predicted noise, along with the previous random noise, is input into the mean squared error loss function. Gradient descent is used to guide the network's training.

[0093] The model is applied to acquire target images, which are then mapped to target feature variables using an image encoder. The target feature variables and the target cuboid sparse point cloud are input as conditional control inputs to the backbone network, and a sampler is used to denoise the random noise in the input. The denoised feature variables are then input to the decoder to generate the final target 3D model.

[0094] For specific limitations regarding a 3D model generation system, please refer to the limitations regarding a 3D model generation method described above, which will not be repeated here. Each module in the aforementioned 3D model generation system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0095] In one embodiment, an electronic device is provided, which may be a computer, and its internal structure diagram may be as follows: Figure 4 As shown, the electronic device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage medium. The database of the computer device is used for generating 3D model data. The network interface of the computer device is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a 3D model generation method.

[0096] Those skilled in the art will understand that, Figure 4The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0097] In one embodiment of this application, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the three-dimensional model generation method described above.

[0098] The computer-readable storage medium and computer program product provided in this embodiment are similar in implementation principle and technical effect to the above method embodiments, and will not be repeated here.

[0099] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in M ​​forms, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Memory Bus Direct RAM (RDRAM), Direct Memory Bus Dynamic RAM (DRDRAM), and Memory Bus Dynamic RAM (RDRAM), etc.

[0100] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0101] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A controllable method for generating three-dimensional models, characterized in that, The method includes: S1 Data Processing: Rendering and cropping the original 3D model, rendering object images from different angles, and sampling point clouds at different scales from the original 3D model to obtain the model's contour information as input for training and learning; then defining a cuboid sparse point cloud based on the length, width, and height of the original 3D model, and using the sparse point cloud as a conditional input to control the volume of the generated model. S2 Model Training: An image encoder maps generated object images from different angles into a latent space, generating latent feature vectors. The resulting backbone network, Transformer, is then trained to obtain the target network. Specifically, a 3D model point cloud is used as input, with the latent feature vectors from the image and a cuboid sparse point cloud as conditional inputs to a cross-attention module. The 3D model point cloud is mapped into the latent space via the encoder, and random noise is added as input to the backbone network. The latent feature vectors from the image and the cuboid sparse point cloud are then used as conditional inputs to the backbone network for training. The backbone network outputs predicted noise, which, along with the previously generated random noise, is input into the mean squared error loss function. Gradient descent is used to guide the network's training. S3 model application: Acquire target images, use image encoders to map target images into target feature variables, and input the target feature variables and the target cuboid sparse point cloud as conditional control inputs into the backbone network, and use samplers to denoise the input random noise; input the denoised feature variables into the decoder to generate the final target 3D model; S1 data processing specifically includes: The original 3D model is rendered to obtain a 2D image; different angles are used during rendering, and the sampled objects are mainly based on the x and y axes, while the z axis remains consistent. The original 3D models were processed into point clouds using 3D tools, and each model was uniformly processed into 100,000 point clouds. The farthest point sampling algorithm was used to sample sets of 8192, 4096, and 2048 points respectively to obtain point cloud data at different scales; Calculate the length, width, and height of the original 3D model, and then calculate the coordinates of the eight vertices of the cuboid. The coordinates of these 8 vertices are used as a sparse point cloud and then normalized.

2. The three-dimensional model generation method according to claim 1, characterized in that, The S2 model training specifically includes: The image encoder is used to encode the rendered object images in the data processing step into feature vectors. The feature map size is 768*1024, and the RGB image is converted into implicit encoding. The 3D model point cloud is mapped to latent space feature vectors through an encoder, and noise is added as input. The image encoding of the image encoder unit and the sparse point cloud of 8-dimensional vertices are used as conditional inputs. The backbone network predicts the noise and updates the network parameters using the gradient descent method. The loss function is composed of the mean squared error loss function, and the inputs are the noise added to the point cloud of the 3D model and the noise predicted by the backbone network.

3. The three-dimensional model generation method according to claim 2, characterized in that, The image encoder encodes the rendered object images from the data processing steps into feature vectors, specifically including: Establish an image encoding network and load the trained image encoder parameters into the network; The parameters of the established image encoding network are frozen. During training, a 512*512 image is encoded into a feature vector, which is used as an implicit vector. The feature map size of the implicit vector is 768*1024.

4. The three-dimensional model generation method according to claim 2, characterized in that, Adding noise as input, the image encoding from the image encoder unit and the sparse point cloud of 8-dimensional vertices are used as conditional inputs. The backbone network predicts the noise, and the network parameters are updated using the gradient descent method. Specifically, this includes: Random noise is gradually added to the latent space feature vectors; specifically, the original feature vectors are gradually added to a set of pure noise feature vectors. The generated pure noise feature vector is added to the backbone network as input, and the image encoding and cuboid point cloud are used as conditional inputs. The backbone network structure adopts a full Transformer module, each module contains one self-attention module and two cross-attention modules. The self-attention module is used for feature mapping and information interaction of the point cloud input. One of the two cross-attention modules is used for information interaction between the point cloud input and the image encoding, and the other is used for global and local interaction between the point cloud input and the cuboid point cloud input. Weights are added to balance the information interaction ratio between them. In the initial training phase, the conditional control between point cloud input and image encoding is trained first. A model that can stably control the generation of 3D model is first trained. The overall learning rate is 1e-4 for training, and the training process is 500 loops. During the fine-tuning phase of training, the cross-attention module that interacts with image encoding does not participate in the update. Instead, sparse points of a cuboid are added as conditional input to train another cross-attention module. The second phase of training mainly trains the interaction between the sparse point cloud of the cuboid and the overall point cloud, so that the vertex information of the custom cuboid can control the size of the generated model. The overall learning rate is 1e-5 for training, and the training process consists of 200 loops.

5. The three-dimensional model generation method according to claim 2, characterized in that, The loss function is composed of the mean squared error loss function. The inputs are the noise added to the 3D model point cloud and the noise predicted by the backbone network, specifically including: The loss function adopted is the mean squared error loss function, and the inputs are the noise predicted by the backbone network and the random noise added to the feature vector of the latent space of the point cloud of the 3D model, respectively. Update the model parameters using gradient descent.

6. The three-dimensional model generation method according to claim 1, characterized in that, S3 model applications also include: Edit the images and custom cuboids needed to generate the 3D model; We construct random noise and use an autoencoder to map the random noise into a feature vector. The images needed to generate the 3D model are mapped into image feature vectors through an image encoder. The image feature vectors are used as input to the backbone network. The image feature vectors and the cuboid sparse point cloud are used as conditional feature vectors. The backbone network outputs predicted noise. We use a sampler to denoise the image encoding to obtain the denoised implicit vector. The final 3D model is obtained by decoding the denoised implicit vectors.

7. A controllable three-dimensional model generation system, characterized in that, The system includes: The data processing module is used to render and crop the original 3D model, render object images from different angles, and sample point clouds of the original 3D model at different scales to obtain the model's contour information as input for training and learning. Then, based on the length, width, and height of the original 3D model, a cuboid sparse point cloud is defined, and the sparse point cloud is used as a conditional input to control the volume of the generated model. The model training module uses an image encoder to map generated object images from different angles into a latent space, generating latent feature vectors for the images. This latent feature vectors are then used to train the backbone network, Transformer, to obtain the trained target network. Specifically, a cross-attention module takes the point cloud of a 3D model as input and the latent feature vectors of the images and the sparse point cloud of a cuboid as conditional control inputs. The 3D model's point cloud is mapped into the latent space by the encoder, and random noise is added as input to the backbone network. The latent feature vectors of the images and the sparse point cloud of the cuboid are then used as conditional inputs to the backbone network for training. The backbone network outputs predicted noise, and this predicted noise, along with the previous random noise, is input into the mean squared error loss function. Gradient descent is used to guide the network's training. The model is applied to acquire target images, using an image encoder to map the target images into target feature variables, and inputting the target feature variables and the target cuboid sparse point cloud as conditional control inputs into the backbone network, and using a sampler to denoise the input random noise; the denoised feature variables are then input into the decoder to generate the final target 3D model; The data processing module specifically includes: The original 3D model is rendered to obtain a 2D image; different angles are used during rendering, and the sampled objects are mainly based on the x and y axes, while the z axis remains consistent. The original 3D models were processed into point clouds using 3D tools, and each model was uniformly processed into 100,000 point clouds. The farthest point sampling algorithm was used to sample sets of 8192, 4096, and 2048 points respectively to obtain point cloud data at different scales; Calculate the length, width, and height of the original 3D model, and then calculate the coordinates of the eight vertices of the cuboid. The coordinates of these 8 vertices are used as a sparse point cloud and then normalized.

8. An electronic device, characterized in that, It includes a memory and a processor, the memory storing a computer program that, when executed by the processor, implements the three-dimensional model generation method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the three-dimensional model generation method as described in any one of claims 1 to 6.

Images