Three-dimensional model generation method and related device

Through multimodal input information and neural network technology, the learning difficulty and time investment problems of CAD software when creating three-dimensional models are solved, and flexible and customized three-dimensional model generation is realized, improving production efficiency and product quality.

WO2024235196A1PCT designated stage expired Publication Date: 2024-11-21MOXIN (HUZHOU) TECH CO LTD

Patent Information

Application Number
PCT/CN2024/092865
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-12
Filing Date
2024-05-13
Publication Date
2024-11-21

AI Technical Summary

Technical Problem

Existing computer-aided design (CAD) software requires a steep learning curve and a lot of time investment when creating 3D models, especially for beginners to create realistic and attractive 3D animations.

Method used

By inputting information in multiple modes, such as text, speech, pictures, emotion vectors, EEG signal and odor information, the image generation neural network and the extraction network are used to convert it into a three-dimensional model, combining the Transformer structure and autoregression generation method to achieve three-dimensional Generation and editing of models.

Benefits of technology

It improves the flexibility and convenience of user interaction, supports highly customized three-dimensional model design and production, reduces human errors, and improves production efficiency and product quality.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure PCTCN2024092865-FTAPPB-I100001
    Figure PCTCN2024092865-FTAPPB-I100001
  • Figure PCTCN2024092865-FTAPPB-I100002
    Figure PCTCN2024092865-FTAPPB-I100002
  • Figure PCTCN2024092865-FTAPPB-I100003
    Figure PCTCN2024092865-FTAPPB-I100003
Patent Text Reader

Abstract

The present invention relates to the technical fields of computer graphics and computer-aided graphic designs. Provided are a three-dimensional model generation method and a related device. The method comprises: step one, a user inputting first input information of one or more modalities; step two, generating one or more matched pictures and / or texts or audios by using the first input information and by means of a picture generation neural network; step three, the user confirming the converted pictures and / or texts or audios; and step four, generating a three-dimensional model by using the pictures and / or texts or audios, or converting the pictures and / or texts or audios into point clouds, and generating a three-dimensional model by using the point clouds. In the present invention, by means of multi-modal inputs, users are allowed to describe required three-dimensional models and attributes thereof in a plurality of manners, such that the flexibility and convenience of interaction between the users and systems are improved, and the users can perform three-dimensional content creation in a familiar manner. By means of the present invention, personalized coloring and stylized processing can be accurately performed on three-dimensional models according to a description from the users, thereby improving the visual appeal and market competitiveness of final products.
Need to check novelty before this filing date? Find Prior Art

Description

A three-dimensional model generation method and related equipment Technical Field

[0001] The present invention relates to the technical field of computer graphics and computer-aided graphic design, and in particular to a three-dimensional model generation method and related equipment. Background Art

[0002] 3D models and animations are used to create realistic simulations, visualizations, and prototypes across various fields, including architecture, engineering, and entertainment. 3D models are often created using computer-aided design (CAD) software. However, creating models using CAD software has a steep learning curve and requires significant time and effort to master. Furthermore, creating complex models is challenging, especially for beginners, who struggle to create realistic and engaging 3D animations. Therefore, a readily accessible and user-friendly CAD modeling method is needed.

[0003] Summary of the Invention

[0004] In order to achieve the above object, the present invention provides the following technical solutions:

[0005] The present invention provides a three-dimensional model generation method, the method comprising:

[0006] Step 1: The user inputs the first input information of one or more modalities;

[0007] Step 2: Generate one or more matching images and / or text or audio by passing the first input information through an image generation neural network;

[0008] In the third step, the user confirms the converted image and / or text or audio;

[0009] The fourth step is to generate a three-dimensional model from the image and / or text or audio or convert the image and / or text or audio into a point cloud and generate a three-dimensional model using the point cloud.

[0010] Furthermore, when the user inputs text or voice information, the three-dimensional model generation method includes:

[0011] In the first step, the user enters text or voice information;

[0012] In the second step, the first step is converted into an image through an image generation neural network;

[0013] In the third step, the user confirms the converted image;

[0014] The fourth step is to generate a three-dimensional model from the image.

[0015] Furthermore, when the user inputs an emotion vector, an EEG signal, or odor information, the three-dimensional model generation method includes:

[0016] In the first step, the user inputs the emotion vector, EEG signal, or odor information;

[0017] In the second step, the input of the first step is converted into an image through an emotion vector or EEG signal information extraction network and a pre-trained image generation neural network;

[0018] In the third step, the user confirms the converted image. This confirmation also includes the algorithm removing the background from the image or the user selecting the background to select only the area of ​​interest to be modeled.

[0019] The fourth step is to generate a three-dimensional model from the image.

[0020] Among them, EEG data can be obtained through an electroencephalograph or similar equipment, and emotion vector data can be obtained through emotion recognition technology.

[0021] The information extraction network in the second step can map the vector to a finite number of discrete codewords by constructing a codebook consisting of a set of representative vectors (called cluster centers), and each input vector is mapped to the closest cluster center. Or it can be extracted by a neural network.

[0022] Furthermore, when the user inputs text or voice information, the three-dimensional model generation method includes:

[0023] In the first step, the user enters text or voice information;

[0024] In the second step, the input from the first step is converted into an image through an image generation neural network;

[0025] In the third step, the user confirms the converted image;

[0026] The fourth step is to convert the image into a point cloud;

[0027] The fifth step is to use the point cloud as input to generate a 3D model;

[0028] The third step of image verification also includes removing the background of the image by the algorithm or selecting the background by the user, so as to select only the area of ​​interest to be modeled;

[0029] In the fourth step, the image is converted into a point cloud by generating a Gaussian sputtering function, or by lifting an image to multiple perspectives through a multi-perspective generation neural network, and then vectorizing the image into a learnable neural network under multiple perspectives. The output value of the neural network (Transformer) is passed through a sampler to obtain a sparse point cloud. The sparse point cloud is supervised by reprojecting it onto the input image and minimizing the loss of the similarity index between the two.

[0030] Furthermore, the first input information includes text information, audio information, color or grayscale image information, thermal imaging information, depth map information, normal map, emotion vector, two-dimensional sketch, three-dimensional sketch, contour extraction, smell, EEG, point cloud, multi-perspective photo, video clip, key pair or code book, physical reading and wealth number.

[0031] Furthermore, the first input information of the multiple modalities can be converted to each other, and the conversion process can be pre-trained on other tasks.

[0032] Furthermore, before the fourth step, the confirmed images and / or texts or audios are edited.

[0033] Furthermore, in the fourth step, the image and / or text or audio is used to generate a 3D model using the trained 3D model generation network;

[0034] The specific training process of the three-dimensional model generation network is as follows:

[0035] (1) Pre-training a neural network by using a large number of training 3D models to obtain a pre-trained encoder and decoder. The specific process is as follows: using a large number of training 3D models as input, the pre-trained encoder compresses the large number of 3D models into a latent space, and the pre-trained decoder restores the original 3D model, wherein each 3D model is compressed into the first eigenvector z in the latent space by the pre-trained encoder;

[0036] Among them, the decoder D T Decode the feature vector z to obtain a set of query points in the encoding table, then sample a grid with a resolution of 128^3 and reconstruct the 3D model using the Marching Cubes algorithm;

[0037] Reconstruct the three-dimensional model X', the specific expression is: X'=D τ (VQ(z))

[0038] Among them, VQ is the mapping of the first eigenvector z to the closest element in the coding table;

[0039] The decoder is a Transformer structure using an L-layer self-attention module (SelfAttn) where SelfAttn(A) = CrossAttn(A,A)

[0040] Then we get an interpolation in The result of this interpolation is input into a fully connected layer to output an occupancy value Occupancy O, where the occupancy value can be converted to a three-dimensional model.

[0041] (1.3) Optimizing the pre-trained encoder and pre-trained decoder through the optimization function;

[0042] The optimization function is:

[0043] Among them, BCE() is the binary cross entropy loss function;

[0044] is the placeholder value predicted by the network;

[0045] is the placeholder value of the real sample;

[0046] In the x sample set, and The mathematical expectation value of the binary cross entropy value; (2) the user inputs a second input information of a modality, and the second input information is encoded into a second feature vector in the latent space by the second encoder Wherein, preferably, the second input information includes text information, audio information, color or grayscale image information, thermal image information, depth map information, normal map, emotion vector, two-dimensional sketch, three-dimensional sketch, contour extraction, smell, EEG, point cloud, multi-view photos, video clips, key pairs or code books, physical readings and wealth numbers;

[0047] The second encoder can be pre-trained on other tasks. For example, when extracting emotion vectors, odors, or EEG information, the encoder can be pre-trained on the recognition task. Alternatively, unlabeled training can be performed using multimodal alignment.

[0048] (3) The second eigenvector The third eigenvector is obtained by attaching it to a first eigenvector z randomly selected from the vector space obtained in pre-training. The fourth eigenvector is generated by the third eigenvector through a vector generation network.

[0049] (4) Input the fourth eigenvector into the pre-trained neural network in step (1) to obtain a three-dimensional model.

[0050] The second feature vector may be obtained by using multiple different encoders, or may be obtained by using different inputs through different encoders, and the multiple feature vectors are attached to the first feature vector in the same manner.

[0051] (5) Continuously optimize the vector generation network so that the distribution of the generated three-dimensional model is close to the distribution of the three-dimensional model in the data set. The loss function used to optimize the neural network during the training process is: the first eigenvector z is continuously added with different Gaussian noises T steps to obtain Z t , and then use a time-constrained 3D UNet network to Z t Perform denoising,

[0052] Where θ represents the parameters of the conditional model in the diffusion model (3D UNet network);

[0053] Represents the input information of the i-th modal input by the user;

[0054] ∈ represents the unconditional model in the diffusion model (3D UNet network);

[0055] C represents the multimodal information input by the user;

[0056] t is the timestamp during the diffusion process;

[0057] is the second eigenvector;

[0058] ∈ θ () is the 3D UNet network function;

[0059] D is the eigenvector of the latent space The dropout operation, F is the concatenation operation;

[0060] Furthermore, the compression of the 3D model into a latent space in step (1.1) is achieved by using a vector quantized variable autoencoder. The implementation expression is:

[0061] Where X refers to the T-SDF representation of the 3D model to be trained; Where D is the dimension of the 3D model to be trained, d is the dimension of the first eigenvector z, and d <D;

[0062] or,

[0063] The compression is achieved through a cross-attention mechanism Cross Attention (CrossAttn), which is expressed as:

[0064] Where X is the point cloud, triangle patch, density, occupancy, hidden plane or spline surface representation of the 3D model to be trained;

[0065] PosEmb() is a column-by-column position embedding function;

[0066] C is the number of channels;

[0067] M represents the number of learnable query vectors in the L set;

[0068] In addition, the first eigenvector z can be further compressed. Specifically, the mean and variance of each layer of eigenvectors are obtained through two fully connected neural networks.

[0069] Among them, fi represents the feature vector of the i-th layer;

[0070] u i,j represents the average value of the jth network channel of the i-th layer feature vector;

[0071] represents the variance of the jth network channel of the i-th layer feature vector;

[0072] The fully connected neural network further compresses the number of channels C to C0, where C0 is much smaller than C. Then, the loss function normalized by KL divergence is calculated:

[0073] Furthermore, the execution principle of the three-dimensional model generation network is:

[0074] Based on the Transformer structure distribution p θ (z i |O) is modeled, taking an arbitrary set O of observed latent variables at known locations and predicting the categorical distribution at any query location i; the model is learned by simply maximizing the log-likelihood of the encoded latent representation, using random order for autoregressive generation, and a non-sequential autoregressive network to model the distribution of the latent variable Z, mapping it to the complete three-dimensional shape X = Dy(Z).

[0075] Furthermore, in step (3), the attachment of the second eigenvector to the first eigenvector can be achieved through a cross-attention algorithm; or through a soft-attention algorithm; or by concatenating the second eigenvector onto the first eigenvector after a convolution operation.

[0076] When attachment is achieved through the cross-attention algorithm, the specific process is as follows:

[0077] The second eigenvector Recorded as S1, the first eigenvector z is recorded as S2, then the calculation formula of the cross-attention algorithm is: sofmax((W Q S2)(W K S1) T )W V S1);

[0078] Among them, W Q is the weight matrix of the query;

[0079] W K is the weight matrix of the key;

[0080] W V is the weight matrix of value;

[0081] 3.1 Calculate K and V of the second eigenvector S1; K = W K S1; V = W V S1

[0082] 3.2 Calculate the Q of the first eigenvector S2

[0083] Q=W Q S2;3.3 Calculate the attention matrix based on K and Q: sofmax((W Q S2)(W V S1);

[0084] 3.4 Apply V to the attention matrix sofmax((W Q S2)(W V S1);

[0085] Furthermore, the method further includes performing scaling, displacement, coloring and / or stylization operations on the generated three-dimensional model.

[0086] The specific steps for coloring a 3D model are:

[0087] (6.11) receiving a user's textual, visual, or audio description of the three-dimensional model, wherein the description is in the form of a user-desired sketch text, a user-desired style image, or a user-speaking or machine-generated audio description of the sketch;

[0088] (6.12) Encode the colored description into a vector A1 in a high-dimensional space through a pre-trained encoder;

[0089] (6.13) storing each vertex of the generated three-dimensional model in a colored neural radiance field by performing a position encoding, and then outputting a color vector for each vertex position of the three-dimensional model; the colored neural radiance field is a multi-layer perceptron containing color information;

[0090] (6.14) Add the color vector of each vertex position of the three-dimensional model to the generated three-dimensional model;

[0091] (6.15) Optimize the colored three-dimensional model.

[0092] The specific steps for stylizing a 3D model are:

[0093] (6.21) receiving a user's stylized description of the three-dimensional model in the form of a textual description, an image, or an audio description of the three-dimensional model, wherein the stylized description is in the form of a textual description of a sketch desired by the user, an image of a style desired by the user, or an audio description of the sketch read aloud by the user or generated by a machine;

[0094] (6.22) Encode the stylized description into a vector A2 in a high-dimensional space through a pre-trained encoder;

[0095] (6.23) storing each vertex of the three-dimensional model generated by the sketch in a stylized neural radiance field by performing a position encoding, and then outputting the displacement of the normal of each vertex position of the three-dimensional model; the stylized neural radiance field is a multi-layer perceptron that contains displacement information;

[0096] (6.24) Add the normal displacement of each vertex position of the three-dimensional model to the generated three-dimensional model,

[0097] (6.25) Optimize the stylized three-dimensional model.

[0098] Furthermore, the method also includes performing scaling and displacement modification operations on different areas or specific corner points of the generated three-dimensional model.

[0099] The present invention also provides a parts processing method,

[0100] The first step is to generate a three-dimensional model using the above-mentioned three-dimensional model generation method;

[0101] The second step is to edit and modify the 3D model;

[0102] The third step is to input the generated 3D model into the parser and convert it into machine-readable instructions;

[0103] The fourth step is that the processing device performs processing according to machine-readable instructions.

[0104] The present invention also provides a parts processing device, which is used for the above-mentioned processing method. The parts processing device includes a motion component, an actuator, and a controller; the controller receives input of modal information, runs the above-mentioned three-dimensional model generation method, and processes to obtain a three-dimensional model. The combination of the motion component and the actuator is a 3D printer or a CNC engraving machine.

[0105] The present invention also provides an electronic terminal, which executes the above-mentioned three-dimensional model generation method.

[0106] The present invention has the following beneficial effects:

[0107] (1) Enhanced multimodal interaction: Through multimodal input, the present invention allows users to describe the required three-dimensional model and its attributes in multiple ways, which increases the flexibility and convenience of user interaction with the system and enables users to create three-dimensional content in a familiar way.

[0108] (2) Innovative coloring and stylization technology: By utilizing the coloring neural radiation field and the stylized neural radiation field, the present invention can accurately perform personalized coloring and stylization on the three-dimensional model according to the user's description, thereby improving the visual appeal and market competitiveness of the final product.

[0109] (3) Highly customized design and production: Users can specify the surface shape, color, style, and other features of the 3D model in great detail, so that the generated 3D model is highly consistent with the specific needs of individuals or markets.

[0110] (4) Improve production efficiency and accuracy: The generated three-dimensional model is directly converted into machine-readable instructions, making the processing process highly automated, reducing human errors, improving production efficiency and product quality, and being applicable to various types of manufacturing equipment. DETAILED DESCRIPTION

[0111] The specific implementation methods of the present invention are described in detail below with reference to the examples. It should be pointed out that the examples are only specific explanations of the invention and should not be regarded as limitations of the invention. The purpose of the examples is to enable those skilled in the art to better understand and reproduce the technical solutions of the present invention. The scope of protection of the present invention shall still be based on the scope defined by the claims.

[0112] Example 1

[0113] This embodiment provides a method for generating a three-dimensional model, the method comprising:

[0114] Step 1: The user inputs first input information in one or more modalities, wherein the first input information includes text information, audio information, color or grayscale image information, thermal imaging information, depth map information, normal map, emotion vector, two-dimensional sketch, three-dimensional sketch, contour extraction, smell, EEG, point cloud, multi-view photos, video clips, key pairs or code books, physical readings and wealth numbers.

[0115] The first input information of the multiple modalities can be converted into each other;

[0116] Step 2: Generate one or more matching images and / or text or audio by passing the first input information through an image generation neural network;

[0117] In the third step, the user confirms the converted image and / or text or audio;

[0118] The fourth step is to generate a three-dimensional model from the image and / or text or audio or convert the image and / or text or audio into a point cloud and use the point cloud to generate a three-dimensional model.

[0119] The method also includes editing the confirmed pictures and / or text or audio, wherein the editing includes regenerating the entire picture and / or text or modifying a part of the picture.

[0120] Example 2

[0121] This embodiment provides a three-dimensional model generation method. When a user inputs text or voice information, the three-dimensional model generation method includes:

[0122] In the first step, the user enters text or voice information;

[0123] In the second step, the first step is converted into an image through an image generation neural network;

[0124] In the third step, the user confirms the converted image;

[0125] The fourth step is to generate a three-dimensional model from the image.

[0126] The method also includes editing the confirmed pictures and / or text or audio, wherein the editing includes regenerating the entire picture and / or text or modifying a part of the picture.

[0127] Example 3

[0128] This embodiment provides a three-dimensional model generation method. When a user inputs an emotion vector, an EEG signal, or odor information, the three-dimensional model generation method includes:

[0129] In the first step, the user inputs the emotion vector, EEG signal, or odor information;

[0130] In the second step, the input of the first step is converted into an image through an emotion vector or EEG signal information extraction network and a pre-trained image generation neural network;

[0131] In the third step, the user confirms the converted image. This confirmation also includes the algorithm removing the background from the image or the user selecting the background to select only the area of ​​interest to be modeled.

[0132] The fourth step is to generate a three-dimensional model from the image.

[0133] This also includes editing the confirmed images and / or text or audio, and the editing includes completely regenerating or modifying a portion of the area.

[0134] Example 4

[0135] This embodiment provides a three-dimensional model generation method. When a user inputs text or voice information, the three-dimensional model generation method includes:

[0136] In the first step, the user enters text or voice information;

[0137] In the second step, the input from the first step is converted into an image through an image generation neural network;

[0138] In the third step, the user confirms the converted image;

[0139] The fourth step is to convert the image into a point cloud;

[0140] The fifth step is to use the point cloud as input to generate a 3D model;

[0141] This also includes editing the confirmed images and / or text or audio, and the editing includes completely regenerating or modifying a portion of the area.

[0142] The third step of image verification also includes removing the background of the image by the algorithm or selecting the background by the user, so as to select only the area of ​​interest to be modeled;

[0143] The fourth step converts the image into a point cloud by generating a Gaussian sputtering function, or by transforming an image to multiple viewpoints through a multi-view neural network. The image is then vectorized and fed into a learnable neural network at these multiple viewpoints. The output of the neural network (Transformer) is passed through a sampler to generate a sparse point cloud. This sparse point cloud is then reprojected onto the input image, minimizing the loss of similarity metrics between the two for supervision. In this fourth step, the point cloud can be edited and viewed in 3D.

[0144] Preferably, the conversion process in the fourth step is performed in real time, and the conversion result is displayed in real time.

[0145] Preferably, after the fifth step, the model is continuously optimized based on the inputs from the previous steps.

[0146] Example 5

[0147] This embodiment provides a training method for a three-dimensional model generation network, which is used in Embodiments 1 to 3 to generate a three-dimensional model from user input or to convert user input into a point cloud.

[0148] The specific training process of the three-dimensional model generation network is as follows:

[0149] (1) Pre-training a neural network by using a large number of training 3D models to obtain a pre-trained encoder and decoder. The specific process is as follows: using a large number of training 3D models as input, the pre-trained encoder compresses the large number of 3D models into a latent space, and the pre-trained decoder restores the original 3D model, wherein each 3D model is compressed into the first eigenvector z in the latent space by the pre-trained encoder;

[0150] Among them, the decoder D T Decode the feature vector z to obtain a set of query points in the encoding table, then sample a grid with a resolution of 128^3 and reconstruct the 3D model using the Marching Cubes algorithm;

[0151] Reconstruct the three-dimensional model X', the specific expression is: X'=D τ (VQ(z))

[0152] Among them, VQ is the mapping of the first eigenvector z to the closest element in the coding table;

[0153] The decoder is a Transformer structure using an L-layer self-attention module (SelfAttn) where SelfAttn(A) = CrossAttn(A,A)

[0154] Then we get an interpolation in The interpolation result is input into a fully connected layer to output an occupancy value Occupancy O.

[0155] (1.3) Optimizing the pre-trained encoder and pre-trained decoder through the optimization function;

[0156] The optimization function is:

[0157] Among them, BCE() is the binary cross entropy loss function;

[0158] is the placeholder value predicted by the network;

[0159] is the placeholder value of the real sample;

[0160] In the x sample set, and The mathematical expectation of the binary cross entropy value;

[0161] (2) The user inputs a second input information of a modality, and the second input information is encoded into a second feature vector in the latent space by the second encoder Wherein, preferably, the second input information includes text information, audio information, color or grayscale image information, thermal image information, depth map information, normal map, emotion vector, two-dimensional sketch, three-dimensional sketch, contour extraction, smell, EEG, point cloud, multi-view photos, video clips, key pairs or code books, physical readings and wealth numbers;

[0162] The second encoder can be pre-trained on other tasks. For example, when extracting emotion vectors, odors, or EEG information, the encoder can be pre-trained on the recognition task. Alternatively, unlabeled training can be performed using multimodal alignment.

[0163] (3) The second eigenvector The third eigenvector is obtained by attaching it to a first eigenvector z randomly selected from the vector space obtained in pre-training. The fourth eigenvector is generated by the third eigenvector through a vector generation network.

[0164] (4) inputting the fourth eigenvector into the pre-trained neural network in step (1) to obtain a three-dimensional model;

[0165] (5) Continuously optimize the vector generation network so that the distribution of the generated three-dimensional model is close to the distribution of the three-dimensional model in the data set. The loss function used to optimize the neural network during the training process is: the first eigenvector z is continuously added with different Gaussian noises T steps to obtain Z t , and then use a time-constrained 3D UNet network to Z t Perform denoising,

[0166] Where θ represents the parameters of the conditional model in the diffusion model (3D UNet network);

[0167] Represents the input information of the i-th modal input by the user;

[0168] ∈ represents the unconditional model in the diffusion model (3D UNet network);

[0169] C represents the multimodal information input by the user;

[0170] t is the timestamp during the diffusion process;

[0171] is the second eigenvector;

[0172] ∈ θ () is the 3D UNet network function;

[0173] D is the eigenvector of the latent space The dropout operation, F is the concatenation operation;

[0174] In step (1.1), the 3D model is compressed into a latent space by using a vector quantized variable autoencoder. The implementation expression is:

[0175] Where X refers to the T-SDF representation of the 3D model to be trained; Where D is the dimension of the 3D model to be trained, d is the dimension of the first eigenvector z, and d <D;

[0176] or,

[0177] The compression is achieved through a cross-attention mechanism Cross Attention (CrossAttn), which is expressed as:

[0178] Where X is the point cloud, triangle patch, density, occupancy, hidden plane or spline surface representation of the 3D model to be trained;

[0179] PosEmb() is a column-by-column position embedding function;

[0180] C is the number of channels;

[0181] M represents the number of learnable query vectors in the L set;

[0182] In addition, the first eigenvector z can be further compressed. Specifically, the mean and variance of each layer of eigenvectors are obtained through two fully connected neural networks.

[0183] Among them, fi represents the feature vector of the i-th layer;

[0184] u i,j represents the average value of the jth network channel of the i-th layer feature vector;

[0185] represents the variance of the jth network channel of the i-th layer feature vector;

[0186] The fully connected neural network further compresses the number of channels C to C0, where C0 is much smaller than C. Then, the loss function normalized by KL divergence is calculated:

[0187] The execution principle of the three-dimensional model generation network is:

[0188] Based on the Transformer structure distribution p θ (z i |O) is modeled, taking an arbitrary set O of observed latent variables at known locations and predicting the categorical distribution at any query location i; the model is learned by simply maximizing the log-likelihood of the encoded latent representation, using random order for autoregressive generation, and a non-sequential autoregressive network to model the distribution of the latent variable Z, mapping it to the complete three-dimensional shape X = Dy(Z).

[0189] Furthermore, in step (3), the attachment of the second eigenvector to the first eigenvector can be achieved through a cross-attention algorithm; or through a soft-attention algorithm; or by concatenating the second eigenvector onto the first eigenvector after a convolution operation.

[0190] When attachment is achieved through the cross-attention algorithm, the specific process is as follows:

[0191] The second eigenvector Recorded as S1, the first eigenvector z is recorded as S2, then the calculation formula of the cross-attention algorithm is: sofmax((W Q S2)(W K S1) T )W V S1);

[0192] Among them, W Q is the weight matrix of the query;

[0193] W K is the weight matrix of the key;

[0194] W V is the weight matrix of value;

[0195] 3.1 Calculate K and V of the second eigenvector S1; K = W K S1; V = W V S1

[0196] 3.2 Calculate the Q of the first eigenvector S2

[0197] Q=W Q S2;3.3 Calculate the attention matrix based on K and Q: sofmax((W Q S2)(W V S1);

[0198] 3.4 Apply V to the attention matrix sofmax((W Q S2)(W V S1)).

[0199] Furthermore, the second feature vector may be obtained by multiple different encoders, or may be obtained by different inputs through different encoders, and the multiple feature vectors are attached to the first feature vector in the same way.

[0200] Furthermore, the vector generation network can be a neural network with a VQ-VAE structure, or a generation adversarial model, or a discrete diffusion model.

[0201] When the image generation network is a VQ-VAE neural network, the specific process of generating the fourth eigenvector is as follows:

[0202] First, K1 vectors are set as a queryable encoding table. A randomly initialized image is input and passed through the encoder CNN to obtain N1 intermediate representations. Then, the nearest neighbor algorithm is used to query the encoding table for the vector that is most similar to these N1 intermediate representations. At the same time, an autoregressive model is trained to convert the input information into the representations in the encoding table. During prediction, the autoregressive model outputs the intermediate vector corresponding to the input and places the similar vector queried in the encoding table in the corresponding position.

[0203] The autoregressive model may be a neural network with a Transformer structure.

[0204] When the image generation network is an adversarial generation model, the specific process of generating the fourth eigenvector is:

[0205] First, set K2 vectors as a queryable encoding table, input a randomly initialized vector through the encoder CNN to obtain N2 intermediate representations, and then use the nearest neighbor algorithm to query the encoding table for the vector most similar to these N2 intermediate representations.

[0206] Example 6

[0207] This embodiment provides a three-dimensional model generation method, which is used in Embodiments 1 to 3 to generate a three-dimensional model from a picture and / or text or audio;

[0208] The specific steps are:

[0209] The first step is to receive an image and / or text or audio. The image and / or text or audio can be directly converted or edited by the user. A text, image, or voice description of the image and / or text or audio can also be input. Specifically, this can be a user-desired image and / or text or audio text, such as "ALEGO-style airplane"; a human- or machine-generated voice describing the image; or a user-desired image of a desired style, such as a picture of a LEGO airplane. Alternatively, this can be a computer-generated description of the image, which can be generated from a fixed template, such as "a [object]"; and a description of the image category can be included to reduce the number of web search domains for generating three-dimensional models from the image, thereby obtaining the desired results more quickly. For example, if the user desires a sketch describing the class of airplane, the text description T can be "an airplane." The class can be manually specified by the user or obtained by a neural network image classifier through a pre-training method. Specifically, the neural network image classifier can be composed of the following structure: an image encoder, including a down-sampling convolutional neural network, and a classifier, including several fully connected neural networks (MLPs), with an activation function between each layer, such as a ReLU activation function; the neural network image classifier is trained on categories with category identifiers and corresponding images. By specifying the image category, the network search domain for generating a three-dimensional model from the image can be reduced, thereby obtaining the desired result more quickly.

[0210] As a preferred technical solution, the description can be explained based on the image rendered by the three-dimensional model through the renderer. For example, if the objects rendered by the rendering module are all black outlines, the description of the picture can be "a black [object]" (a black [name of the category to which the sketch belongs]); if the object rendered by the rendering module is a gray projection image, the description of the picture can be "a grey [object]" (a gray [name of the category to which the sketch belongs]).

[0211] In the second step, an encoder is used to extract features from the image or speech to obtain a compressed vector Z. The encoder can be a convolutional neural network, including one or more convolutional layers, or the encoder portion of a residual neural network with a ResNet structure.

[0212] In the third step, the decoder uses the vector Z to reconstruct or generate a 3D model. The decoder D can be a multi-layer perceptron with n outputs, which are the positions of each vertex or facet. It can also generate a model based on a specific template. The template can be a general template, such as a spherical 3D model, or a 3D model template designed or acquired for a certain category. For example, if the user wants the image or voice model to be a car, then the 3D model of a car already in the dataset can be used as the template. Alternatively, if the user wants the image or voice model to be a human body, then the average shape of all human body data in the dataset can be used as the template.

[0213] Preferably, key point extraction can be performed on the universal template. This involves extracting key control points that influence shape variation from all corner points of the model. This can be achieved through manual annotation or by automatically learning the model within the dataset and selecting the K points with the largest displacement range. During the learning process, regression can be used to determine the spatial displacement range of the control points within each template. This range distribution is then applied to the decoder as a conditional constraint, forcing the decoder to produce more natural results without generating excessively variable, abnormal results.

[0214] The fourth step is to optimize the generated 3D model using a pre-trained neural network aligned to the same latent space in multiple modalities as additional information input, wherein the multiple modalities include at least the modality corresponding to the user input information and the image or image modality, specifically:

[0215] (4.41) Rendering the 3D model using a differentiable renderer to obtain a rendered image, and calculating the similarity (IoU) between the rendered image and the image; the rendered image is a given perspective, and the 3D model can obtain a perspective pi (i=1, 2, ...V). As a preferred technical solution, the sampling of the rotation angle of the 3D model can adopt Gaussian distribution sampling or average sampling; the differentiable renderer can be a SoftRas renderer;

[0216] (4.42) The rendering image is converted into a feature vector Z1 through an encoder E1, and the user input is converted into a feature vector Z2 through another encoder E2. The similarity Cos between the feature vector Z1 and the feature vector Z2 is calculated.<Z1,Z2> ; The cosine similarity factor Cos is used for similarity calculation. If there is more than one rendering image P, the same operation is performed on each image Pi, and the obtained cosine similarity factors are averaged to obtain the Cos value. Encoder E1 and encoder E2 come from encoders in a multimodal model that has been pre-trained on a large number of text and image pairs. When the input T is text, E2 is a text encoder, and when the input T is a picture, E2 is a picture encoder. Large-scale pre-training of E1 and E2 can map text and / or picture inputs to a shared feature space, calculate the similarity in this shared feature space, and minimize the similarity by training the neural network, so that the result rendered by the rendering module can match the text description T or the image input I as much as possible.

[0217] The multimodal model can be an existing CLIP (Contrastive Language-Image Pretraining) model or a GLIP (Grounded Language-Image Pretraining) model. Taking the CLIP model as an example, it is pre-trained using paired image-text pairs (for example, if the input is a picture of a dog, the corresponding text also indicates that it is a dog). These text and image are respectively output through the CLIP text encoder and the CLIP image encoder. Contrastive learning is then performed on these output text features and image features, and the learned text encoder and image encoder share the same feature space.

[0218] During the rendering process, multi-view sampling is performed. The selection of the view can be uniform sampling within a range of view angles or Gaussian distribution sampling. In addition, the image can be rotated and randomly cropped. The optimizer can be an Adam optimizer or an SGD optimizer.

[0219] 4.43) The similarity calculated in step (4.42) and step (4.41) is used to obtain the loss function L, and the similarity Cos in step (4.42) is used to obtain the loss function L.<Z1,Z2> Take the negative sign and add one to the similarity IoU in step (4.41) for weighted addition to get the loss function L. The specific expression is: L = l(1-Cos<Z1,Z2> ) + kIoU, where l and k are manually set parameters, with l = k = 0.1. By inputting a large number of samples and using an optimizer to adjust the parameters of the encoder E, this loss function is minimized. Preferably, training can be performed using data pairs from only one item class, or a mixture of multiple item classes.

[0220] You can use data pairs from only one item class for training, or you can use a mixture of multiple item classes for training.

[0221] As a preferred technical solution, the loss function L also includes a Laplacian Flatten loss.

[0222] As a preferred technical solution, the loss function L also includes a model symmetry loss Lvsym, and the calculation formula of the symmetric loss is:

[0223] Among them, v i are the vertices for generating the three-dimensional model,

[0224] v j It is to generate the j-th vertex of the three-dimensional model;

[0225] N is the total number of vertices in the generated 3D model

[0226] T is the transformation to generate the 3D model

[0227] n is the unit vector along the normal direction; I is the image.

[0228] As a preferred technical solution, the loss function L also includes a model symmetry loss LIsym, which is as follows:

[0229] in, is the view in step (4.41), It is a rendering of the projection rendering of the three-dimensional model after the transformation T;

[0230] γ() i,k It is a manually set hyperparameter, which can also be left unset;

[0231] j represents the horizontal coordinate of the rendering of the three-dimensional model; k represents the vertical coordinate of the rendering of the three-dimensional model;

[0232] m represents the total number of given view angles.

[0233] In some preferred embodiments, the generated 3D model is rendered in a uniformly sampled interval, and the similarity is calculated. The perspective with the highest similarity is selected as the anchor perspective, and then a Gaussian distribution centered on the anchor perspective is used to render multiple perspectives from randomly sampled views.

[0234] Example 7

[0235] This embodiment provides a method for post-processing a three-dimensional model, which is used to perform scaling, displacement, coloring and / or stylization operations on the three-dimensional model generated in Embodiments 1 to 4.

[0236] The specific steps for coloring the three-dimensional model are as follows:

[0237] (6.11) receiving a user's textual, pictorial, or audio description of the three-dimensional model, wherein the description is in the form of a textual description of the image desired by the user, a style diagram desired by the user, or a user-speaking or machine-generated audio description of the image;

[0238] (6.12) Encode the colored description into a vector A1 in a high-dimensional space through a pre-trained encoder;

[0239] (6.13) storing each vertex of the generated three-dimensional model in a colored neural radiance field by performing a position encoding, and then outputting a color vector for each vertex position of the three-dimensional model; the colored neural radiance field is a multi-layer perceptron containing color information;

[0240] (6.14) Add the color vector of each vertex position of the three-dimensional model to the generated three-dimensional model;

[0241] (6.15) Optimize the colored three-dimensional model.

[0242] The specific steps for stylizing the 3D model are as follows:

[0243] (6.21) receiving a user's stylized description of the three-dimensional model in the form of a text, image, or voice message, wherein the stylized description is in the form of a text message of a desired image, a picture of a desired style, or a user-speaking or machine-generated voice message describing the picture;

[0244] (6.22) Encode the stylized description into a vector A2 in a high-dimensional space through a pre-trained encoder;

[0245] (6.23) storing each vertex of the three-dimensional model generated from the image in a stylized neural radiance field by performing a position encoding, and then outputting the normal displacement of each vertex position of the three-dimensional model; the stylized neural radiance field is a multi-layer perceptron that contains displacement information;

[0246] (6.24) Add the normal displacement of each vertex position of the three-dimensional model to the generated three-dimensional model,

[0247] (6.25) Optimize the stylized three-dimensional model.

[0248] The method further includes performing scaling and displacement modification operations on different regions or specific corner points of the generated three-dimensional model.

[0249] Example 8

[0250] This embodiment provides a parts processing method, including:

[0251] The first step is to generate a three-dimensional model using any of the three-dimensional model generation methods of Examples 1-4;

[0252] The second step is to edit and modify the 3D model;

[0253] The third step is to input the generated 3D model into the parser and convert it into machine-readable instructions;

[0254] The fourth step is that the processing device performs processing according to machine-readable instructions.

[0255] Example 9

[0256] In this embodiment, a parts processing device is provided, which is used for the above-mentioned processing method. The parts processing device includes a motion component, an actuator, and a controller; the controller receives input of modal information, runs any three-dimensional model generation method of any one of embodiments 1-4, and processes to obtain a three-dimensional model. The combination of the motion component and the actuator is a 3D printer or a CNC engraving machine.

[0257] Example 10

[0258] This embodiment provides an electronic terminal that executes any one of the three-dimensional model generation methods of embodiments 1-4.

[0259] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application.

[0260] It should be noted that the technical features not described in detail in the present invention can be implemented by any existing technology.

Claims

1. A three-dimensional model generation method, characterized in that: The method comprises: Step 1: The user inputs the first input information of one or more modalities; Step 2: Generate one or more matching images and / or text or audio by passing the first input information through an image generation neural network; In the third step, the user confirms the converted image and / or text or audio; in the fourth step, the image and / or text or audio is used to generate a three-dimensional model or the image and / or text or audio is converted into a point cloud and the point cloud is used to generate a three-dimensional model.

2. A three-dimensional model generation method according to claim 1, characterized in that: In the first step, the user enters text or voice information; In the second step, the first step is converted into an image through an image generation neural network; In the third step, the user confirms the converted image; The fourth step is to generate a three-dimensional model from the image.

3. A three-dimensional model generation method according to claim 1, characterized in that: In the first step, the user inputs the emotion vector, EEG signal, or odor information; In the second step, the input of the first step is converted into an image through an emotion vector or EEG signal information extraction network and a pre-trained image generation neural network; In the third step, the user confirms the converted image. This confirmation also includes the algorithm removing the background from the image or the user selecting the background to select only the area of ​​interest to be modeled. The fourth step is to generate a three-dimensional model from the image.

4. A three-dimensional model generation method according to claim 1, characterized in that: In the first step, the user enters text or voice information; In the second step, the input from the first step is converted into an image through an image generation neural network; In the third step, the user confirms the converted image; The fourth step is to convert the image into a point cloud; The fifth step is to use the point cloud as input to generate a 3D model; The third step of image verification also includes removing the background of the image by the algorithm or selecting the background by the user, so as to select only the area of ​​interest to be modeled; In the fourth step, the image is converted into a point cloud by generating a Gaussian sputtering function, or by lifting an image to multiple perspectives through a multi-perspective generation neural network, and then vectorizing the image into a learnable neural network under multiple perspectives. The output value of the neural network (Transformer) is passed through a sampler to obtain a sparse point cloud. The sparse point cloud is supervised by reprojecting it onto the input image and minimizing the loss of the similarity index between the two.

5. A three-dimensional model generation method according to claim 1, characterized in that: The first input information includes text information, audio information, color or grayscale image information, thermal imaging information, depth map information, normal map, emotion vector, two-dimensional sketch, three-dimensional sketch, contour extraction, smell, EEG, point cloud, multi-view photos, video clips, key pairs or code books, physical readings and wealth numbers.

6. A three-dimensional model generation method according to claim 5, characterized in that: The first input information of the multiple modalities can be converted into each other.

7. A three-dimensional model generation method according to claim 1, characterized in that: The method includes: before the fourth step, editing the confirmed picture and / or text or audio.

8. A three-dimensional model generation method according to claim 1, 2 or 3, characterized in that: Generate a 3D model from the image and / or text or audio using a trained 3D model generation network; The specific training process of the three-dimensional model generation network is as follows: (1) Pre-training a neural network by training a large number of three-dimensional models, thereby obtaining pre-trained The encoder and decoder of the proposed method are as follows: a large number of trained 3D models are used as input, the pre-trained encoder compresses the large number of 3D models into a latent space, and the pre-trained decoder restores the original 3D model, wherein each 3D model is compressed into the first eigenvector z in the latent space by the pre-trained encoder; (2) The user inputs a second input information of a modality, and the second input information is encoded into a second feature vector in the latent space by the second encoder (3) The second eigenvector The third eigenvector is obtained by attaching it to a first eigenvector z randomly selected from the vector space obtained in pre-training. The fourth eigenvector is generated by the third eigenvector through a vector generation network. (4) inputting the fourth eigenvector into the pre-trained neural network in step (1) to obtain a three-dimensional model; (5) Continuously optimize the vector generation network so that the distribution of the generated three-dimensional model is close to the distribution of the three-dimensional model in the dataset.

9. A three-dimensional model generation method according to claim 8, characterized in that: In step (1), the 3D model is compressed into a latent space by using a vector quantized variable autoencoder. The implementation expression is: Where X refers to the T-SDF representation of the 3D model to be trained; Where D is the dimension of the 3D model to be trained, d is the dimension of the first eigenvector z, and d <D; or, The compression is achieved through a cross-attention mechanism Cross Attention (CrossAttn), which is expressed as: Among them, X refers to the point cloud, triangle patch, density, occupancy rate, hidden plane of the 3D model to be trained or spline surface representation; PosEmb() is a column-by-column position embedding function; C is the number of channels; M represents the number of learnable query vectors in the L set; In addition, the first eigenvector z can be further compressed. Specifically, the mean and variance of each layer of eigenvectors are obtained through two fully connected neural networks. Among them, fi represents the feature vector of the i-th layer; u i,j represents the average value of the jth network channel of the i-th layer feature vector; represents the variance of the jth network channel of the i-th layer feature vector; The fully connected neural network further compresses the number of channels C to C0, where C0 is much smaller than C. Then, the loss function normalized by KL divergence is calculated:

10. A three-dimensional model generation method according to claim 1, characterized in that: The execution principle of the three-dimensional model generation network is: Based on the Transformer structure distribution p θ (z i |O) is modeled, taking an arbitrary set O of observed latent variables at known locations and predicting the categorical distribution at any query location i; the model is learned by simply maximizing the log-likelihood of the encoded latent representation, using random order for autoregressive generation, and a non-sequential autoregressive network to model the distribution of the latent variable Z, mapping it to the complete three-dimensional shape X = Dy(Z).

11. A three-dimensional model generation method according to claim 8, characterized in that: In step (3), the attachment of the second eigenvector to the first eigenvector may be achieved through a cross-attention algorithm; or through a soft-attention algorithm; or by concatenating the second eigenvector onto the first eigenvector after a convolution operation.

12. A three-dimensional model generation method according to claim 1, 2, 3 or 4, characterized in that: The method further includes performing scaling, displacement, coloring and / or stylization operations on the generated three-dimensional model.

13. A three-dimensional model generation method according to claim 1, 2, 3 or 4, characterized in that: The method further includes performing scaling and displacement modification operations on different regions or specific corner points of the generated three-dimensional model.

14. A parts processing method, characterized in that: The first step is to generate a three-dimensional model using the method described in any one of claims 1 to 13; The second step is to edit and modify the 3D model; The third step is to input the generated 3D model into the parser and convert it into machine-readable instructions; The fourth step is that the processing device performs processing according to machine-readable instructions.

15. A parts processing device, characterized in that: The device is used for the processing method of claim 9, and the part processing device includes a motion component, an actuator, and a controller; the controller receives input of modal information, runs any method described in claims 1-13, and converts the generated three-dimensional model into machine-readable instructions. The combination of the motion component and the actuator is a 3D printer or a CNC engraving machine.

16. An electronic terminal, characterized in that: The electronic terminal executes the method according to any one of claims 1 to 15.

Citation Information

Patent Citations

  • A angle-image multi-stage neural network based 3D reconstruction method

    CN109389671A

  • Neural radiation field model training method and device, human face generation method and device, and server

    CN113822969A

  • Text regulation and control three-dimensional face generation method based on StyleCLIP and 3D DDFA

    CN114998690A

  • Three-dimensional model automatic modeling method based on text information guidance

    CN115994990A

  • Multi-modal image search

    US20200104318A1

Cited By

  • CLIP-based property-to-microstructure reverse generation method

    CN119206098A

  • Facility three-dimensional model generation method and device, equipment and medium

    CN119251412A

  • Three-dimensional model driving method and device and electronic equipment

    CN119722880A

  • Multi-view collaborative cross-modal 3D perception model training method based on data screening optimization

    CN119888410A

  • Aircraft three-dimensional aerodynamic configuration design method, device, equipment and medium

    CN120086985A