Training method and device of 3D generative model, equipment and storage medium

Abstract

The invention discloses a training method and device for a 3D generative model, equipment and a storage medium, and belongs to the technical field of artificial intelligence. The method comprises the following steps: generating 3D construction data based on a generation rule of 3D data; generating 3D prediction data corresponding to the 3D construction data through an expression learning model according to the initialized vector representation; adjusting parameters of the expression learning model by taking minimization of the difference between the 3D prediction data and the 3D construction data as a target to obtain a trained expression learning model; and training a 3D generation model based on the trained expression learning model and the 3D sample data. According to the method, the expression learning model is pre-trained by generating the 3D construction data, and the problem that 3D real data is difficult to obtain is avoided. Moreover, the expression learning model is trained by using the 3D construction data, so that the problem of limitation of training samples in related technologies is avoided, the generalization of the expression learning model is improved, and the reliability of subsequent 3D generation model training is ensured.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and particularly relates to a training method, device, equipment and storage medium for a 3D generation model. Background Art

[0002] 3D (3Dimensional) generation models are widely used in many fields such as game development, animation production, virtual reality, industrial design, and medical imaging.

[0003] In related technologies, the training method of a 3D generation model may include the following steps: obtaining 3D sample data; first pre-training an MLP (Multilayer Perceptron) model using some of the 3D sample data to obtain a trained MLP model; then using the trained MLP model to generate the triplane expression corresponding to each 3D sample data in the remaining 3D sample data; and finally using the triplane expression to train the 3D generation model to obtain a trained 3D generation model.

[0004] For the above method, on the one hand, the 3D sample data is less and the acquisition difficulty is great. On the other hand, the method of only using some 3D sample data to train the MLP will bring the problem of training sample limitation, which will make the generalization of the trained MLP relatively poor. Summary of the Invention

[0005] Embodiments of this application provide a training method, device, equipment and storage medium for a 3D generation model. The technical solutions provided by the embodiments of this application are as follows:

[0006] According to one aspect of the embodiments of this application, a training method for a 3D generation model is provided, and the method includes:

[0007] Generating 3D construction data based on the generation rule of 3D data, where the 3D construction data is 3D data for constructing a first 3D model, and the generation rule is used to define the conditions that the 3D data needs to meet;

[0008] Generating 3D prediction data corresponding to the 3D construction data by an expression learning model according to an initialized vector representation, where the 3D prediction data is 3D data predicted by the expression learning model and used to construct the first 3D model;

[0009] Taking minimizing the difference between the 3D prediction data and the 3D construction data as the goal, adjusting the parameters of the expression learning model to obtain a trained expression learning model;

[0010] Based on the trained expression learning model and 3D sample data, train a 3D generation model, where the 3D sample data is 3D data for constructing a second 3D model, and the 3D generation model is used to generate 3D data corresponding to an image based on the image.

[0011] According to one aspect of the embodiments of the present application, there is provided a training device for a 3D generation model, the device including:

[0012] A first generation module, configured to generate 3D construction data based on the generation rules of 3D data, where the 3D construction data is 3D data for constructing a first 3D model, and the generation rules are used to define the conditions that the 3D data needs to meet;

[0013] A second generation module, configured to generate 3D prediction data corresponding to the 3D construction data through an expression learning model according to an initialized vector representation, where the 3D prediction data is 3D data predicted by the expression learning model and used to construct the first 3D model;

[0014] An adjustment module, configured to adjust the parameters of the expression learning model with the goal of minimizing the difference between the 3D prediction data and the 3D construction data, and obtain a trained expression learning model;

[0015] A training module, configured to train a 3D generation model based on the trained expression learning model and 3D sample data, where the 3D sample data is 3D data for constructing a second 3D model, and the 3D generation model is used to generate 3D data corresponding to an image based on the image.

[0016] According to one aspect of the embodiments of the present application, there is provided a computer device, the computer device including a processor and a memory, where a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the above-mentioned 3D generation model training method.

[0017] According to one aspect of the embodiments of the present application, there is provided a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and the computer program is loaded and executed by a processor to implement the above-mentioned 3D generation model training method.

[0018] According to one aspect of the embodiments of the present application, there is provided a computer program product, the computer program product including a computer program, the computer program being stored in a computer-readable storage medium, and the processor reads and executes the computer program from the computer-readable storage medium to implement the above-mentioned 3D generation model training method.

[0019] The technical solution provided by the embodiments of the present application at least includes the following beneficial effects:

[0020] By generating 3D construction data to pre-train an expression learning model, on the one hand, it avoids the problem that it is difficult to obtain 3D real data. On the other hand, using 3D construction data to train the expression learning model avoids the problem of the limitation of training samples in related technologies, improves the generalization of the expression learning model, and thus ensures the reliability of subsequent 3D generation model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a schematic diagram of a computer system provided by an embodiment of the present application;

[0022] Figure 2 is a flowchart of a method for training a 3D generation model provided by an embodiment of the present application;

[0023] Figure 3 is a flowchart of a training method for an expression learning model provided by an embodiment of the present application;

[0024] Figure 4 is a flowchart of the training and usage process of a 3D generation model provided by an embodiment of the present application;

[0025] Figure 5 is a flowchart of the training and usage process of a 3D generation model provided by another embodiment of the present application;

[0026] Figure 6 is a flowchart of a method for training a 3D generation model provided by another embodiment of the present application;

[0027] Figure 7 is a block diagram of a training device for a 3D generation model provided by an embodiment of the present application;

[0028] Figure 8 is a block diagram of the structure of a computer device provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0029] To make the objectives, technical solutions, and advantages of the present application clearer, the following will further describe the embodiments of the present application in detail with reference to the accompanying drawings.

[0030] Please refer to Figure 1 , which shows a schematic diagram of a computer system provided by an embodiment of the present application. The computer system may include a model training device 110 and a model using device 120.

[0031] The model training device 110 can be an electronic device such as a mobile phone, a desktop computer, a tablet computer, a laptop computer, a vehicle-mounted terminal, a server, an intelligent robot, an intelligent TV, a multimedia playback device, or some other electronic device with strong computing capabilities. This application does not make any limitations in this regard. The model training device 110 is used to train a 3D generation model, which is used to generate 3D data corresponding to an image based on the image. The 3D data refers to data used to describe an object in a three-dimensional space. Based on this 3D data, a 3D model corresponding to the image can be constructed.

[0032] In some embodiments, the solution provided in this application can be applied to scenarios such as games, autonomous driving, architectural design, mechanical manufacturing, and animation production. Exemplarily, when applied to the game application scenario, the 3D generation model can generate its corresponding 3D data according to a 2D game image (such as a virtual character image of the game), and this 3D data is used to construct a 3D virtual character model corresponding to the virtual character image. Exemplarily, when applied to the autonomous driving application scenario, the 3D generation model can generate its corresponding 3D environment data according to the driving environment images such as roads, pedestrians, and vehicles captured by a camera. Based on this 3D environment data, the autonomous driving system of the vehicle can construct a 3D environment model to more accurately perceive the surrounding environment, accurately plan the driving path, and improve the safety of autonomous driving.

[0033] In the embodiments of this application, the 3D generation model is a deep neural network model. Optionally, the model training device 110 can adopt a machine learning method to train the expression learning model so that it has better performance. Optionally, the training process of the 3D generation model is as follows (only a brief description is provided here, and the specific training process is described in the following embodiments): Based on the generation rule of 3D data, 3D construction data is generated; through the expression learning model, 3D prediction data corresponding to the 3D construction data is generated according to the initialized vector representation; with the goal of minimizing the difference between the 3D prediction data and the 3D construction data, the parameters of the expression learning model are adjusted to obtain a trained expression learning model; based on the trained expression learning model and 3D sample data, the 3D generation model is trained.

[0034] In some embodiments, the model training devices for the above-mentioned expression learning model and the 3D generation model may not be the same device. In this case, the first model training device is used to generate 3D construction data based on the generation rules of 3D data; the expression learning model generates 3D prediction data corresponding to the 3D construction data according to the initialized vector representation; with the goal of minimizing the difference between the 3D prediction data and the 3D construction data, the parameters of the expression learning model are adjusted to obtain the trained expression learning model. The first model training device sends the trained expression learning model to the second model training device, and the second model training device is used to train the 3D generation model based on the trained expression learning model and 3D sample data. The above-mentioned first model training device and the second model training device are two different model training devices.

[0035] The model using device 120 may be an electronic device such as a mobile phone, a desktop computer, a tablet computer, a laptop computer, a vehicle-mounted terminal, a server, an intelligent robot, an intelligent TV, a multimedia playback device, AR (Augmented Reality), VR (Virtual Reality), MR (Mixed Reality), etc., or some other electronic devices with strong computing capabilities. The present application does not make any limitations in this regard. The model using device 120 may use the trained 3D generation model to generate 3D prediction data corresponding to the first image according to the first image, and the 3D prediction data is used to construct a 3D model corresponding to the first image.

[0036] The model training device 110 and the model using device 120 may be two independent devices or the same device.

[0037] In the method provided by the embodiments of the present application, the execution subject of each step may be a computer device, which refers to an electronic device with data calculation, processing, and storage capabilities. Among them, when the computer device is a server, the server may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The computer device may be Figure 1 the model training device 110, or the model using device 120.

[0038] Please refer to Figure 2 , which shows a flowchart of a method for training a 3D generation model provided by an embodiment of the present application. The execution subject of each step of this method may be a computer device. For example, the computer device may be Figure 1 the model training device 110 in the computer system shown. This method may include at least one of the following steps 210 to 240.

[0039] Step 210: Generate 3D construction data based on the generation rules of 3D data. The 3D construction data is 3D data for constructing the first 3D model, and the generation rules are used to define the conditions that the 3D data needs to meet.

[0040] The 3D data can be used to describe at least one of the following attribute information of the 3D model: shape, size, structure, surface features, and material. The 3D construction data refers to the 3D data constructed based on the generation rules, which facilitates the subsequent direct application of the 3D construction data to train the expression learning model and avoids the problem that it is difficult to obtain 3D real data in the related art. The first 3D model refers to the 3D object corresponding to the 3D construction data. Exemplarily, the 3D construction data may include at least one construction sub-data, and each construction sub-data can be used to indicate a position in the three-dimensional space. Then, through the positions of each construction sub-data in the 3D construction data, the first 3D model can be constructed in the three-dimensional space.

[0041] Step 220: Generate 3D prediction data corresponding to the 3D construction data through the expression learning model according to the initialized vector representation. The 3D prediction data is the 3D data predicted by the expression learning model and used to construct the first 3D model.

[0042] The initialized vector representation refers to a randomly initialized vector representation. The dimension of this vector is not limited in this application. It should be noted that when the parameters of the expression learning model are adjusted subsequently, the initialized vector representation will also be adjusted and updated.

[0043] In some embodiments, the 3D construction data includes N construction sub-data, and each construction sub-data corresponds to an initialized vector representation. The initialized vector representations corresponding to each construction sub-data may be the same or different, and this application does not make a limitation on this. Through the expression learning model, according to the initialized vector representation corresponding to each construction sub-data, generate the prediction sub-data corresponding to each construction sub-data, so as to generate the 3D prediction data corresponding to the 3D construction data, where the 3D prediction data also includes N prediction sub-data, and N is a positive integer.

[0044] In some embodiments, the expression learning model is constructed based on an MLP, which is a feedforward artificial neural network composed of multiple fully connected neuron layers. The structure of the MLP may include an input layer, a hidden layer, and an output layer. The input layer receives the initialized vector representation as input. The number of neurons in the input layer is the same as the dimension of the initialized vector. For example, if the initialized vector representation is a 128-dimensional vector, then the input layer has 128 neurons, and each neuron corresponds to a dimension in the vector representation and is responsible for receiving the value on that dimension. The hidden layer is the key part of the MLP for feature extraction and non-linear transformation. It usually contains multiple fully connected layers, and each layer consists of a certain number of neurons. The number of hidden layers can be adjusted according to the complexity of the task and the characteristics of the data. The number of neurons in each hidden layer also needs to be determined according to specific circumstances. To introduce non-linearity, an activation function can be applied to each neuron in the hidden layer. Common activation functions include ReLU (Rectified Linear Unit), Sigmoid, Tanh, etc. The number of neurons in the output layer is related to the dimension of the 3D prediction data. For example, the number of neurons in the output layer is the same as the dimension of the predicted sub-data in the 3D prediction data, so as to generate the final 3D prediction data.

[0045] Step 230: Adjust the parameters of the expression learning model with the goal of minimizing the difference between the 3D prediction data and the 3D construction data, and obtain the trained expression learning model.

[0046] In some embodiments, the 3D prediction data includes N predicted sub-data, and the 3D construction data includes N construction sub-data. Among them, the construction sub-data and the predicted sub-data are in a one-to-one correspondence relationship, one construction sub-data corresponds to one predicted sub-data, and different construction sub-data correspond to different predicted sub-data. Adjust the parameters of the expression learning model with the goal of minimizing the difference between each predicted sub-data and the construction sub-data, and obtain the trained expression learning model. It should be noted that in this process, the parameters of the initialized vector representation will also be updated. The updated vector representation refers to the vector representation of the 3D data corresponding to the corresponding construction sub-data.

[0047] In some embodiments, the loss function can be the mean squared error, the average mean squared error, etc., which is used to measure the difference between the 3D prediction data and the 3D construction data. The expression learning model is trained using the supervised learning method. Taking the 3D construction data as the label, the expression learning model continuously adjusts the weights so that the input initialized vector can generate 3D prediction data consistent with the 3D construction data.

[0048] Step 240: Based on the trained expression learning model and 3D sample data, train a 3D generation model. The 3D sample data is 3D data used to construct a second 3D model, and the 3D generation model is used to generate 3D data corresponding to an image based on the image.

[0049] The 3D sample data refers to the collected 3D data, and the second 3D model refers to the 3D object corresponding to the 3D sample data. The second 3D model is different from the first 3D model corresponding to the above-mentioned 3D construction data.

[0050] In some embodiments, the 3D generation model includes the trained expression learning model, that is, the trained expression learning model serves as a sub-network of the 3D generation model. During the training process of the 3D generation model, the parameters of the trained expression learning model are fixed. In other words, during the training of the 3D generation model, only the parameters of other sub-networks outside the trained expression learning model need to be adjusted, thereby accelerating the training process of the 3D generation model. And since the expression learning model is trained based on 3D construction data, it avoids the problem of limited training samples caused by only using partial 3D sample data in the related art, thereby improving the generalization of the expression learning model.

[0051] In summary, the technical solution provided by the embodiments of the present application pre-trains the expression learning model by generating 3D construction data. On the one hand, it avoids the problem of difficult acquisition of 3D real data. On the other hand, using 3D construction data to train the expression learning model avoids the problem of limited training samples in the related art and improves the generalization of the expression learning model, thereby ensuring the reliability of the subsequent training of the 3D generation model.

[0052] The following introduces the generation process of the 3D construction data.

[0053] In some embodiments, the generation rules of the 3D data include: the numerical range of the 3D data. Based on the generation rules of the 3D data, generate 3D construction data, including: based on the numerical range of the 3D data, generate 3D construction data, and the 3D construction data covers the numerical range.

[0054] The numerical range of the 3D data refers to the set of all values of the 3D data in the three-dimensional space system. If the values of the 3D data are continuous values, the numerical range of the 3D data is the upper and lower limit intervals of the values of the 3D data. If the values of the 3D data are discrete values, the numerical range of the 3D data is the discrete points of the values of the 3D data.

[0055] In some embodiments, the 3D data corresponds to multiple expression forms, each expression form corresponding to a numerical range, and different expression forms usually correspond to different numerical expression ranges. Exemplarily, the expression forms may include SDF (Sign Distance Function), Occupancy, and color channel representation, which are not limited in this application. Among them, SDF is a function used to represent the surface of a three-dimensional object. For any point in space, the value of SDF represents the directed distance from the point to the object surface. In 3D model generation and processing, SDF can accurately describe the shape of a 3D object, and the position relationship between the point and the object surface can be clearly distinguished through the numerical range. The numerical range of SDF usually ranges from (-1, 1). When the point is inside the object, the SDF value is negative; when the point is on the object surface, the SDF value is 0; when the point is outside the object, the SDF value is positive. This range is determined according to the positive and negative characteristics of the directed distance and general normalization processing, which is convenient for unified processing and learning of the model.

[0056] Occupancy refers to, in three-dimensional space, for each discrete spatial unit (such as a voxel), using a numerical value to represent the probability or state of the unit being occupied by an object. It is an expression form used to describe the distribution of objects in 3D space. In voxel-based 3D modeling, each voxel has a corresponding Occupancy value. When the Occupancy value is 1, it means the voxel is completely occupied by the object; when the Occupancy value is 0, it means the voxel is not occupied by the object. By recording and analyzing the Occupancy values of each voxel, a spatial distribution model of the 3D object can be constructed. The numerical range of Occupancy is the discrete values 0 and 1.

[0057] Color channel representation refers to, in the context of 3D data expression, a way of using specific numerical values to quantify and describe the color information of each point or region in three-dimensional space. Exemplarily, the color channel representation can be RGB (Red Green Blue) three-channel color. In the RGB model, the numerical value of each channel varies from 0 to 255. 0 means the intensity of the color component is 0, that is, there is no component of this color; 255 means the intensity of the color component reaches the maximum. Then the numerical expression range should be any combination of ([0, 255], [0, 255], [0, 255]). In some embodiments, in addition to the RGB color channel, the color channel representation can also be other color channel representation methods, such as CMYK (Cyan Magenta Yellow Key (Black)), HSV (Hue Saturation Value), etc., which are not limited in this application.

[0058] The numerical range covered by the 3D structure data means that the value range of the structural sub-data included in the 3D structure data covers all the values of the 3D data. If the values of the 3D data are continuous values, the values of the structural sub-data can continuously vary within the upper and lower limit intervals of the 3D data values. And for any value within this interval, a corresponding value can be found in the structural sub-data or can be obtained by performing reasonable mathematical operations such as interpolation and fitting on the structural sub-data. In some embodiments, a sampling step size is determined, and points are evenly sampled within the upper and lower limit intervals by the set sampling step size as the structural sub-data included in the 3D structure data. If the values of the 3D data are discrete values, the values of the structural sub-data are the respective discrete points of the 3D data values. The sampling step size can be set according to the experience of relevant technicians, and this application does not limit it.

[0059] Exemplarily, the expression corresponding to the 3D data can be SDF, and the numerical range of SDF can be (-1, 1). Then the values of the structural sub-data included in the 3D structure data can continuously vary within the interval (-1, 1). For any value within this interval, a corresponding value can be found in the structural sub-data or can be obtained by performing reasonable mathematical operations such as interpolation and fitting on the structural sub-data. If discretization processing is to be performed on it, an appropriate sampling step size can be determined, and points are evenly sampled within the interval (-1, 1) as the structural sub-data to approximately represent the continuous change of SDF in the three-dimensional space. Exemplarily, the expression corresponding to the 3D data can be the RGB three-channel color. Since the value range of each channel in the RGB three channels is discrete integers in the range of [0, 255], then the values of the structural sub-data included in the 3D structure data are the combinations of the respective discrete points within the value ranges of each RGB channel, which can include 256 3 combinations.

[0060] In the above method, by generating 3D structure data, and the 3D structure data can cover the numerical range of the 3D data. This way of generating structure data based on the numerical range further avoids the influence of uneven sample data distribution caused by using 3D sample data to train the expression learning model, and improves the generalization of the expression learning model.

[0061] In some embodiments, the generation rule of the 3D data may further include the geometric topology of the 3D model corresponding to the 3D data. In some embodiments, 3D construction data is generated based on the conditions that the geometric topology of the 3D model needs to satisfy. Exemplarily, it is stipulated that the 3D model corresponding to the generated 3D data needs to conform to a specific geometric shape, such as a sphere, a cylinder, a cube, etc. For example, when the application scenario of the 3D generation model is the industrial field, the 3D generation model is used to generate 3D data of a mechanical part corresponding to a mechanical part image, and the 3D data can construct a 3D model corresponding to the mechanical part image. In this scenario, it may be required that a certain component must be in the shape of a cylinder, and there are precise numerical range requirements for its dimensions such as radius and height. Exemplarily, it is stipulated that the connection manner between the respective components of the 3D model conforms to a specific topological relationship. For example, when generating a 3D model of the human skeleton, it is stipulated how the bones at the joint parts are connected, whether it is a hinge connection, a ball-and-socket connection or other types of connections, and there are corresponding restrictions on the degrees of freedom and ranges of motion of the connections. The above method, by stipulating the geometric topology of the 3D model corresponding to the generated 3D construction data, enables the trained expression learning model to be applicable to the requirements of a specific scenario, makes the first 3D representation output by the expression learning model more accurate, and ensures the reliability of the subsequent training of the 3D generation model.

[0062] In some embodiments, the 3D data is used to indicate at least one of the following: geometric features, color features. The geometric features are used to characterize the structural information of the first 3D model, and the color features are used to characterize the color information of the first 3D model.

[0063] The structural information is used to indicate the shape, size, position, orientation of an object in three-dimensional space, and the relative relationship between its various parts. The structural information can be used to indicate the position of each 3D sub-data in the 3D data, and the relative relationship between the respective 3D sub-data. The color information refers to the color values of the 3D model. The color information may include the color channel values of each 3D sub-data in the 3D data, such as RGB color channel values.

[0064] In some embodiments, when the 3D data is used to indicate geometric features, the corresponding expression mode of the 3D data may be SDF, Occupancy. When the 3D data is used to indicate color features, the corresponding expression mode of the 3D data may be color channel representation, such as RGB three-channel color representation.

[0065] In the above method, the 3D data covers both geometric features and color features, and can comprehensively represent the information of the 3D model.

[0066] In some embodiments, such as Figure 3As shown, it shows a flowchart of a method for training an expression learning model provided by an embodiment of the present application, which may include the following steps S1 to S4.

[0067] Step S1, generate 3D construction data, where the 3D construction data can be SDF, RGB, Occupancy. The 3D construction data may include N construction sub-data, and N is a positive integer.

[0068] Step S2, randomly sample any one of the N construction sub-data to generate its corresponding initialized vector representation.

[0069] Step S3, through the MLP model, according to the construction sub-data (such as construction SDF, construction RGB, construction occupancy), generate the corresponding prediction sub-data (such as prediction SDF, prediction RGB, prediction occupancy) of the construction sub-data.

[0070] Step S4, with the goal of minimizing the difference between the construction sub-data and the prediction sub-data, adjust the parameters of the MLP model to obtain the trained MLP model. Exemplarily, with the goal of minimizing the difference between construction SDF and prediction SDF, adjust the parameters of the MLP model to obtain the trained MLP model. Exemplarily, with the goal of minimizing the difference between construction RGB and prediction RGB, adjust the parameters of the MLP model to obtain the trained MLP model. Exemplarily, with the goal of minimizing the difference between construction occupancy and prediction occupancy, adjust the parameters of the MLP model to obtain the trained MLP model.

[0071] In the above method, since when training the MLP model, the 3D construction data covers both geometric features and color features, it can comprehensively represent the information of the 3D model, improving the prediction accuracy and generalization ability of the trained MLP model.

[0072] Next, two implementation methods for training the 3D generation model are introduced.

[0073] Method 1

[0074] In some embodiments, as Figure 4 shown, it shows a flowchart of the training and use process of a 3D generation model provided by an embodiment of the present application. Among them, sub Figure 1 is the model training process, sub Figure 2This is the process of using the model. The model training process is as follows: The 3D generation model includes a first neural network and a second neural network; Based on the trained expression learning model and 3D sample data, train the 3D generation model, including: Determine the first 3D data representation corresponding to the 3D sample data through the trained expression learning model; Generate the first implicit representation corresponding to the first 3D data representation through the first neural network according to the first 3D data representation; Generate the predicted implicit representation corresponding to the sample image through the second neural network according to the sample image corresponding to the 3D sample data and the first random noise; With the goal of minimizing the difference between the predicted implicit representation and the first implicit representation, adjust the parameters of the second neural network to obtain the trained 3D generation model.

[0075] 3D data representation refers to the representation of various information contained in a 3D model in the real world or virtual world in a specific format or manner that can be recognized, processed, and stored by a computer. Exemplarily, 3D data representation can be a triplane representation, a voxel representation, a point cloud representation, a polygon mesh representation, an implicit surface representation, etc., and the present application is not limited thereto. Among them, the triplane representation encodes information in 3D space based on three mutually perpendicular two-dimensional planes (usually the XY, XZ, and YZ planes). These three planes are called the triplane feature grid. Specific feature information, such as feature values related to color, density, etc., is stored on each plane. When querying the attributes of a certain point in 3D space, bilinear interpolation operations are performed on these three planes to obtain the features at the corresponding positions, and then the features obtained from different planes are combined (such as stitching or other forms of fusion) to obtain the complete feature representation of the 3D point. For the voxel representation, a voxel is short for volume element, similar to a pixel in a two-dimensional image. The voxel representation divides the three-dimensional space into regular small cubes, and each small cube is a voxel. The 3D data is represented by marking whether the voxel is occupied by an object and assigning corresponding attributes (such as density, color, etc.). This representation method is simple and intuitive and is suitable for dealing with some problems based on spatial discretization. For the point cloud representation, a point cloud is a set composed of a large number of three-dimensional points, and each point contains coordinate information in three-dimensional space. Some point clouds may also contain additional information such as color and normal. For the polygon mesh representation, the polygon mesh is one of the most common 3D data representation methods. It consists of a series of polygons (usually triangles), and these polygons are connected together by sharing vertices to form a closed surface to represent a 3D object. For the implicit surface representation, the implicit surface defines the surface of a 3D object through a function, and the points where the function value is zero form the surface of the object. Different from explicit representations (such as polygon meshes), the implicit surface does not need to explicitly store the points on the surface, but determines the position of the surface through function calculations.

[0076] In some embodiments, the 3D sample data includes M sample sub-data, and through the trained expression learning model, the first 3D data representation corresponding to each sample sub-data is determined.

[0077] In some embodiments, determining a first 3D data representation corresponding to 3D sample data through a trained expression learning model includes: generating an initialized 3D data representation; generating 3D prediction data corresponding to the 3D sample data according to the initialized 3D data representation through the trained expression learning model; and updating the initialized 3D data representation with the goal of minimizing the difference between the 3D prediction data and the 3D sample data, so as to obtain the updated 3D data representation as the first 3D data representation.

[0078] The initialized 3D data representation refers to a randomly initialized 3D data representation. If the 3D data representation is a traiplane representation, the initialized 3D data representation can be feature vectors randomly initialized on three mutually perpendicular planes (XY, XZ, and YZ planes), and the feature vectors on different planes can have different dimensions. For different 3D data representation methods, the number and dimensions of the feature vectors included in the initialized 3D data representation can vary, and this application does not limit the number and dimensions of the feature vectors included in the initialized 3D data representation.

[0079] In some embodiments, the 3D sample data includes M sample sub-data, each sample sub-data corresponds to an initialized 3D data representation, and the initialized 3D data representations corresponding to each sample sub-data can be the same or different, and this application does not limit this. The trained expression learning model generates prediction sub-data corresponding to each sample sub-data according to the initialized 3D data representation corresponding to each sample sub-data. In some embodiments, the 3D sample data includes M sample sub-data, and the 3D prediction data includes M prediction sub-data. Among them, the sample sub-data and the prediction sub-data are in a one-to-one correspondence relationship, one sample sub-data corresponds to one prediction sub-data, and different sample sub-data correspond to different prediction sub-data. With the goal of minimizing the difference between the sample sub-data and the prediction sub-data, the initialized 3D data representation is updated to obtain the updated 3D data representation as the first 3D data representation corresponding to the sample sub-data. It should be noted that during the process of updating the initialized 3D data representation, the parameters of the trained expression learning model remain fixed.

[0080] The above method can accurately obtain the first 3D data representation corresponding to each sample sub-data in the 3D sample data through the trained expression learning model.

[0081] The implicit representation (Latent), also known as implicit encoding, is a representation form that maps high-dimensional data to a low-dimensional latent space through a specific encoding method. It can capture the internal structure and features of the first 3D representation, remove redundant information in the first 3D representation, so that the first 3D representation can be analyzed, processed, and generated more efficiently in the low-dimensional space. In some embodiments, such asFigure 4 As shown, the first neural network includes an encoder and a decoder. The encoder of the first neural network generates a first implicit representation corresponding to the first 3D data representation according to the first 3D data representation.

[0082] In some embodiments, the pre-trained first neural network generates a first implicit representation corresponding to the first 3D data representation according to the first 3D data representation. In some embodiments, the first neural network includes an encoder and a decoder; the method for determining the pre-trained first neural network includes: generating a second implicit representation corresponding to the first 3D data representation through the encoder according to the first 3D data representation; generating a reconstructed first 3D data representation through the decoder according to the second implicit representation corresponding to the first 3D data representation; aiming at minimizing the difference between the first 3D data representation and the reconstructed first 3D data representation, adjusting the parameters of the first neural network to obtain the pre-trained first neural network, and the pre-trained first neural network is used to generate a first implicit representation corresponding to the first 3D data representation based on the first 3D data representation.

[0083] It should be noted that during the training stage of the first neural network, the encoder of the first neural network generates a second implicit representation according to the first 3D data representation. After obtaining the pre-trained first neural network, the encoder of the pre-trained first neural network generates a first implicit representation based on the first 3D data representation. Among them, the second implicit representation is different from the first implicit representation. In other words, during the pre-training process and after the pre-training of the first neural network, although both are based on the first 3D data representation to generate its corresponding implicit representation. However, the second implicit representation generated during the training process is different from the first implicit representation generated after the pre-training is completed.

[0084] In some embodiments, the first neural network is a VAE (Variational Autoencoder) model, the encoder is the encoder of the VAE model, and the decoder is the decoder of the VAE model. The VAE model is a generative model, and its core advantage is that it can not only compress and reconstruct data like a traditional autoencoder, but also learn the latent probability distribution of the data. The encoder of the VAE model is used to map the input first 3D data representation into a latent space. The VAE encoder outputs a set of parameters, and this set of parameters is used to indicate the mean and variance, which are used to define a Gaussian distribution in the latent space, that is, the second implicit representation. For example, when the first 3D data representation is the triplane representation of 3D data, the encoder will analyze the geometric feature information of the triplane representation and convert it into the mean and variance of the Gaussian distribution in the latent space. The VAE encoder not only simply reduces the dimension of the first 3D data representation, but also mines the latent probability structure behind the first 3D data representation.

[0085] The VAE decoder samples according to the latent distribution output by the encoder to obtain a specific latent vector (i.e., the second implicit representation), and then maps this latent vector back to the original data space to generate the reconstructed first 3D data representation. In this process, the decoder needs to learn how to generate meaningful 3D data representations from the distribution in the latent space.

[0086] In some embodiments, the loss function L for training the first neural network = reconstruction loss + β × KL divergence regularization term. Among them, the reconstruction loss is used to indicate the difference between the first 3D data representation and the reconstructed first 3D data representation, the KL divergence regularization term is used to constrain the latent distribution to be close to the standard normal distribution, and β is a hyperparameter used to control the weight of the regularization term. With the goal of minimizing this loss function, the parameters of the first neural network are adjusted to obtain the pre-trained first neural network. In the above method, during the pre-training process, with the goal of minimizing the difference between the first 3D data representation and the reconstructed first 3D data representation, the regularization of the latent distribution also needs to be considered. This is because we hope that the latent space has a good structure to facilitate subsequent generation and interpolation operations. Specifically, in the VAE model, the encoder does not directly output a definite latent vector, but outputs the parameters (usually the mean μ and variance σ 2 ) that describe the latent distribution. Generally, it is assumed that this latent distribution is a Gaussian distribution. Regularization is to constrain this latent distribution to meet specific conditions, usually making this latent distribution as close as possible to the standard normal distribution (a Gaussian distribution with a mean of 0 and a variance of 1).

[0087] The above method facilitates the subsequent generation of the first implicit representation corresponding to the first 3D data representation by pre-training the first neural network. And this method trains the first neural network and the second neural network separately, and uses this decoupled method to train the 3D generation model, which is beneficial to improving the efficiency and stability of model training and reducing the training complexity.

[0088] In some embodiments, the sample image is a two-dimensional image of the second 3D model corresponding to the 3D sample data. The sample image can be multiple images of the second 3D model from different perspectives. The present application does not limit the number of sample images.

[0089] The first random noise refers to a numerical sequence with randomness characteristics artificially introduced during the training of a specific machine learning model and the data generation process. Random noise is usually sampled from a certain probability distribution, such as the Gaussian distribution (normal distribution) commonly. Sampling from the Gaussian distribution means that the values of the noise are symmetrically distributed around the mean, most of the values are concentrated near the mean, and the probability of the values getting farther away from the mean is smaller.

[0090] In some embodiments, the second neural network is a diffusion model. In the framework of the diffusion model, the first random noise generally refers to the random signal introduced at the beginning stage of the diffusion process. The diffusion model is based on the core idea that by gradually adding noise to the data, the original data is converted into pure noise, which is called the forward diffusion process; then starting from the pure noise, the noise is gradually removed to restore the original data, which is called the reverse diffusion process. And the first random noise is the noise used at the beginning of the reverse diffusion process. The diffusion model starts the reverse diffusion process according to the sample image corresponding to the 3D sample data and the first random noise. In the reverse diffusion process, the model starts from the noise, gradually removes the noise, and at the same time combines the information provided by the sample image to continuously adjust its own state, and finally generates a predicted implicit representation corresponding to the sample image. The predicted implicit representation is also a low-dimensional feature vector, which tries to approximate the first implicit representation as much as possible to reflect the true features of the 3D sample data.

[0091] During the training process, the loss function is the difference between the predicted implicit representation and the first implicit representation. Optimization algorithms (such as stochastic gradient descent, Adam optimizer, etc.) can be used to update the parameters of the second neural network according to the loss value. The optimization algorithm will adjust the parameters in the direction of reducing the loss value according to the gradient information of the loss function, so as to continuously improve the performance of the diffusion model. After multiple iterations of training, when the loss value converges to a small value, the trained 3D generation model is obtained. The trained 3D generation model also includes the above-mentioned first neural network and the trained expression learning model. For the specific structure of the 3D generation model, please refer to the detailed description below.

[0092] The following introduces the usage method of the model corresponding to Method 1. The execution subject of each step of this method can be a computer device. For example, this computer device can be Figure 2 the model usage device 120 in the computer system shown.

[0093] In some embodiments, as Figure 4 sub- Figure 2 shown, the trained 3D generation model includes the trained second neural network, and further includes: obtaining a first image; generating a predicted implicit representation corresponding to the first image through the trained second neural network according to the first image and the second random noise; generating a second 3D data representation through the decoder of the first neural network according to the predicted implicit representation corresponding to the first image; generating 3D predicted data corresponding to the first image through the trained expression learning model according to the second 3D data representation.

[0094] The first image is the image for which 3D prediction data is to be generated. In some embodiments, based on the 3D prediction data corresponding to the first image, a 3D model corresponding to the first image is constructed. In some embodiments, the decoder of the first neural network after pre-training generates a second 3D data representation according to the predicted implicit representation corresponding to the first image. For detailed content, please refer to the corresponding description on the training side above, and this application will not elaborate here.

[0095] In the above method, since the 3D generation model uses the trained second neural network in combination with the decoder of the first neural network and multiple sub-networks of the trained expression learning model, the generalization ability of the 3D generation model can be significantly improved. Each sub-network extracts and transforms features to varying degrees when processing data. This multi-level and multi-network processing method enables the 3D generation model to learn more essential and abstract features of the sample images. When encountering a new first image that has not appeared in the training set, the model can still accurately generate the corresponding 3D prediction data.

[0096] Method 2

[0097] In some embodiments, as Figure 5 shown, it shows a flowchart of the training and use process of the 3D generation model provided by another embodiment of this application. Among them, sub- Figure 1 is the model training process, and sub- Figure 2 is the model use process. The model training process is as follows. The 3D generation model includes: a third neural network and a trained expression learning model; based on the trained expression learning model and 3D sample data, training the 3D generation model includes: generating a 3D data representation corresponding to the sample image through the third neural network according to the sample image corresponding to the 3D sample data; generating a 3D prediction data corresponding to the sample image through the trained expression learning model according to the 3D data representation; adjusting the parameters of the third neural network with the goal of minimizing the difference between the 3D prediction data and the 3D sample data to obtain the trained 3D generation model.

[0098] In some embodiments, the third neural network is used to process the sample image corresponding to the 3D sample data and convert the 2D sample image into a 3D data representation. The third neural network is used to extract 3D spatial structure information from 2D information and transform it into a 3D data representation form suitable for subsequent processing.

[0099] The third neural network learns the mapping relationship from sample images to 3D data representations through a large amount of training data. During the training process, it continuously adjusts its own parameters so that for a given sample image, it can generate a 3D data representation that is as close as possible to the real 3D structure. This learning process is based on an optimization algorithm, aiming to minimize the difference between the 3D predicted data and the 3D sample data, thereby obtaining a trained 3D generation model. It should be noted that during the process of adjusting the parameters of the third neural network to obtain a trained 3D generation model, the parameters of the trained expression learning model are fixed.

[0100] In some embodiments, the third neural network is a transformer model. The transformer model includes an encoder and a decoder, where the encoder is used to encode the sample image into an image feature vector. The decoder is used to decode the image feature vector into a 3D data representation. Specifically, the encoder may include a self-attention mechanism (self) and an MLP. When processing a sample image, first, the sample image is divided into blocks, and each image block is mapped to a corresponding embedding vector. At the same time, a position encoding is added to the embedding vector to retain its position information in the sample image. Then, through the multi-head self-attention mechanism, the encoder enables the embedding vector of each image block to pay attention to the information of other image blocks, thereby capturing the dependencies and global features between different parts of the image. After that, the MLP further performs non-linear transformation and information integration on these features, and layer normalization is used after each sub-layer operation to ensure the stability of training. After multiple such operations, the sample image is finally transformed into a series of feature vectors, which contain multi-faceted information such as the spatial structure and semantics of the sample image, and are used to form a sequence that can comprehensively reflect the features of the sample image.

[0101] The decoder may include a cross-attention mechanism (cross), a self-attention mechanism (self), and an MLP. Among them, the cross-attention mechanism is used to effectively fuse the sample image feature vectors extracted by the encoder into the decoder's processing flow during the generation of the 3D data representation, so that the decoder can better utilize the global information of the image to guide the generation of the 3D data representation. The cross-attention mechanism allows each position in the decoder to focus on these feature vectors output by the encoder. For example, when the decoder wants to generate a specific part of a 3D object, through cross-attention, it can find information related to this part from the image features extracted by the encoder, such as clues about the shape and texture of this part of the object. The self-attention mechanism is used to capture and model the dependencies and context information between the internal elements of the generated 3D data sequence during the generation of the 3D data representation by the decoder, so as to ensure the coherence and consistency of the generated 3D data in terms of structure and semantics. The MLP is used to perform further non-linear transformation and feature extraction on the feature information output by the cross-attention mechanism and the self-attention mechanism, and map it to the output space of the 3D data representation, thereby generating the final 3D data representation.

[0102] In the above method, since the parameters of the trained representation learning model do not need to be adjusted during the training process of the 3D generation model, this method avoids the problem of excessive training parameter data volume during the training process of the end-to-end 3D generation model, thereby accelerating the training process of the 3D generation model.

[0103] The following introduces the usage method of the model corresponding to Method 2. The execution subject of each step of this method can be a computer device. For example, this computer device can be Figure 2 the model usage device 120 in the computer system shown.

[0104] In some embodiments, as Figure 5 sub Figure 2 shown, in some embodiments, the trained 3D generation model includes a trained third neural network. The method further includes: obtaining a first image; generating a third 3D data representation corresponding to the first image through the trained third neural network according to the first image; generating 3D prediction data corresponding to the first image through the trained representation learning model according to the third 3D data representation. In some embodiments, a 3D model corresponding to the first image is constructed based on the 3D prediction data corresponding to the first image. For detailed content, please refer to the corresponding description on the training side above. This application will not elaborate here.

[0105] In the above method, since the 3D generation model adopts two sub-networks, namely the trained third neural network combined with the trained representation learning model, it can significantly improve the generalization ability of the 3D generation model, making the generated predicted 3D data more accurate.

[0106] As shown Figure 6 in the figure, it shows a flowchart of a training method for a 3D generation model provided by another embodiment of the present application. It may include the following steps S1 to S4A, or steps S1 to S4B.

[0107] Step S1, generate 3D construction data.

[0108] Step S2, pre-train an expression learning model to obtain a trained expression learning model. Exemplarily, the expression learning model can be an MLP model.

[0109] The following implementation steps are divided into two branches, namely S3A to S4A and S3B to S4B respectively.

[0110] Step S3A, fix the parameters of the trained expression learning model and generate a first 3D data representation corresponding to the 3D sample data. The 3D sample data may include multiple sample sub-data, and generate a first 3D data representation corresponding to each sample sub-data.

[0111] Step S4A, use the first 3D data representation to train the 3D generation model. This step S1 to S4A corresponds to Method 1 of the training method introduced above. For detailed content, please refer to the corresponding content above. The present application will not elaborate here.

[0112] Step S3B, fix the parameters of the trained expression learning model and connect it as a module of the 3D generation model. During the training process of the 3D generation model, only adjust the parameters of other network modules except the trained expression learning model, which speeds up the training process of the 3D generation model.

[0113] Step S4B, use the sample images to train the 3D generation model. This step S1 to S4B corresponds to Method 2 of the training method introduced above. For detailed content, please refer to the corresponding content above. The present application will not elaborate here.

[0114] It should be noted that as long as the model training for which the training data generation rule can be determined in advance, it can be pre-trained through the solution of the present application, that is, the application scenario of the present application is not limited to the 3D generation scenario.

[0115] The following is an embodiment of the device of the present application. For details not described in detail in the embodiment of the device of the present application, reference can be made to the above method embodiment.

[0116] Please refer to Figure 7, which shows a block diagram of a training device for a 3D generation model provided by an embodiment of the present application. The device has the function of implementing the above-mentioned 3D generation model training method, and this function can be implemented by hardware or by hardware executing corresponding software. The device can be a computer device or can be set in a computer device. The device 700 may include: a first generation module 710, a second generation module 720, an adjustment module 730, and a training module 740.

[0117] The first generation module 710 is configured to generate 3D construction data based on the generation rule of 3D data. The 3D construction data is 3D data for constructing a first 3D model, and the generation rule is used to define the conditions that the 3D data needs to meet.

[0118] The second generation module 720 is configured to generate 3D prediction data corresponding to the 3D construction data through an expression learning model according to the initialized vector representation. The 3D prediction data is 3D data predicted by the expression learning model and used to construct the first 3D model.

[0119] The adjustment module 730 is configured to adjust the parameters of the expression learning model with the goal of minimizing the difference between the 3D prediction data and the 3D construction data, and obtain a trained expression learning model;

[0120] The training module 740 is configured to train a 3D generation model based on the trained expression learning model and 3D sample data. The 3D sample data is 3D data for constructing a second 3D model, and the 3D generation model is used to generate 3D data corresponding to an image based on the image.

[0121] In some embodiments, the generation rule of the 3D data includes: the numerical range of the 3D data; the first generation module 710 is configured to generate the 3D construction data based on the numerical range of the 3D data, and the 3D construction data covers the numerical range.

[0122] In some embodiments, the 3D expression is used to indicate at least one of the following: geometric features, color features. The geometric features are used to characterize the structural information of the first 3D model, and the color features are used to characterize the color information of the first 3D model.

[0123] In some embodiments, the 3D generation model includes a first neural network and a second neural network; the training module 740 is configured to determine a first 3D data representation corresponding to the 3D sample data through the trained expression learning model; generate a first implicit representation corresponding to the first 3D data representation through the first neural network according to the first 3D data representation; generate a predicted implicit representation corresponding to the sample image through the second neural network according to the sample image corresponding to the 3D sample data and a first random noise; and adjust the parameters of the second neural network with the goal of minimizing the difference between the predicted implicit representation and the first implicit representation to obtain the trained 3D generation model.

[0124] In some embodiments, the training module 740 is configured to generate an initialized 3D data representation; generate 3D predicted data corresponding to the 3D sample data through the trained expression learning model according to the initialized 3D data representation; and update the initialized 3D data representation with the goal of minimizing the difference between the 3D predicted data and the 3D sample data to obtain the updated 3D data representation as the first 3D data representation.

[0125] In some embodiments, the first neural network includes an encoder and a decoder; the apparatus 700 further includes: a third generation module ( Figure 7 not shown in the figure).

[0126] The third generation module is configured to generate a second implicit representation corresponding to the first 3D data representation through the encoder according to the first 3D data representation; generate a reconstructed first 3D data representation through the decoder according to the second implicit representation corresponding to the first 3D data representation; and adjust the parameters of the first neural network with the goal of minimizing the difference between the first 3D data representation and the reconstructed first 3D data representation to obtain the pre-trained first neural network, and the pre-trained first neural network is configured to generate a first implicit representation corresponding to the first 3D data representation based on the first 3D data representation.

[0127] In some embodiments, the trained 3D generation model includes a trained second neural network, and the apparatus 700 further includes: a fourth generation module ( Figure 7 not shown in the figure).

[0128] The fourth generation module is configured to obtain a first image; generate a predicted implicit representation corresponding to the first image according to the first image and second random noise by the trained second neural network; generate a second 3D data representation according to the predicted implicit representation corresponding to the first image by the decoder of the first neural network; and generate 3D predicted data corresponding to the first image according to the second 3D data representation by the trained expression learning model.

[0129] In some embodiments, the 3D generation model includes: a third neural network and the trained expression learning model; the training module 700 is configured to generate a 3D data representation corresponding to the sample image according to the sample image corresponding to the 3D sample data by the third neural network; generate 3D predicted data corresponding to the sample image according to the 3D data representation by the trained expression learning model; and adjust the parameters of the third neural network with the goal of minimizing the difference between the 3D predicted data and the 3D sample data to obtain a trained 3D generation model.

[0130] In some embodiments, the trained 3D generation model includes a trained third neural network, and the apparatus 700 further includes: a fifth generation module ( Figure 7 not shown in the figure).

[0131] The fifth generation module is configured to obtain a first image; generate a third 3D data representation corresponding to the first image according to the first image by the trained third neural network; and generate 3D predicted data corresponding to the first image according to the third 3D data representation by the trained expression learning model.

[0132] In summary, the technical solution provided by the embodiments of the present application pre-trains the expression learning model by generating 3D construction data. On the one hand, it avoids the problem that it is difficult to obtain 3D real data. On the other hand, using 3D construction data to train the expression learning model avoids the problem of training sample limitations in the related art, improves the generalization of the expression learning model, and thus ensures the reliability of subsequent 3D generation model training.

[0133] It should be noted that when the apparatus provided in the above embodiments implements its functions, only the division of the above-mentioned various functional modules is used for illustration. In practical applications, the above functions can be allocated to different functional modules according to actual needs, that is, the content structure of the device is divided into different functional modules to complete all or part of the functions described above.

[0134] Regarding the apparatus in the above embodiments, the specific manners in which each module performs operations have been described in detail in the embodiments related to the method, and will not be elaborated herein.

[0135] Please refer to Figure 8 , which shows a structural block diagram of a computer device 800 provided in an embodiment of the present application.

[0136] Generally, the computer device 800 includes: a processor 810 and a memory 820.

[0137] The processor 810 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 810 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), and PLA (Programmable Logic Array). The processor 810 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the wake state, also known as the CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 810 may be integrated with a GPU (Graphics Processing Unit), and the GPU is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 810 may further include an AI processor, and the AI processor is used to process computing operations related to machine learning.

[0138] The memory 820 may include one or more computer-readable storage media, and the computer-readable storage media may be non-transitory. The memory 820 may further include a high-speed random access memory and a non-volatile memory, such as one or more disk storage devices and flash storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 820 is used to store a computer program, and the computer program is configured to be executed by one or more processors to implement the above-mentioned 3D generation model training method.

[0139] Those skilled in the art can understand that Figure 8 the structure shown in

[0140] does not constitute a limitation on the computer device 800, and may include more or fewer components than shown in the figure, or combine certain components, or adopt different component arrangements.

[0141] Optionally, the computer-readable storage medium may include: ROM (Read-Only Memory), RAM (Random-Access Memory), SSD (Solid State Drives), or optical discs, etc. Among them, the random access memory may include ReRAM (Resistance Random Access Memory) and DRAM (Dynamic Random Access Memory).

[0142] In some embodiments, a computer program product is also provided. The computer program product includes a computer program, and the computer program is stored in a computer-readable storage medium. The processor reads and executes the computer program from the computer-readable storage medium to implement the above-mentioned 3D generation model training method.

[0143] It should be understood that "a plurality of" as mentioned herein refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. In addition, the step numbers described in this article only exemplarily show a possible execution sequence between steps. In some other embodiments, the above steps may not be executed in the order of the numbers. For example, two steps with different numbers are executed simultaneously, or two steps with different numbers are executed in the reverse order of the illustration. The embodiments of the present application do not limit this.

[0144] The above are only exemplary embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A training method for a 3D generative model, characterized in that: The method comprises: Generate 3D construction data based on a generation rule of 3D data, wherein the 3D construction data is 3D data used to construct a first 3D model, and the generation rule is used to define a condition that the 3D data must satisfy; Generate 3D prediction data corresponding to the 3D construction data according to the initialized vector representation through the expression learning model, wherein the 3D prediction data is 3D data predicted by the expression learning model and used to construct the first 3D model; With the goal of minimizing the difference between the 3D prediction data and the 3D construction data, adjusting the parameters of the expression learning model to obtain a trained expression learning model; A 3D generation model is trained based on the trained expression learning model and the 3D sample data, wherein the 3D sample data is 3D data used to construct a second 3D model, and the 3D generation model is used to generate 3D data corresponding to the image based on the image.

2. The method according to claim 1, characterized in that The generation rule of the 3D data includes: the value range of the 3D data; The generating of 3D construction data based on the generating rule of 3D data includes: The 3D construction data is generated based on the numerical range of the 3D data, and the 3D construction data covers the numerical range.

3. The method according to claim 2, characterized in that The 3D data is used to indicate at least one of the following: a geometric feature and a color feature, wherein the geometric feature is used to characterize structural information of the first 3D model, and the color feature is used to characterize color information of the first 3D model.

4. The method according to any one of claims 1 to 3, characterized in that: The 3D generation model includes a first neural network and a second neural network; The step of training a 3D generation model based on the trained expression learning model and the 3D sample data includes: Determining a first 3D data representation corresponding to the 3D sample data by using the trained expression learning model; generating, by the first neural network, a first implicit representation corresponding to the first 3D data representation according to the first 3D data representation; Generate a predicted implicit representation corresponding to the sample image according to the sample image corresponding to the 3D sample data and the first random noise through the second neural network; With the goal of minimizing the difference between the predicted implicit representation and the first implicit representation, the parameters of the second neural network are adjusted to obtain a trained 3D generative model.

5. The method according to claim 4, characterized in that The determining, by means of the trained expression learning model, a first 3D data representation corresponding to the 3D sample data comprises: generating an initialized 3D data representation; Generate 3D prediction data corresponding to the 3D sample data according to the initialized 3D data representation through the trained expression learning model; With the goal of minimizing the difference between the 3D prediction data and the 3D sample data, the initialized 3D data representation is updated to obtain an updated 3D data representation as the first 3D data representation.

6. The method according to claim 4 or 5, characterized in that: The first neural network includes an encoder and a decoder; the method further includes: generating, by the encoder, a second implicit representation corresponding to the first 3D data representation according to the first 3D data representation; generating, by the decoder, a reconstructed first 3D data representation according to a second implicit representation corresponding to the first 3D data representation; With the goal of minimizing the difference between the first 3D data representation and the reconstructed first 3D data representation, the parameters of the first neural network are adjusted to obtain a pre-trained first neural network, and the pre-trained first neural network is used to generate a first implicit representation corresponding to the first 3D data representation based on the first 3D data representation.

7. The method according to any one of claims 4 to 6, characterized in that: The trained 3D generative model includes a trained second neural network, and the method further includes: acquiring a first image; Generate a predicted implicit representation corresponding to the first image according to the first image and second random noise by the trained second neural network; generating, by a decoder of the first neural network, a second 3D data representation based on the predicted implicit representation corresponding to the first image; Generate 3D prediction data corresponding to the first image according to the second 3D data representation through the trained expression learning model.

8. The method according to any one of claims 1 to 3, characterized in that: The 3D generation model includes: a third neural network and the trained expression learning model; The step of training a 3D generation model based on the trained expression learning model and the 3D sample data includes: generating, by the third neural network, a 3D data representation corresponding to the sample image according to the sample image corresponding to the 3D sample data; Generate 3D prediction data corresponding to the sample image according to the 3D data representation by the trained expression learning model; With the goal of minimizing the difference between the 3D prediction data and the 3D sample data, the parameters of the third neural network are adjusted to obtain a trained 3D generation model.

9. The method according to claim 8, characterized in that The trained 3D generative model includes a trained third neural network, and the method further includes: acquiring a first image; generating, based on the first image, a third 3D data representation corresponding to the first image by the trained third neural network; Generate 3D prediction data corresponding to the first image according to the third 3D data representation through the trained expression learning model.

10. A training device for a 3D generative model, characterized in that: The device comprises: A first generating module, configured to generate 3D construction data based on a generation rule of 3D data, wherein the 3D construction data is 3D data used to construct a first 3D model, and the generation rule is used to define a condition to be satisfied by the 3D data; A second generating module, configured to generate 3D prediction data corresponding to the 3D construction data according to the initialized vector representation through an expression learning model, wherein the 3D prediction data is 3D data predicted by the expression learning model and used to construct the first 3D model; An adjustment module, configured to adjust the parameters of the expression learning model with the goal of minimizing the difference between the 3D prediction data and the 3D construction data, so as to obtain a trained expression learning model; A training module is used to train a 3D generation model based on the trained expression learning model and 3D sample data, wherein the 3D sample data is 3D data used to construct a second 3D model, and the 3D generation model is used to generate 3D data corresponding to the image based on the image.

11. A computer device, characterized in that: The computer device comprises a processor and a memory, wherein a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which is loaded and executed by a processor to implement the method according to any one of claims 1 to 9.

13. A computer program product, characterized in that The computer program product comprises a computer program, which is loaded and executed by a processor to implement the method according to any one of claims 1 to 9.

Images