Method, device, equipment, medium and product for generating three-dimensional model

Abstract

The application discloses a three-dimensional model generation method, device, equipment, medium and product. The method comprises the following steps: obtaining model prompt information; in the i-th prediction stage, obtaining the primitive parameters corresponding to the primitives predicted in the first i-1 prediction stages; fusing the model prompt information and the primitive parameters corresponding to the primitives predicted in the first i-1 prediction stages to obtain a fused feature representation; predicting the fused feature representation to obtain the primitive parameters corresponding to the primitives predicted in the i-th prediction stage until the end of the prediction stage to obtain a three-dimensional model. The primitive parameters of the primitives predicted in the previous prediction stages are used to constrain the primitive parameters of the primitives determined subsequently, the coordination between the primitives is improved, the problem that multiple primitives determined in parallel cause misalignment between the multiple primitives and weak matching is avoided, the accuracy of primitive determination is improved, and the expression ability of the primitives to the model components is improved.

Description

Technical Field

[0001] This application relates to the field of machine learning technology, and in particular to a method, apparatus, device, medium, and product for generating three-dimensional models. Background Technology

[0002] In computational graphics, a primitive refers to a simple geometric shape used to construct or decompose complex 3D models, such as a cuboid primitive or a sphere primitive. In some implementations, the primitives are transformed and then spliced ​​together to obtain a 3D model.

[0003] In related technologies, after obtaining the prompt information, a three-dimensional model is obtained by stitching together the primitives obtained from the prompt information through a pre-trained machine learning model.

[0004] However, when building a 3D model using primitives, poor coordination between multiple primitives can lead to a lower accuracy of the 3D model. Summary of the Invention

[0005] This application provides a method, apparatus, device, medium, and product for generating three-dimensional models. The technical solutions provided by this application include the following aspects.

[0006] According to one aspect of the embodiments of this application, a method for generating a three-dimensional model is provided, the method comprising:

[0007] Obtain model hint information, which indicates the requirement to generate the three-dimensional model using at least one primitive, where the primitive refers to a three-dimensional geometric element;

[0008] In the i-th prediction stage, the primitive parameters corresponding to the primitives predicted in the previous i-1 prediction stages are obtained, where i > 1 and i is an integer. The primitive parameters are used to indicate the geometric shape of the primitive.

[0009] By fusing the model prompt information and the primitive parameters corresponding to the primitives predicted in the first i-1 prediction stages, a fused feature representation is obtained;

[0010] The fusion feature representation is predicted to obtain the primitive parameters corresponding to the primitives predicted in the i-th prediction stage, until the prediction stage ends and the three-dimensional model is obtained.

[0011] According to one aspect of the embodiments of this application, a three-dimensional model generation apparatus is provided, the apparatus comprising:

[0012] The acquisition module is used to acquire model prompt information, which indicates the requirement to generate the three-dimensional model through at least one primitive, where the primitive refers to a three-dimensional geometric element.

[0013] The acquisition module is further configured to acquire the primitive parameters corresponding to the primitives predicted in the previous i-1 prediction stages in the i-th prediction stage, where i > 1 and i is an integer, and the primitive parameters are used to indicate the geometric shape of the primitive.

[0014] The fusion module is used to fuse the model prompt information and the primitive parameters corresponding to the primitives predicted in the first i-1 prediction stages to obtain a fused feature representation;

[0015] The prediction module is used to predict the fused feature representation, obtain the primitive parameters corresponding to the primitives predicted in the i-th prediction stage, and obtain the three-dimensional model at the end of the prediction stage.

[0016] According to one aspect of the present application, a computer device is provided, the computer device including a processor and a memory, the memory storing a computer program, the computer program being loaded and executed by the processor to implement the above-described method for generating a three-dimensional model.

[0017] According to one aspect of the embodiments of this application, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, the computer program being loaded and executed by a processor to implement the above-described method for generating a three-dimensional model.

[0018] According to one aspect of the embodiments of this application, a computer program product is provided, the computer program product including a computer program stored in a computer-readable storage medium, and a processor reading from the computer-readable storage medium and executing the computer program to implement the above-described method for generating a three-dimensional model.

[0019] The technical solutions provided in this application can bring at least the following beneficial effects:

[0020] By iteratively determining the primitive parameters of at least two primitives, the process of sequentially determining primitives to build a 3D model in human cognition is simulated. The primitive parameters of the primitives predicted in the preceding prediction stage are used to constrain the primitive parameters of the subsequently determined primitives, thereby improving the coordination between the primitives and avoiding the problems of misalignment and weak compatibility among multiple primitives caused by determining multiple primitives in parallel. This improves the accuracy of primitive determination and enhances the ability of primitives to express the components of the model. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the interface for a three-dimensional model generation process provided in an exemplary embodiment of this application;

[0022] Figure 2 This is a schematic diagram of a computer system provided in an exemplary embodiment of this application;

[0023] Figure 3 This is a flowchart of a method for generating a three-dimensional model provided in an exemplary embodiment of this application;

[0024] Figure 4 This is a schematic diagram of the feature generation process of model prompt information provided in an exemplary embodiment of this application;

[0025] Figure 5 This is a schematic diagram of a primitive prediction process provided in an exemplary embodiment of this application;

[0026] Figure 6 This is a schematic diagram of the decoding framework of a decoder provided in an exemplary embodiment of this application;

[0027] Figure 7 This is a flowchart of a training method for a primitive prediction network provided in an exemplary embodiment of this application;

[0028] Figure 8 This is a flowchart of a method for generating a three-dimensional model provided in another exemplary embodiment of this application;

[0029] Figure 9 This is a schematic diagram of the transformation result of a cube primitive provided in an exemplary embodiment of this application;

[0030] Figure 10 This is a schematic diagram of the transformation of a hexagonal prism primitive provided in an exemplary embodiment of this application;

[0031] Figure 11 This is a block diagram of a three-dimensional model generation apparatus provided in an exemplary embodiment of this application;

[0032] Figure 12 This is a block diagram of a three-dimensional model generation apparatus provided in another exemplary embodiment of this application;

[0033] Figure 13 This is a structural block diagram of a computer device provided in one embodiment of this application. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0035] First, a brief introduction to the terms used in the embodiments of this application will be given.

[0036] Computer Vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to using cameras and computers to replace human eyes in recognizing and measuring targets, and then performing image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, 3D model generation, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, and map building, as well as common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0037] In computational graphics, a primitive refers to a geometric shape used to construct or decompose complex 3D models. In some implementations, a primitive refers to a 3D geometric shape, typically including: cube primitives, cylinder primitives, ellipsoid primitives, hyperquadratic surface primitives, etc. Each 3D geometric primitive has a basic form. For example, the basic form of a cube primitive is a cube with a length, width, and height of 3, 2, and 4, respectively; the basic form of a cylinder primitive is a cylinder with a radius of 1 and a height of 2.

[0038] 3D model generation: This technique has applications in various fields, used to generate 3D models corresponding to text / images or point cloud data. The 3D model generation involved in this application embodiment mainly involves constructing a 3D model through the transformation and splicing of various preset types of primitives.

[0039] The scenarios for generating 3D models include at least one of the following: 1. In game development and user-generated content (UGC) scenarios, users can express their need to generate 3D models by inputting text or images. For example, if a user inputs "create a car", then based on this text condition, a 3D model of the vehicle is generated by splicing primitives. Figure 1 This is a schematic diagram of the interface for a three-dimensional model generation process provided in an exemplary embodiment of this application, such as... Figure 1As shown, a virtual scene and a model creation area 100 are displayed. The model creation area 100 includes a prompt information input area 110. The user enters a prompt information in the prompt information input area 110, such as "Create a car in the scene". Based on the prompt information, a three-dimensional model 120 of the vehicle is displayed in the virtual scene. This three-dimensional model is obtained by splicing together 8 primitives, including the roof (cube primitive), windows (triangular prism primitive), body (quadratic frustum primitive), bottom (cube primitive), wheels (two cylindrical primitives), and rearview mirrors (two ellipsoidal primitives).

[0040] 2. In the field of industrial design, it is often necessary to abstract complex parts into 3D models that conform to engineering specifications for structural analysis. Illustratively, the user inputs point cloud data of an industrial structure. Based on this point cloud data as the model generation condition, a 3D model corresponding to the industrial structure, composed of primitives, is generated. For example, if the user inputs point cloud data of an engine block, a 3D model corresponding to the structure of the point cloud data is automatically generated, including the main engine structure (cubic primitives) and the piston bore structure (cylindrical primitives).

[0041] 3. In the field of Extended Reality (XR), including Virtual Reality (VR) and Augmented Reality (AR) scenarios, taking AR navigation as an example, after collecting point cloud data of buildings in the environment through the navigation device's camera, a corresponding 3D model of the building is automatically generated. For example, during AR navigation, the camera collects environmental depth information to generate point cloud data of buildings. The system automatically generates a 3D model of the building based on the point cloud data and displays the 3D model of the building in the AR screen, including the main body of the building (cube primitive), the roof of the building (triangular prism primitive), etc.

[0042] It is worth noting that the above-described scenarios for generating 3D models are merely illustrative examples, and the embodiments of this application do not limit the scenarios for generating 3D models.

[0043] Please refer to Figure 2 This illustrates a schematic diagram of a computer system provided in one embodiment of this application.

[0044] The computer system 200 includes: a terminal 220 and a server 240.

[0045] Terminal 220 has an application installed and running that has the function of generating 3D models. The device type of terminal 220 includes at least one of the following: smartphone, laptop, desktop computer, tablet, smart robot, AR (Augmented Reality) device, VR (Virtual Reality) device, in-vehicle terminal, etc.

[0046] Terminal 220 is connected to server 240 via wireless or wired network.

[0047] Those skilled in the art will understand that the number of the aforementioned devices can be more or less. For example, there may be only one device, or there may be dozens or hundreds of devices, or even more. This application does not limit the number or type of devices.

[0048] Server 240 includes at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. Server 240 provides functional services for implementing model training. Optionally, server 240 undertakes the main computing work, and terminal 220 undertakes the secondary computing work; or, server 240 undertakes the secondary computing work, and terminal 220 undertakes the main computing work; or, server 240 and terminal 220 collaborate on computing using a distributed computing architecture.

[0049] It is worth noting that the aforementioned server 240 can be implemented as a physical server or as a cloud server in the cloud. In some embodiments, the aforementioned server 240 can also be implemented as a node in a blockchain system.

[0050] Optionally, the method for generating the three-dimensional model provided in this application embodiment can be executed independently by the terminal 220 or the server 240, or it can be implemented by the terminal 220 and the server 240 in cooperation, which is not limited here.

[0051] In one example, the method for generating the 3D model is implemented independently by terminal 220. For instance, terminal 220 is equipped with an Artificial Intelligence (AI) chip, which deploys a primitive prediction network. When a user requests the generation of a 3D model, the application in terminal 220 obtains model prompts from the user, such as text prompts, image prompts, or point cloud data prompts. These prompts are then input into the primitive prediction network, which generates a 3D model composed of at least one primitive based on the prompts. The structure of this 3D model matches the model prompts. During the prediction of the 3D model, the primitive parameters of subsequent primitives are determined iteratively based on the primitive parameters of previously determined primitives. Specifically, the model prompts are first obtained, and the primitive parameters of the first primitive are output based on these prompts. Then, the primitive parameters of the first primitive are fused with the model prompts to obtain a fused feature representation. The primitive parameters of the second primitive are then predicted based on this fused feature representation, and so on. That is, when predicting the primitive parameters of the i-th primitive, the primitive parameters of the first i-1 primitives are obtained, and the primitive parameters of the first i-1 primitives and the model hint information are fused to obtain the fused feature representation, thereby predicting the primitive parameters of the i-th primitive.

[0052] Optionally, when predicting a 3D model, at least one primitive parameter is first obtained, including primitive type parameters and primitive shape parameters. The primitive type parameter expresses the geometric shape of the primitive, and the primitive shape parameter expresses the transformation method of the primitive on the basic shape, such as: stretching ratio, rotation direction, scaling ratio, etc. Then, due to the symmetry of some primitives, there are cases where different shape parameters correspond to the same transformation result. For example, rotating a cube primitive 180° around the x-axis and rotating it 180° around the y-axis express different rotation methods, but the rotation result is the same. Therefore, in this embodiment, an equivalent transformation set is set for each type of primitive. The equivalent transformation set corresponding to the primitive type parameter is obtained. The equivalent transformation set includes at least one candidate transformation scheme. At least one candidate transformation scheme has the same transformation result on the basic shape of the primitive. The primitive shape parameter is updated based on at least one candidate transformation scheme to obtain the updated primitive shape parameter, thereby constructing the 3D model.

[0053] In another example, the method for generating the 3D model is implemented collaboratively by terminal 220 and server 240. Illustratively, server 240 is equipped with a primitive prediction network to provide backend services for the application running on terminal 220. The user sends model hints to server 240 through the application on terminal 220. After receiving the model hints, server 240 inputs them into the primitive prediction network. The primitive prediction network then generates the 3D model using the method provided in this embodiment. Server 240 sends the 3D model to terminal 220, and terminal 220 displays the 3D model generated based on the model hints to the user.

[0054] Taking the generation process of a 3D model executed by server 240 as an example, after obtaining model hint information, server 240 inputs the model hint information into the primitive prediction network to obtain primitive parameters of at least one primitive. These include primitive type parameters and primitive shape parameters. The primitive type parameters express the geometric shape of the primitive, and the primitive shape parameters express the transformation method of the primitive on the basic shape. When predicting the 3D model, the primitive parameters of subsequent primitives are determined iteratively based on the primitive parameters of the previously determined primitives. That is, first, model hint information is obtained, and the primitive parameters of the first primitive are output according to the model hint information; then, the primitive parameters of the first primitive are fused with the model hint information to obtain a fused feature representation, and the primitive parameters of the second primitive are predicted based on the fused feature representation, and so on. That is, when predicting the primitive parameters of the i-th primitive, the primitive parameters of the previous i-1 primitives are obtained, and the primitive parameters of the previous i-1 primitives and the model hint information are fused to obtain a fused feature representation, thereby predicting the primitive parameters of the i-th primitive.

[0055] In an optional embodiment, the prediction process for at least two primitives in this application is performed iteratively. Figure 3 This is a flowchart illustrating a method for generating a three-dimensional model according to an exemplary embodiment of this application. The method is implemented using a computer device (which can be configured as follows). Figure 2 The method is executed by the terminal 220 or server 240 shown, and includes at least the following steps 310 to 340.

[0056] Step 310: Obtain model prompt information.

[0057] Model hints are used to indicate the requirement to generate a 3D model from at least one primitive, which refers to a 3D geometric element.

[0058] In some embodiments, model hints are used to express the need to generate a 3D model, and the model hints may take at least one of the following forms:

[0059] 1. Text prompts, which are text descriptions of the requirements for generating a 3D model.

[0060] Text prompts are used to express the requirements for generating a 3D model through text. For example, a text prompt might include "Generate a tricycle," which means that a vehicle needs to be generated with 3 wheels.

[0061] Among them, the text prompt information is the text content directly entered by the user, or the text prompt information is the text description information extracted from the image content entered by the user, or the text prompt information is the text conversion information obtained by converting the voice content entered by the user.

[0062] As an illustration, a text input box is displayed in the 3D model generation interface, where the user enters text content as a text prompt.

[0063] Alternatively, the 3D model generation interface may include an image upload area where users upload images captured in photos or stored locally. Taking the server-side 3D model generation process as an example, the terminal sends the user-uploaded image to the server. The server then uses a text generation model to predict the image and generate text description information. The text generation model is a pre-trained machine learning model used to describe the image content.

[0064] Alternatively, the 3D model generation interface can include an audio recording control. Users can trigger the recording operation by activating this control, which in turn activates the terminal's microphone to capture and record audio content. Taking the server-side 3D model generation process as an example, the terminal sends the user-recorded audio content to the server. The server then uses a text conversion model to convert the audio content into text, generating a text description. This text conversion model is a pre-trained machine learning model designed for converting speech to text.

[0065] 2. Image prompts, which means that the content of the input image serves as the requirement for generating a 3D model.

[0066] Image prompts are used to express the requirements for generating a 3D model through images. For example, if an image is input that includes a tricycle, such as an image taken by a camera of the tricycle, then the image indicates that the 3D model to be generated is a vehicle that matches the tricycle in the image.

[0067] The image prompts are either images directly uploaded by the user, or images generated based on text input by the user.

[0068] As an illustration, the 3D model generation interface includes an image upload area where users can upload images captured by cameras or stored locally.

[0069] Alternatively, the 3D model generation interface can include a text input box where the user enters text content. Taking the server-side 3D model generation process as an example, the terminal sends the user-inputted text content to the server, which then generates an image corresponding to the text content using an image generation model. This image generation model is a pre-trained machine learning model for generating images based on text.

[0070] 3. Point cloud prompts, which are the requirements for generating a 3D model expressed through point cloud data.

[0071] Point cloud hints are used to express the requirements for generating a 3D model through point cloud data. For example, if the outline of the point cloud data is the outline of a tricycle, then the point cloud data indicates that the 3D model to be generated is a vehicle, and the outline of the vehicle matches the structure of the point cloud data.

[0072] The point cloud prompt information can be obtained by converting depth images captured by a depth camera; or by directly acquiring point cloud data using a LiDAR; or by converting text content into point cloud data; or by converting images into point cloud data.

[0073] 4. Mesh prompts, which are the requirements for generating a 3D model expressed through a mesh model.

[0074] Among them, mesh prompts are prompts generated through mesh construction. For example, a model framework is built using a triangular mesh, and the mesh prompts indicate how to convert the mesh model into a 3D model composed of primitives.

[0075] 5. Voxel hints, which are the requirements for generating a 3D model by expressing the model built from voxel blocks.

[0076] Among them, voxel hints are hints generated by building voxel blocks. For example, a model framework is built using voxel blocks, and the model built using voxel hints is converted into a three-dimensional model composed of primitives.

[0077] It is worth noting that the above-mentioned method of obtaining model prompt information is only an illustrative example, and the specific method of obtaining model prompt information is not limited in the embodiments of this application.

[0078] In some embodiments, after obtaining model prompt information, the information feature representation of the model prompt information is obtained.

[0079] Information feature representation is used to express model prompts using feature vectors.

[0080] In some embodiments, an information feature representation of the model prompt information is generated by a pre-trained encoder. The encoder maps the model prompt information to a preset feature space, obtaining a vector representation of the model prompt information in the preset feature space, which serves as the information feature representation of the model prompt information.

[0081] In some embodiments, the information form of the model prompt information includes at least one of text form, image form, and point cloud form; in some embodiments, the information feature representation is extracted by the same encoder for model prompt information of different information forms, that is, the model prompt information is mapped to the same preset feature space for model prompt information of different information forms; or, the information feature representation is extracted by different encoders for model prompt information of different information forms.

[0082] Optionally, if the encoder is trained to extract information feature representations from model prompts in a preset form, then when other forms of model prompts are obtained, the other forms of model prompts are converted into the preset form, thereby extracting information feature representations from the converted model prompts.

[0083] Indicatively, if the encoder is trained to extract information feature representations from model cue information in point cloud form, then if model cue information in image form is obtained, the image-based model cue information is converted into point cloud-based model cue information through a pre-trained point cloud conversion model, and the encoder extracts information feature representations from the converted point cloud-based model cue information.

[0084] Indicative Figure 4 This is a schematic diagram of the feature generation process of model prompt information provided in an exemplary embodiment of this application, such as... Figure 4 As shown, point cloud data 410 is acquired and input into shape encoder 420, and the output is the information feature representation C corresponding to point cloud data 410.

[0085] Step 320: In the i-th prediction stage, obtain the primitive parameters corresponding to the primitives predicted in the previous i-1 prediction stages, where i > 1 and is an integer.

[0086] Optionally, if the primitive obtained in the (i-1)th prediction stage does not belong to the last primitive used to construct the 3D model, the primitive parameters corresponding to the primitives predicted in the previous (i-1)th prediction stages are obtained.

[0087] Primitive parameters are used to indicate the geometric shape of a primitive.

[0088] In some embodiments, in the first prediction stage, the above-mentioned information feature representation is predicted to obtain the primitive parameters corresponding to the primitives predicted in the first prediction stage.

[0089] Schematic, the information feature representation is input into a pre-trained primitive prediction network to predict the primitive parameters p1 = τ(P) of the first primitive. c1 ;s1,r1,t1), where P c1 s1 is the primitive type parameter, representing the primitive type of the first primitive, that is, the geometry of the primitive; s1 and r1 are primitive shape parameters, where s1 represents the scaling factor of the first primitive along at least one preset axis in the basic shape, and r1 represents the rotation angle of the first primitive around at least one preset rotation axis in the basic shape; t1 represents the coordinate position of the primitive center of the first primitive in the preset three-dimensional space.

[0090] Starting from the second prediction stage, in the second prediction stage, the primitive parameters corresponding to the primitives predicted in the first prediction stage are obtained. In the third prediction stage, the primitive parameters corresponding to the primitives predicted in the first two prediction stages are obtained.

[0091] Step 330: Integrate the model prompt information and the primitive parameters corresponding to the primitives predicted in the previous i-1 prediction stages to obtain the fused feature representation.

[0092] In some embodiments, the information feature representation of the model prompt information is obtained. Also, the parameter feature sequence corresponding to the primitive parameters predicted in the first i-1 prediction stages is obtained. The information feature representation and the parameter feature sequence are fused to obtain the fused feature representation of the i-th prediction stage.

[0093] By fusing information feature representation and parameter feature sequence, when predicting the i-th primitive in the i-th prediction stage, both the model prompt information and the previously predicted primitives are used as constraints. This ensures that the i-th primitive meets the prompt requirements of the model prompt information and improves the fit between the i-th primitive and the previous primitives.

[0094] In the process of obtaining the parameter feature sequence, the primitive parameters predicted in the first i-1 prediction stages are encoded by the parameter encoder to obtain the parameter feature representations of the primitive parameters predicted in the first i-1 prediction stages respectively. The parameter feature representations of the primitive parameters predicted in the first i-1 prediction stages are arranged in order to obtain the parameter feature sequence.

[0095] That is, the primitive parameters predicted in the first i-1 prediction stages are encoded to obtain the parameter feature representations corresponding to the first i-1 prediction stages respectively. The parameter feature representations corresponding to the first i-1 prediction stages are then concatenated in order to obtain the parameter feature sequence.

[0096] By mapping the primitive parameters predicted in the first i-1 prediction stages to the feature space, we obtain parameter feature representations. Then, we arrange and concatenate the parameter feature representations in the order of the prediction stages to obtain a parameter feature sequence. The parameter feature sequence expresses the previously determined primitives and their determination order, which improves the tightness of the relationship between primitives and improves the accuracy of the 3D model.

[0097] In some embodiments, after obtaining the predicted primitive parameters, each prediction stage encodes the predicted primitive parameters to obtain parameter feature representations and stores the parameter feature representations. Thus, when the parameter feature representations of the preceding stages are needed in subsequent prediction stages, the parameter feature representations of the preceding stages are obtained from the stored parameter feature representations, and the parameter feature representations of the preceding stages are concatenated according to the stage order to obtain a parameter feature sequence.

[0098] In some embodiments, the primitive parameter includes at least two sub-parameters. The at least two sub-parameters are encoded using the same parameter encoder to obtain the parameter feature representation corresponding to the primitive parameter; alternatively, the at least two sub-parameters are encoded using different parameter encoders, and sub-parameter feature representations are obtained separately. These sub-parameter feature representations are then fused to obtain the parameter feature representation corresponding to the primitive parameter. Illustratively, the primitive parameters include primitive type parameters, primitive scaling parameters, primitive rotation parameters, and primitive position parameters. The primitive type parameter expresses the geometric shape type of the primitive, such as parameter a for a cube primitive, parameter b for a sphere primitive, and parameter c for a hexagonal prism primitive. The primitive scaling parameter expresses the scaling factor of the primitive along at least one axis in its basic shape, where the axis controlling the scaling is predefined. The primitive rotation parameter indicates the rotation parameter of the primitive about at least one rotation axis in its basic shape, where the rotation axis and the axis controlling the scaling may be the same or different, and the rotation axis is predefined. The primitive position parameter indicates the coordinate position of the primitive center or a specified target point in three-dimensional space. In this system, a spatial coordinate system is constructed in three-dimensional space with a preset point as the origin. The primitive position parameter represents the coordinate position of the primitive center in the spatial coordinate system.

[0099] The type feature representation is obtained by encoding the primitive type parameter through the first encoder; the scaling feature representation is obtained by encoding the primitive scaling parameter through the second encoder; the rotation feature representation is obtained by encoding the primitive rotation parameter through the third encoder; and the position feature representation is obtained by encoding the primitive position parameter through the fourth encoder. The type feature representation, scaling feature representation, rotation feature representation, and position feature representation are fused to obtain the parameter feature representation corresponding to the primitive parameter.

[0100] In this embodiment, the example of encoding each sub-parameter of the primitive parameters by different parameter encoders is used for illustration. That is, the discretized primitive type parameter, primitive scaling parameter, primitive rotation parameter, and primitive position parameter are encoded into feature vectors by an embedding network (also known as the aforementioned parameter encoder).

[0101] In some embodiments, for the primitive p i , by primitive type P ci ∈P and its rigid body transformation τ(·) in three-dimensional space are jointly defined, that is, p i =τ(P ci ;s i r i , t i ), P ci Let s represent the primitive type of the i-th primitive, that is, the geometry of the i-th primitive. i r represents the scaling factor of the i-th primitive along at least one axis. i t represents the rotation angle of the i-th primitive along at least one rotation axis. i This represents the coordinate position of the center of the i-th primitive in three-dimensional space. A spatial coordinate system is constructed in three-dimensional space with a preset point as the origin, and t... i This represents the coordinate position of the center of the i-th primitive in the spatial coordinate system.

[0102] Schematic representation of the primitive type parameter P of the discretized primitive. ci By embedding network e c Encoding is used as a type feature representation; the primitive scaling parameter s of the primitives is used to represent the primitives. i By embedding network e s Encode as a scaling feature representation; the primitive rotation parameter r of the primitive is used. i By embedding network e r Encoding is a rotational feature representation; and, the primitive position parameter t of the primitive is taken. i By embedding network e t The encoding is a positional feature representation. The above-mentioned feature representations, including scaled, rotated, and positional feature representations, are then fused together, such as through concatenation, feature addition, and feature multiplication, to obtain a parametric feature representation.

[0103] Schematic representation of the parametric features of the i-th primitive h i =ε(e c (Pc i ), e s (s i ), e r (r i ), e t (t iThis parameter characteristic can also be called the primitive token h of the i-th primitive. i Where ε(·) represents the encoding fusion operation.

[0104] It is worth noting that the above primitive type parameter P ci During encoding, the complete primitive type parameter P can be... ci Encoding can also be used for c to express types. i Regarding encoding, this application embodiment does not limit the encoding method of primitive type parameters.

[0105] In some embodiments, obtaining the fused feature representation includes at least one of the following methods:

[0106] 1. Concatenate the information feature representation and the parameter feature sequence to obtain the fused feature representation;

[0107] 2. Concatenate the information feature representation and the parameter feature sequence to obtain the intermediate feature representation; decode the intermediate feature representation using a pre-trained decoder to obtain the fused feature representation.

[0108] Indicatively, the fused feature representation is obtained by concatenating the decoded information feature representation and the parameter feature sequence obtained by pre-trained Transformer network.

[0109] For example: f i =S(C; h1, h2, ..., h i-1 ), where f i Let h represent the fusion feature representation of the i-th prediction stage, C represent the information feature representation, h1 represent the parametric feature representation corresponding to the first primitive obtained from the prediction, h2 represent the parametric feature representation corresponding to the second primitive obtained from the prediction, and h i-1 This represents the parametric feature representation corresponding to the (i-1)th primitive obtained from the prediction. S represents the Transformer decoder.

[0110] It is worth noting that the Transformer network described above is only an illustrative example. In this embodiment, the prediction process of primitives in the 3D model can also be achieved by using diffusion models, graph neural networks, or other networks. Alternatively, a hybrid approach can be used in this embodiment, such as using an autoregressive Transformer to generate primitive parameters, a diffusion model to further denoise and reduce the cumulative error of sequence generation, and a graph neural network to optimize local parameters by considering spatial and semantic relationships.

[0111] Step 340: Predict the fusion feature representation and obtain the primitive parameters corresponding to the primitives predicted in the i-th prediction stage, until the prediction stage ends and the three-dimensional model is obtained.

[0112] In some embodiments, when predicting the fused feature representation, at least two primitives are obtained from a predefined set of primitives, wherein the set of primitives includes at least two types of primitives, and the at least two types of primitives have different geometric shapes. Each primitive corresponds to a basic shape, such as: the basic shape of the cube primitive is a cube with length, width, and height of 3, 2, and 4, respectively.

[0113] When predicting the fusion feature representation, the primitive to be applied is selected from at least two primitives provided by the primitive set, and the basic morphological transformation of the primitive is performed to obtain the transformed primitive. The transformed primitive is then applied to the construction of the 3D model.

[0114] It is worth noting that, in the above embodiments, the example of obtaining at least two primitives from a predefined primitive set is used for illustration. In some embodiments, primitives may also be obtained from a public primitive database or through other means, and this application does not limit this.

[0115] In some embodiments, after obtaining the fused feature representation, the fused feature representation is decoded by a decoder to obtain the primitive parameters of the primitives predicted in the i-th prediction stage.

[0116] Indicative Figure 5 This is a schematic diagram of a primitive prediction process provided in an exemplary embodiment of this application, as shown below. Figure 5 As shown, for image 500, point cloud data 510 of the image content in image 500 is generated, such as... Figure 5 As shown in image 500, the image content is a chair. Point cloud data 510 is input as model cue information into encoder 520, and the output is the information feature representation C of point cloud data 510. The information feature representation C is decoded by Transformer decoder 530 to first obtain the start token. <sos>The starting token <sos>Used to express the initial prediction of primitives in the 3D model. This information feature representation C and the starting token are predicted by decoder 530. <sos>The fused feature representation f0 of the first prediction stage is obtained. The decoder 540 predicts the fused feature representation f0, and the primitives predicted in the first prediction stage are obtained. Obtain the primitives predicted in the first prediction stage. The equivalent transformation set corresponding to the primitive type parameter is obtained, and the primitive morphological parameter is updated according to at least one candidate transformation scheme in the equivalent transformation set to obtain the updated primitive morphological parameter, which constitutes the primitive p0 predicted in the first prediction stage.

[0117] In the second prediction stage, the primitive parameters of the primitive p0 predicted in the first prediction stage are encoded by encoder 550 to obtain the parametric feature representation h0. The information feature representation C and the parametric feature representation h0 are fused to obtain the intermediate feature representation, which is then decoded by Transformer decoder 530 to obtain the fused feature representation f1. The fused feature representation f1 is then predicted by decoder 540 to obtain the primitive predicted in the second prediction stage. Obtain the primitives predicted in the second prediction stage The equivalent transformation set corresponding to the primitive type parameter is obtained, and the primitive morphological parameter is updated according to at least one candidate transformation scheme in the equivalent transformation set to obtain the updated primitive morphological parameter, which constitutes the primitive p1 predicted in the second prediction stage.

[0118] This process continues until the prediction is complete.

[0119] In some embodiments, the primitive parameters corresponding to the primitives predicted in the i-th prediction stage are obtained by decoding the fusion feature representation through the first decoder; the terminator prediction result of the i-th prediction stage is obtained by decoding the fusion feature representation through the second decoder.

[0120] By predicting terminators, that is, dynamically determining whether each prediction stage belongs to the last primitive prediction stage for generating the 3D model, the problem of rigid and inflexible 3D model construction by fixed-length primitives is avoided, thus improving the flexibility and accuracy of determining primitives to construct the 3D model.

[0121] The fused feature representation is decoded by an additional second decoder to obtain the terminator prediction result. A separate branch is used to determine the position of the terminator. That is, a separate decoder branch dynamically determines whether each prediction stage belongs to the last primitive prediction stage of generating the 3D model based on the training, which improves the flexibility and accuracy of determining the primitives to construct the 3D model.

[0122] In some embodiments, the determination of the end of the prediction process includes at least one of the following:

[0123] 1. Predict the fusion feature representation of the (i+1)th prediction stage to obtain the termination symbol prediction result of the (i+1)th prediction stage. The termination symbol prediction result indicates whether the prediction process ends in the i-th prediction stage, i.e., whether the prediction process ends in the i-th prediction stage. Correspondingly, the termination symbol prediction result indicates whether the primitives predicted in the i-th prediction stage are the last primitives constituting the 3D model. In response to the termination symbol generated in the (i+1)th prediction stage, generate the 3D model based on the primitives determined in the i-th prediction stage.

[0124] 2. Predict the fusion feature representation of the i-th prediction stage to obtain the termination symbol prediction result of the i-th prediction stage. The termination symbol prediction result indicates whether the prediction process ends in the i-th prediction stage, i.e., whether the prediction process ends in the i-th prediction stage. Correspondingly, the termination symbol prediction result indicates whether the primitive predicted in the i-th prediction stage is the last primitive constituting the 3D model. In response to the termination symbol generated for the i-th prediction stage, generate the 3D model based on the primitives determined in the i-th prediction stage.

[0125] In some embodiments, the fused feature representation is input to the terminator decoder, and the output is the terminator prediction result. (Illustrative example, such as...) Figure 5 As shown, after the decoder predicts the primitive parameters of the primitive in the nth prediction stage, it generates the fusion feature representation of the (n+1)th prediction stage based on the n primitives generated in the previous n prediction stages, and predicts the fusion feature representation through the decoder 530. If the output terminator <EOS> is output, it means that the primitive parameters of the primitive in the (n+1)th prediction stage are not output. The primitives predicted in the previous n prediction stages constitute a three-dimensional model.

[0126] Optionally, the above Figure 5 The decoder 540 shown includes decoders for at least two sub-parameters of the primitive parameters. In some embodiments, there is a dependency between the decoders of the at least two sub-parameters, i.e., the decoder 540 is implemented as a cascaded primitive decoder, and the decoder 540 explicitly models the dependencies between the primitive sub-parameters.

[0127] In some embodiments, the primitive parameters include primitive type parameters, primitive scaling parameters, primitive rotation parameters, and primitive position parameters. Accordingly, the decoder 540 described above includes a category decoder, a scaling decoder, a rotation decoder, and a position decoder. Optionally, the primitive class parameter of the i-th primitive is obtained by decoding the fusion feature representation of the i-th prediction stage through a class decoder; the primitive position parameter of the i-th primitive is obtained by decoding the fusion feature representation of the i-th prediction stage and the class feature representation of the primitive class parameter predicted in the i-th prediction stage through a position decoder; the primitive rotation parameter of the i-th primitive is obtained by decoding the fusion feature representation of the i-th prediction stage, the class feature representation of the primitive class parameter predicted in the i-th prediction stage, and the position feature representation of the primitive position parameter predicted in the i-th prediction stage through a rotation decoder; and the primitive scaling parameter of the i-th primitive is obtained by decoding the fusion feature representation of the i-th prediction stage, the class feature representation of the primitive class parameter predicted in the i-th prediction stage, the position feature representation of the primitive position parameter predicted in the i-th prediction stage, and the rotation feature representation of the primitive rotation parameter predicted in the i-th prediction stage through a scaling decoder.

[0128] To illustrate, the decoding process of decoder 540 includes the following:

[0129] Category decoding: c i =D c (f i ), where f i D represents the fusion feature representation of the i-th prediction stage. c For category decoders, c i This represents the primitive category parameter predicted in the i-th prediction stage.

[0130] Position decoding: t i =D t (f i e c (c i ), where e c D represents the encoder that encodes the primitive category parameters. t For the location decoder, t i This represents the primitive position parameters predicted in the i-th prediction stage.

[0131] Rotation Decoding: r i =D r (f i e c (c i ), e t (t i ), where e t D represents the encoder that encodes the position parameters of primitives. r For rotation decoder, r i This represents the primitive position parameters predicted in the i-th prediction stage.

[0132] Scalable Decoding: s i =D s (f i e c (c i ), e t (t i ), e r (r i ), where e r D represents the encoder that encodes the rotational parameters of the primitives. s For a scaling decoder, s i This represents the primitive scaling parameter predicted in the i-th prediction stage.

[0133] Indicative Figure 6 This is a schematic diagram of the decoding framework of decoder 540 provided in an exemplary embodiment of this application. Figure 6 As shown, for the four sub-parameters in the primitive parameters, the decoder 540 includes a four-layer cascaded decoding network. The first layer of the decoding network is used to process the fused feature representation f. i Decoding yields the primitive category parameter 610 of the i-th primitive; the second-layer decoding network is used to process the fused feature representation f. i The primitive category parameter 610 output by the first layer decoding network is decoded to obtain the primitive position parameter 620 of the i-th primitive; based on the cascade relationship, the third layer decoding network is used to decode the fused feature representation f. i The class feature representation of the primitive class parameter 610 output by the first layer decoding network and the position feature representation of the primitive position parameter 620 output by the second layer decoding network are decoded to obtain the primitive rotation parameter 630 of the i-th primitive; based on the cascade relationship, the fourth layer decoding network is used to decode the fused feature representation f i The primitive category parameter 610 output by the first layer decoding network, the position feature representation of the primitive position parameter 620 output by the second layer decoding network, and the rotation feature representation of the primitive rotation parameter 630 output by the third layer decoding network are decoded to obtain the primitive scaling parameter 640 of the i-th primitive.

[0134] The design of the aforementioned decoder 540 takes into account the natural relationships between primitive attributes, including: the choice of primitive type affects its possible position, rotation, and scaling parameters, and is consistent with human abstract logic, selecting the type first, determining the position, and then adjusting the rotation angle and scaling ratio.

[0135] When constructing a 3D model, a 3D model composed of n primitives can be approximately represented as: in, n represents the total number of primitives used to construct the 3D model. Where p i =τ(P ci ;s i r i , t i ).

[0136] In summary, the method provided in this application, by iteratively determining the primitive parameters of at least two primitives, simulates the process of gradually building a three-dimensional model by sequentially determining primitives in human cognition. By constraining the primitive parameters of subsequently determined primitives through the primitive parameters of primitives already predicted in the preceding prediction stage, the method improves the coordination between primitives, avoids the problems of misalignment and weak compatibility among multiple primitives caused by determining multiple primitives in parallel, improves the accuracy of primitive determination, and enhances the expressive ability of primitives to represent the components of the model.

[0137] In an optional embodiment, the prediction method for the above-described 3D model is performed through a primitive prediction network. The primitive prediction network includes the aforementioned... Figure 5 The encoder and decoder are shown. Optionally, the training process of the primitive prediction network includes steps 710 to 730 as follows.

[0138] Step 710: Obtain the sample prompt information and the corresponding reference 3D model.

[0139] The reference 3D model is a 3D model constructed by at least one first primitive.

[0140] The reference 3D model is a model used to characterize the 3D representation of the sample prompt information by annotating the sample prompt information. The reference 3D model is a 3D model obtained by manually splicing at least one first primitive.

[0141] In some embodiments, the sample prompt information includes at least one of sample text content, sample image, and sample point cloud data.

[0142] When acquiring sample prompt information and the corresponding reference 3D model, at least one of the following situations applies.

[0143] 1. Obtaining Sample Hint Information: Data in text, image, or point cloud format can be obtained as sample hint information; alternatively, the feature representation of the sample information encoded by an encoder can be obtained as sample hint information. When obtaining the feature representation of the sample information encoded by the encoder, the encoder is a pre-trained encoder used to encode the hint information. In some embodiments, the encoders for the sample hint information and the model hint information are the same, and the encoders for both are independent of the primitive prediction network; alternatively, the encoders for both the sample hint information and the model hint information are pre-trained encoders within the primitive prediction network.

[0144] In some embodiments, after obtaining sample prompt information, if the sample prompt information is implemented as point cloud data, it is directly input into the encoder to obtain the sample information feature representation; if the sample prompt information is implemented as information in other data forms, it is first converted into point cloud data, and then the converted point cloud data is encoded by the encoder to obtain the sample information feature representation. Specifically, the point cloud data is encoded into a sample information feature representation (or token sequence) of a preset length using a shape encoder.

[0145] 2. Obtaining a reference 3D model: In some embodiments, a 3D model composed of at least one first primitive is manually designed as a reference 3D model for the sample prompt information. The reference 3D model is bound to the sample prompt information and used as a 3D model to express the information content of the sample prompt information.

[0146] The reference 3D model is expressed as follows: n1 represents the total number of the first primitives used to construct the reference 3D model. p i1 =τ(P ci1 ;s i1 r i1 , t i1 The order in which the first primitives are represented in the reference 3D model indicates the order in which the first primitives are determined to be used to construct the reference 3D model, that is, according to p. 11 p 21 ... p n1 The order determines the first primitive.

[0147] Step 720: Predict sample prompt information through the primitive prediction network to obtain the predicted 3D model.

[0148] The predicted 3D model is a 3D model that is constructed by at least one second primitive.

[0149] At least one sequentially arranged second primitive is obtained through the primitive prediction network in the order of prediction stages, wherein one second primitive is predicted in each prediction stage.

[0150] The sample cue information is input into the shape encoder, and the output is the sample information feature representation of the sample cue information. Here, the shape encoder is the encoder in the primitive prediction network and is trained synchronously during training, or the shape encoder is an encoder outside the primitive prediction network and is not trained in the current training.

[0151] The execution process of the primitive prediction network is as follows: the feature representation of the sample information is decoded by the first decoder, and the starting token is inserted first. <sos>The starting token <sos>Used to express the initial prediction of the second primitive in the predicted 3D model. The sample information feature representation and the starting token are predicted by the first decoder. <sos>The fusion feature representation f of the first prediction stage is obtained. 21 The fused feature representation f is predicted by the second decoder. 21 The second primitive obtained from the first prediction stage is obtained. Obtain the second primitive obtained from the first prediction stage. The equivalent transformation set corresponding to the primitive type parameters is determined, and the primitive morphological parameters are updated according to at least one candidate transformation scheme in the equivalent transformation set to obtain the updated primitive morphological parameters, which constitute the second primitive p predicted in the first prediction stage. 21 .

[0152] In the second prediction stage, the second primitive p predicted in the first prediction stage is encoded by the encoder. 21 The primitive parameters are used to obtain the parameter characteristic representation h. 21 The sample information feature representation and parameter feature representation h 21 The intermediate feature representations are obtained by fusing the data and then decoded by the first decoder to obtain the fused feature representation f. 22 The fused feature representation f is predicted by the second decoder. 22 The second primitive obtained from the second prediction stage is obtained. Obtain the second primitive obtained from the second prediction stage. The equivalent transformation set corresponding to the primitive type parameters is determined, and the primitive morphological parameters are updated according to at least one candidate transformation scheme in the equivalent transformation set to obtain the updated primitive morphological parameters, which constitute the second primitive p predicted in the second prediction stage. 22 .

[0153] This process continues until the prediction is complete.

[0154] In some embodiments, the fused feature representation is input to the terminator decoder, and the terminator prediction result is output. If the terminator is output in the i-th prediction stage or the (i+1)-th stage... <eos>, which means that the primitives predicted in the first i prediction stages constitute a three-dimensional model.

[0155] The predicted 3D model is expressed as follows: n2 represents the total number of second primitives used to construct the predicted 3D model. p i2 =τ(P ci2 ;s i2 r i2 , t i2 The order in which the second primitives are expressed in the predicted 3D model indicates the order in which the second primitives are determined by the primitive prediction network to be used to construct the predicted 3D model, that is, according to p. 12 p 22 ... p n2 The order determines the second primitive.

[0156] Step 730: Train the primitive prediction network based on the difference between the predicted 3D model and the reference 3D model.

[0157] In some embodiments, a loss value is obtained based on the difference between the reference 3D model and the predicted 3D model, and a primitive prediction network is trained based on the loss value.

[0158] When obtaining the loss value based on the difference between the reference 3D model and the predicted 3D model, at least one of the following losses is included:

[0159] 1. Obtain a sequence of first primitives consisting of at least one first primitive in the reference 3D model. The sequence of first primitives is a sequence obtained by arranging the first primitives in the generation order of at least one first primitive. Obtain a sequence of second primitives consisting of at least one second primitive in the prediction 3D model. The sequence of second primitives is a sequence obtained by arranging the second primitives in the prediction order. Train a primitive prediction network based on the difference between the first primitive sequence and the second primitive sequence.

[0160] Among them, when training the primitive prediction network based on the difference between the first primitive sequence and the second primitive sequence, at least one of the following training methods is included:

[0161] 1.1. Based on the primitive parameters of the i-th first primitive in the first primitive sequence and the primitive parameters of the i-th second primitive in the second primitive sequence, obtain the first loss value, where i is a positive integer; train the primitive prediction network using the first loss value.

[0162] In some embodiments, cross-entropy loss, i.e., the first loss value, is calculated for the first primitive in the first primitive sequence of the reference 3D model and the second primitive in the second primitive sequence of the predicted 3D model. Furthermore, to ensure that the order of the primitives predicted by the primitive prediction network conforms to human cognitive processes, the primitive prediction network is trained under supervised supervision according to the order of the first primitive in the first primitive sequence and the order of the second primitive in the second primitive sequence.

[0163] Optionally, when obtaining the first loss value, at least one of the following conditions may be included:

[0164] 1. Align the first primitive sequence and the second primitive sequence. According to the one-to-one correspondence between the first primitive and the second primitive after alignment, calculate the cross-entropy loss between the i-th first primitive and the i-th second primitive to obtain the first loss value between each first primitive-second primitive pair.

[0165] 2. During the prediction of the 3D model, at least two second primitives are generated iteratively. After a primitive is predicted in each prediction stage, the cross-entropy loss is calculated based on the first primitive in the reference 3D model corresponding to the prediction stage and the currently predicted second primitive, to obtain the first loss value corresponding to the current prediction stage. For example, in the i-th prediction stage, after fusing the sample prompt information and the primitive parameters of the second primitives predicted in the previous i-1 prediction stages, a fused feature representation is obtained. After predicting the fused feature representation through the primitive prediction network to obtain the i-th second primitive, the cross-entropy loss is calculated between the i-th second primitive and the i-th first primitive in the first primitive sequence to obtain the first loss value; or, in the i-th prediction stage, after fusing the sample prompt information and the primitive parameters of the previous i-1 first primitives in the first primitive sequence, a fused feature representation is obtained. After predicting the fused feature representation through the primitive prediction network to obtain the i-th second primitive, the cross-entropy loss is calculated between the i-th second primitive and the i-th first primitive in the first primitive sequence to obtain the first loss value. Optionally, after obtaining the first loss value in the i-th prediction stage, the primitive prediction network can be trained directly based on the first loss value, or the total loss value can be calculated after the prediction is completed to train the primitive prediction network.

[0166] In some embodiments, if the i-th second primitive does not exist, that is, the number of second primitives in the sequence of second primitives is less than i, then the i-th second primitive is treated as 0 and the cross-entropy loss is calculated with the i-th first primitive.

[0167] The calculation process for cross-entropy loss is illustrated in Formula 1 below:

[0168] Formula 1:

[0169] Where n1 is the number of first primitives in the reference 3D model, and i represents the i-th first primitive.

[0170] The cross-entropy loss is obtained as the first loss value based on the difference between the first and second primitives, which improves the efficiency of loss value calculation and the training efficiency of the primitive prediction network.

[0171] 1.2. Based on the termination positions of the first primitive sequence and the second primitive sequence, obtain the second loss value; train the primitive prediction network using the second loss value.

[0172] The termination position is the position in the prediction stage that the primitive prediction network dynamically determines when it ends the prediction process.

[0173] Optionally, when obtaining the second loss value, at least one of the following conditions may be included:

[0174] 1. Align the first primitive sequence and the second primitive sequence, and calculate the binary cross-entropy loss according to the termination positions of the first primitive sequence and the second primitive sequence after alignment, to obtain the second loss value.

[0175] 2. In the prediction process of the 3D model, at least two second primitives are generated sequentially through iteration. After the prediction of the terminator in each prediction stage, the binary cross-entropy loss is calculated based on the prediction result of the terminator in the prediction stage and the position of the terminator in the first primitive sequence to obtain the second loss value corresponding to the current prediction stage. For example, in the i-th prediction stage, after fusing the sample prompt information and the primitive parameters of the second primitives predicted in the previous i-1 prediction stages, a fused feature representation is obtained. The fused feature representation is then used by the primitive prediction network to predict the termination symbol prediction result (i.e., whether a termination symbol is generated) for the i-th prediction stage. Finally, a binary cross-entropy loss is calculated based on the presence of the termination symbol at the i-th position in the first primitive sequence to obtain the second loss value. Alternatively, in the i-th prediction stage, after fusing the sample prompt information and the primitive parameters of the previous i-1 first primitives in the first primitive sequence, a fused feature representation is obtained. The fused feature representation is then used by the primitive prediction network to predict the termination symbol prediction result (i.e., whether a termination symbol is generated) for the i-th prediction stage. Finally, a binary cross-entropy loss is calculated based on the presence of the termination symbol at the i-th position in the first primitive sequence to obtain the second loss value. Optionally, after obtaining the second loss value for the i-th prediction stage, the primitive prediction network can be trained directly based on the second loss value, or the total loss value can be calculated after the prediction is completed to train the primitive prediction network.

[0176] In some embodiments, a second loss value is obtained by calculating the binary cross-entropy loss at the termination positions of the first primitive sequence corresponding to the reference 3D model and the second primitive sequence corresponding to the predicted 3D model. The calculation process of the binary cross-entropy loss is illustrated in Formula 2 below:

[0177] Formula 2:

[0178] Where a1 represents the probability of outputting a terminator in the first primitive sequence, softmax(f i1 )1 represents the probability of outputting a terminator in the second primitive sequence during the i-th prediction stage, softmax(f i1 )2 represents the probability that no terminator is output in the second primitive sequence during the i-th prediction stage.

[0179] By calculating the binary cross-entropy loss by referencing the first primitive sequence of the 3D model and the terminator in the second primitive sequence of the predicted 3D model, the second loss value is determined, which improves the accuracy of the primitive prediction network in dynamically determining the length of the primitive sequence.

[0180] 2. Obtain the first point cloud data corresponding to the reference 3D model. The first point cloud data is used to represent the reference 3D model in point cloud data format. Convert the predicted 3D model into second point cloud data. The second point cloud data is used to represent the predicted 3D model in point cloud data format. Based on the distance between the first point cloud in the first point cloud data and the second point cloud in the second point cloud data, obtain the third loss value. Train the primitive prediction network using the third loss value.

[0181] Specifically, when acquiring the first point cloud data corresponding to the reference 3D model, model point sampling is performed on the reference 3D model to obtain the first point cloud data; or, the point cloud data corresponding to the sample prompt information is acquired as the first point cloud data.

[0182] When acquiring the second point cloud data corresponding to the predicted 3D model, model point sampling is performed on the predicted 3D model to obtain the second point cloud data.

[0183] The calculation process for distance loss is shown in Formula 3 below:

[0184] Formula 3: L CD =CD(pc) pred pc gt )

[0185] Among them, pc pred This represents the point cloud in the second point cloud data corresponding to the predicted 3D model; pc gt This represents the point cloud in the first point cloud data corresponding to the reference 3D model, and CD represents distance calculation.

[0186] The distance loss value is determined by the distance difference between the point cloud of the predicted 3D model and the point cloud of the reference 3D model. This expresses the error of the predicted 3D model obtained from the perspective of the overall model structure, and further improves the training accuracy of the primitive prediction network.

[0187] The method provided in this application integrates point cloud distance loss to enhance geometric constraints, thereby improving the accuracy of the generated primitives in conforming to the geometric features of the original shape while maintaining cognitive rationality, and thus improving the primitive prediction accuracy.

[0188] In some embodiments, after obtaining the first loss value, the second loss value, and the third loss value, the first loss value, the second loss value, and the third loss value are fused to obtain a total loss value, and the primitive prediction network is trained based on the total loss value.

[0189] To illustrate, the first loss value, the second loss value, and the third loss value are weighted and combined to obtain the total loss value.

[0190] As an illustration, the combination loss is shown in Formula 4 below:

[0191] Formula 4: L = αL c +βL EOS +γL CD

[0192] Here, α, β, and γ are pre-set weighting coefficients used to balance the three loss values.

[0193] The trained primitive prediction network is obtained by training the encoder and decoder used in the above-mentioned primitive prediction network using loss training.

[0194] It is worth noting that the above training scheme is only an illustrative example. Other training mechanisms can also be introduced in the embodiments of this application, such as: introducing a course learning strategy to conduct training in stages from easy to difficult (e.g., progressing from single primitive training to multi-primitive training, or from training of single-class primitives to training of multi-class primitives); or, introducing other physical feasibility constraints, such as connectivity detection, to improve the accuracy of physical connections between primitives.

[0195] In summary, the method provided in this application, by setting an equivalent transformation set corresponding to each primitive type, expresses the symmetry of the primitives in the form of a set, that is, expresses multiple transformation methods that produce the same result for the primitives. When predicting the primitives that constitute the three-dimensional model, after obtaining the primitive morphological parameters, the primitive morphological parameters are updated through multiple transformation methods in the equivalent transformation set. By combining the equivalent transformation set with the primitive morphological parameters, the expression method of the primitive morphological parameters is determined and unified according to the combination result, thereby unifying the expression meaning of the primitive morphological parameter encoding result, improving the encoding accuracy, thereby correspondingly improving the decoding accuracy, improving the accuracy of primitive prediction and the accuracy of three-dimensional model generation.

[0196] The method provided in this application improves semantic rationality and category generalization by constructing a large-scale manually labeled dataset across categories and learning human primitive abstraction patterns. This enhances the training accuracy of the primitive prediction network and the accuracy of the primitive prediction network in predicting primitives to construct 3D models.

[0197] In an optional embodiment, in the above... Figure 3 In the application embodiments shown, or in the above Figure 7 In the training example shown, after the primitive parameters of the primitives are predicted, the primitive parameters are updated according to the ambiguity elimination principle. Figure 8 This is a flowchart illustrating a method for generating a 3D model according to an embodiment of this application. The method is implemented using a computer device (which can be configured as follows). Figure 2 The method, executed by the terminal 220 or server 240 shown, may include steps 810 to 840.

[0198] Step 810: Obtain model prompt information.

[0199] Model hints are used to indicate the requirement to generate a 3D model from at least one primitive, which refers to a 3D geometric element.

[0200] In some embodiments, model hints are used to express the need to generate a 3D model. These model hints can be implemented as model hints during the application phase or as sample hints during the training phase.

[0201] Step 820: Predict the model prompt information and obtain the primitive parameters of at least one primitive.

[0202] Primitive parameters include primitive type parameters and primitive shape parameters. Primitive type parameters indicate the geometry of the primitive, while primitive shape parameters indicate how the basic shape of the primitive is transformed.

[0203] In some embodiments, when the 3D model includes at least two primitives, the model prediction prompts information to sequentially obtain at least two primitives, wherein the geometry of the at least two primitives is the same or different. That is, the 3D geometric elements corresponding to at least one primitive are the same or different.

[0204] In some embodiments, when providing information to the prediction model, at least two primitives are obtained from a predefined set of primitives. The set of primitives includes at least two types of primitives, and the at least two types of primitives have different geometric shapes. Each primitive corresponds to a basic form; for example, the basic form of the cube primitive is a cube with length, width, and height of 3, 2, and 4, respectively.

[0205] When the prediction model provides information, select the appropriate primitive from at least two primitives provided by the primitive set, perform a basic morphological transformation on the primitive to obtain the transformed primitive, and apply the transformed primitive to the construction of the 3D model.

[0206] It is worth noting that, in the above embodiments, the example of obtaining at least two primitives from a predefined primitive set is used for illustration. In some embodiments, primitives may also be obtained from a public primitive database or through other means, and this application does not limit this.

[0207] Optionally, when the prediction model prompts for at least two primitive parameters, the process of obtaining the at least two primitives includes at least one of the following processes:

[0208] First, predict the model prompt information to obtain the primitive parameters corresponding to the first primitive. Then, fuse the model prompt information and the primitive parameters corresponding to the first primitive to obtain a fused feature representation. Predict the fused feature representation to obtain the primitive parameters corresponding to the second primitive. When predicting the primitive parameters of the third primitive, fuse the model prompt information, the primitive parameters corresponding to the first primitive, and the primitive parameters corresponding to the second primitive to obtain a fused feature representation. Predict the fused feature representation to obtain the primitive parameters corresponding to the third primitive, and so on until the prediction result is obtained. That is, fuse the model prompt information and the primitive parameters corresponding to the first i-1 primitives to obtain a fused feature representation. Predict the fused feature representation to obtain the primitive parameters corresponding to the i-th primitive, where i > 1 and i is an integer.

[0209] That is, the prediction process for the primitive parameters of at least two primitives is iterative, predicting the primitive parameters of the next primitive based on the primitive parameters of the previously predicted primitives.

[0210] By iteratively determining the primitive parameters of at least two primitives, the process of sequentially determining primitives to build a three-dimensional model in human cognition is simulated. The primitive parameters of the previously determined primitives are used to constrain the primitive parameters of the subsequently determined primitives, thereby improving the accuracy of primitive determination and the ability of primitives to express the components of the model.

[0211] Second, the prediction model provides information and outputs the primitive parameters corresponding to at least two primitives in parallel.

[0212] That is, the prediction process for the primitive parameters of at least two primitives is carried out in parallel.

[0213] Third, predict the model prompts to obtain the primitive parameters corresponding to the first primitive. Then, fuse the model prompts and the primitive parameters corresponding to the first primitive to obtain a fused feature representation. Predict the fused feature representation to obtain the primitive parameters corresponding to the second primitive. When predicting the primitive parameters of the third primitive, fuse the model prompts and the primitive parameters corresponding to the second primitive to obtain a fused feature representation, and predict the fused feature representation to obtain the primitive parameters corresponding to the third primitive, and so on until the prediction result is obtained. That is, fuse the model prompts and the primitive parameters corresponding to the (i-1)th primitive to obtain a fused feature representation, predict the fused feature representation to obtain the primitive parameters corresponding to the ith primitive, where i > 1 and i is an integer.

[0214] That is, the prediction process for the primitive parameters of at least two primitives is iterative. Based on the primitive parameters of the primitives obtained in the most recent prediction in the historical prediction stage, the primitive parameters of the current primitive are predicted.

[0215] The generation of the next primitive is determined by the generation of the previous primitive, and the generation of the next primitive is constrained by the generation of the previous primitive. This improves the coordination and matching between two adjacent primitives and improves the accuracy of the generated 3D model.

[0216] It is worth noting that the above primitive prediction process is merely an illustrative example, and the specific implementation of the primitive prediction process is not limited in the embodiments of this application. In the embodiments of this application, the example of obtaining the fused feature representation by fusing the model prompt information and the primitive parameters of the first i-1 primitives, and then predicting the primitive parameters of the i-th primitive, is used for illustration.

[0217] In some embodiments, the number of primitives constituting the 3D model is dynamically determined. In some embodiments, the model hint information and the primitive parameters of the first i-1 primitives are fused to obtain a fused feature representation, and the fused feature representation is used to predict whether the i-th primitive belongs to the last primitive constituting the 3D model. That is, whether a terminator is generated after the i-th primitive is generated. In response to the generation of a terminator after the generation of the i-th primitive, the 3D model is constituted based on the i primitives.

[0218] In this embodiment, the following case is used as an example for illustration:

[0219] In the application phase, the prediction model prompts information to obtain the primitive parameters of the first primitive; the model prompts information and the primitive parameters of the first primitive are fused to obtain the fused feature representation and predict the primitive parameters of the second primitive; the model prompts information and the primitive parameters of the first two primitives are fused to obtain the fused feature representation and predict the primitive parameters of the third primitive, and so on, until the prediction ends.

[0220] During the training phase, the primitive parameters of the first second primitive are obtained by predicting the sample prompt information; the sample prompt information and the primitive parameters of the first second primitive are fused to obtain the fused feature representation and predict the primitive parameters of the second second primitive; the sample prompt information and the primitive parameters of the first two second primitives are fused to obtain the fused feature representation and predict the primitive parameters of the third second primitive, and so on, until the prediction ends.

[0221] Alternatively, during the training phase, for the primitive parameters of each second primitive in the second primitive sequence of the predicted 3D model, the primitive parameters output by the primitive prediction network are used; for the primitive parameters of each first primitive in the first primitive sequence of the reference 3D model, the updated primitive parameters are obtained through equivalent transformation set transformation. The primitive parameters of the second primitive are then trained under supervision using the updated primitive parameters of the first primitive, so that the primitive parameters output by the primitive prediction network meet the ambiguity requirements.

[0222] Step 830: Obtain the set of equivalent transformations corresponding to the primitive type parameters.

[0223] The equivalent transformation set includes at least one candidate transformation scheme, and at least one candidate transformation scheme has the same transformation result on the basic form of the primitive.

[0224] The equivalent transformation set is used to update the primitive form parameters. Specifically, after predicting the primitive parameters of the i-th primitive in the i-th prediction stage, the equivalent transformation set corresponding to the primitive type parameter of the i-th primitive is obtained. This equivalent transformation set is then used to update the primitive form parameters and is used to predict the primitive parameters of the (i+1)-th primitive. Similarly, after predicting the primitive parameters of the first primitive in the first prediction stage, the equivalent transformation set corresponding to the primitive type parameter of the first primitive is obtained. This equivalent transformation set is then used to update the primitive form parameters of the first primitive and is used to predict the primitive parameters of the second primitive.

[0225] The set of equivalent transformations is a set of transformation schemes constructed based on the symmetry of primitives. Due to the symmetry of primitives (such as cube primitives and cylinder primitives), different combinations of scaling and rotation parameters will produce the same transformation result. (Illustrative example) Figure 9 This is a schematic diagram of the transformation result of a cube primitive provided in an exemplary embodiment of this application. For example... Figure 9 As shown, for a cube element 900 with a width of 2, a length of 3, and a height of 4, in transformation scheme 1, after rotating 90° clockwise around the width axis, stretching the length axis and compressing the height axis, cube 910 is obtained; then, rotating 30° counterclockwise around the width axis, cube 911 is obtained. In transformation scheme 2, after rotating 180° clockwise around the width axis, cube 921 is obtained; and in transformation scheme 3, rotating 30° counterclockwise around the width axis, cube 931 is obtained. Figure 9 As shown, cubes 911, 921 and 931 are the same result obtained by performing different transformations on cube primitive 900.

[0226] Therefore, for each primitive type parameter, calculate the set of symmetry axes of the primitive and the symmetry order corresponding to each axis; define the set of equivalent transformations, which includes the union of equivalent rotation operations on the primitive. The set of equivalent transformations includes at least two rotation operations, where each rotation axis corresponds to at least one rotation operation, the j-th rotation axis corresponds to m rotation operations, and the k-th rotation operation corresponding to the j-th rotation axis refers to a rotation of 2πk / m around the j-th rotation axis.

[0227] Figure 10 This is a schematic diagram of the transformation of a hexagonal prism primitive provided in an exemplary embodiment of this application, as shown below. Figure 10 As shown, the base of the hexagonal prism primitive 1000 is a regular hexagon. Rotating the hexagonal prism primitive 1000 around its central axis 1010 by 60°, 120°, 180°, 240°, 300°, and 360° yields the same result. Alternatively, this transformation can be expressed as rotating the hexagonal prism primitive 1000 around its central axis 1010 clockwise by 0°, 60°, 120°, and 180°, and counterclockwise by 60° and 120° yielding the same result. Since there are 5 equivalent transformations of the hexagonal prism primitive within a 360° range around its central axis, the symmetry order of the central axis of the hexagonal prism primitive is 6.

[0228] In some embodiments, the rotation axis is a pre-defined axis for the primitive, or the rotation axis is an axis automatically identified after symmetry analysis of the primitive.

[0229] Optionally, for the case of an infinite axis of symmetry, the rotation axis is determined at preset angle intervals using a predefined method, or a specified axis of symmetry is determined within the infinite axis of symmetry using a predefined method. Figure 10 Taking the hexagonal prism primitive 1000 as an example, any straight line passing through the center of the regular hexagon and parallel to the base in the waistline of the hexagonal prism primitive 1000 can be realized as the rotation axis 1020 of the hexagonal prism primitive 1000. Rotating 180° around the rotation axis can obtain the same transformation result as the hexagonal prism primitive 1000. Four rotation axes of the hexagonal prism primitive 1000 are predefined, including the central axis 1010, and the rotation axis 1020 passing through the centers of two opposite prism faces and passing through the center of the hexagonal prism primitive 1000 and parallel to the base (wherein, since the hexagonal prism primitive 1000 has three pairs of opposite prism faces, the rotation axis 1020 includes axes corresponding to the three pairs of opposite prism faces respectively).

[0230] Taking a sphere as an example, in a sphere, any straight line passing through the center of the sphere can serve as the axis of rotation, and rotation at any angle can coincide with the basic shape of the sphere. Therefore, for a sphere, the axis of rotation and the corresponding order of symmetry are pre-defined.

[0231] Schematic, given a predefined set of primitives P = {P1, P2, ..., P...} N } where N refers to a set of primitives containing N geometric shapes, such as: N=3, which represents a set of primitives including cube, ellipsoid, and cylinder primitives; N=5, which represents a set of primitives including cube, ellipsoid, cylinder, hexagonal prism, and triangular pyramid primitives. The value of N is determined based on the predefined primitive type.

[0232] For each primitive type P ci Calculate its set of symmetry axes V and each axis V j The symmetric order m j Define the rotation-symmetric set R as the union of equivalent rotation operations. Here, the rotation-symmetric set R is the set of the aforementioned equivalent transformations. Where k represents the k-th symmetry order, m j Let v denote the symmetry order corresponding to the j-th rotation axis. j Let represent the j-th rotation axis, Rot(v, θ) represent the rotation angle θ around the rotation axis v, and n represent the number of rotation axes corresponding to the primitive.

[0233] Step 840: Update the primitive morphological parameters based on at least one candidate transformation scheme, and generate a three-dimensional model based on the primitives corresponding to the updated primitive morphological parameters.

[0234] In some embodiments, the primitive morphological parameters are combined with at least one candidate transformation scheme to obtain at least one morphological transformation scheme. The transformation result of the at least one morphological transformation scheme on the basic morphology of the primitive matches the primitive morphological parameters. From the at least one morphological transformation scheme, a target morphological transformation scheme that meets preset requirements is obtained to obtain the updated primitive morphological parameters.

[0235] By combining primitive morphological parameters with at least one candidate transformation scheme and using at least one candidate transformation scheme as a reference, multiple different expressions of primitive morphological parameters are determined. Then, one expression is selected from the different expressions to update the primitive morphological parameters, which improves the consistency of primitive morphological parameter expression and avoids the ambiguity that may exist when expressing primitive morphological parameters, thus preventing inaccurate encoding results in the training or application process.

[0236] In some embodiments, when combining the primitive shape parameters with at least one candidate transformation scheme to obtain at least one shape transformation scheme, the at least one candidate transformation scheme includes at least one rotation scheme. The rotation scheme includes a rotation axis and a rotation angle; that is, the rotation scheme refers to rotating the primitive about the rotation axis by a rotation angle. Then, the first rotation scheme from the primitive shape parameters is obtained. The first rotation scheme includes a rotation axis and a rotation angle. The rotation angle in the first rotation scheme is added to the rotation angle in the at least one candidate transformation scheme according to the rotation axis to obtain at least one shape transformation scheme.

[0237] Since ambiguity in the expression of primitive shape parameters often arises from the angle of the primitive rotation parameter, such as the fact that a cube primitive rotates 180° and remains the same, the primitive shape parameter is combined with the rotation angle of the candidate transformation scheme to obtain the shape transformation scheme, thereby eliminating the ambiguity in the expression of angle.

[0238] That is, taking the rotation axis as a reference, the rotation angle of the same rotation axis in the first rotation scheme is added to the rotation angle in the candidate transformation scheme to obtain the shape transformation scheme. If the rotation axis does not rotate in the first rotation scheme or the candidate transformation scheme, then 0° is used as the rotation angle to be added.

[0239] Optionally, the rotation angle in the first rotation scheme is added to the rotation angle in at least one candidate transformation scheme according to the rotation axis to obtain the sum of rotation angles corresponding to at least one rotation axis; the sum of rotation angles is corrected to a preset angle range to obtain at least one shape transformation scheme.

[0240] In some embodiments, if the clockwise rotation angle α exceeds 180° but does not reach 360°, it is calculated as a counterclockwise rotation of 360°-α.

[0241] In some embodiments, if the rotation angle x calculated in the shape transformation scheme exceeds 360°, the rotation angle in the shape transformation scheme is obtained according to x-360°.

[0242] Schematic, in the first rotation scheme, the cube primitive is rotated 210° clockwise around its wide side axis and 10° clockwise around its height axis. The long side axis, wide side axis, and height axis of the cube primitive all pass through the geometric center of the cube primitive.

[0243] The equivalent transformation set of the cube primitive includes the following candidate transformation schemes:

[0244] ① Rotate the long axis 180° clockwise;

[0245] ② Rotate the wide-side axis 180° clockwise;

[0246] ③ Rotate the high-side axis 180° clockwise.

[0247] Then, by combining the first rotation scheme with candidate transformation scheme ①, the first shape transformation scheme is obtained as follows: the long side axis is rotated 180° clockwise, the wide side axis is rotated 210° clockwise, and the high side axis is rotated 10° clockwise.

[0248] Combining the first rotation scheme with candidate transformation scheme ②, we obtain the second shape transformation scheme as follows: the long side axis rotates 0° clockwise, the wide side axis rotates 390° clockwise, and the high side axis rotates 10° clockwise; since the wide side axis rotates more than 360° clockwise, we calculate that the wide side axis rotates 30° clockwise.

[0249] Combining the first rotation scheme with candidate transformation scheme ③, we obtain the third morphological transformation scheme as follows: the long side axis rotates 0° clockwise, the wide side axis rotates 210° clockwise, and the high side axis rotates 190°. Since the wide side axis and the high side axis rotate more than 180°, we calculate that the wide side axis rotates 150° counterclockwise and the high side axis rotates 170° counterclockwise.

[0250] It is worth noting that the above-described rotational combination process is merely an illustrative example, and the embodiments of this application do not limit the specific combination method.

[0251] In some embodiments, the rotation angles of the primitives corresponding to at least one morphological transformation scheme are obtained. The primitive rotation angle refers to the rotation angle of the basic shape of the primitive in the morphological transformation scheme. A target morphological transformation scheme whose primitive rotation angle meets preset requirements is obtained, resulting in updated primitive morphological parameters. Optionally, a target morphological transformation scheme with the smallest primitive rotation angle is obtained, resulting in updated primitive morphological parameters. Here, the primitive rotation angle refers to the sum of the rotation angles corresponding to each rotation axis of the primitive; or, the primitive rotation angle refers to the average rotation angle corresponding to each rotation axis of the primitive.

[0252] In some embodiments, the rotation L1 norm corresponding to at least one morphological transformation scheme is obtained, where the norm is a function with a length concept. Optionally, a rotation matrix is ​​obtained through the rotation axis and rotation angle, and the rotation L1 norm is calculated based on the rotation matrix. The morphological transformation scheme with the smallest rotation L1 norm is selected from at least one morphological transformation scheme as the target morphological transformation scheme, and the transformation parameters in the target morphological transformation scheme are obtained to obtain the updated primitive morphological parameters.

[0253] Indicatively, the above method of rotating the cube primitive 210° clockwise around the wide side axis and 10° clockwise around the high side axis, combined with three candidate transformation schemes, yields the result with the smallest rotation angle, i.e., rotating the long side axis 0° clockwise, the wide side axis 30° clockwise, and the high side axis 10° clockwise.

[0254] In some embodiments, for a primitive pi, the primitive type P ci ∈P and its rigid body transformation τ(·) in three-dimensional space are jointly defined, that is, p i =τ(P ci ;s i r i , t i ), P ci Let s represent the primitive type of the i-th primitive, that is, the geometry of the i-th primitive. i r represents the scaling factor of the i-th primitive along at least one axis. i t represents the rotation angle of the i-th primitive along at least one rotation axis. i This represents the coordinate position of the center of the i-th primitive in three-dimensional space. A spatial coordinate system is constructed in three-dimensional space with a preset point as the origin, and t... i This represents the coordinate position of the center of the i-th primitive in the spatial coordinate system.

[0255] The primitive parameters of the i-th primitive are obtained by prediction, and the above p is also obtained by prediction. i , where (s i r i , t i ) represents the predicted original transformation, which is then compared with P. ci At least one candidate transformation scheme combination from the equivalent transformation set R of the corresponding primitive type. That is, based on at least one candidate transformation scheme, the original transformation is performed to obtain at least one morphological transformation scheme.

[0256] In at least one morphological transformation scheme, select the one that minimizes the L1 norm of the rotation as the new transformation (s). i ', r i ',t i '),in,

[0257] After updating the original transformation using the equivalent transformation set, we obtain the updated i-th primitive p. i =t(P ci ;s i ', r i ',t i ').

[0258] In some embodiments, since the process of generating primitives is iterative, taking the fusion model hint information and the primitive parameters of the first i-1 primitives to obtain the fusion feature representation and predict the primitive parameters of the i-th primitive as an example, that is, after obtaining the primitive parameters of the i-1 primitive, the equivalent transformation set is first obtained through the primitive type parameter of the i-1 primitive, and the primitive parameters of the i-1 primitive are updated through at least one candidate transformation scheme in the equivalent transformation set to obtain the updated parameters. The fusion feature representation is obtained from the updated primitive parameters of the first i-1 primitives in the fusion model hint information set, and the primitive parameters of the i-th primitive are predicted by predicting the fusion feature representation.

[0259] In some embodiments, an equivalent transformation set corresponding to the first primitive is obtained, the equivalent transformation set including at least one candidate transformation scheme. The primitive morphological parameters of the first primitive are updated based on at least one candidate transformation scheme to obtain the updated primitive morphological parameters of the first primitive. The updated primitive morphological parameters of the first primitive are applied to the subsequent prediction stage to predict the primitive parameters of subsequent primitives.

[0260] To illustrate, in the second prediction stage, the primitive parameters corresponding to the primitives predicted in the first prediction stage are obtained, including the primitive type parameters and the updated primitive shape parameters.

[0261] To illustrate, in the third prediction stage, the primitive parameters corresponding to the primitives predicted in the first two prediction stages are obtained, including the primitive type parameters and the updated primitive morphology parameters, and so on.

[0262] Taking the third prediction stage as an example, the primitives predicted in the first two prediction stages are obtained, including the primitive p1 = τ(P) predicted in the first prediction stage. c1 ;s1,r1,t1) and the primitive p2=τ(P) predicted in the second prediction stage c2 ;s2,r2,t2), where p1 and p2 include the primitive parameters obtained by updating the primitive morphological parameters through the equivalent transformation set.

[0263] It is worth noting that this application uses the method of designing an equivalent transformation set to eliminate ambiguity in parameter expression as an example for illustration. This method of eliminating ambiguity in parameter expression can also be used to provide other mathematical representations that guarantee parameter uniqueness. For example, calculating the principal axis of primitive principal component analysis aligns the parameter expression with the coordinate system, thereby eliminating rotational symmetry ambiguity.

[0264] In summary, the method provided in this application, by setting an equivalent transformation set corresponding to each primitive type, expresses the symmetry of the primitives in the form of a set, that is, expresses multiple transformation methods that produce the same result for the primitives. When predicting the primitives that constitute the three-dimensional model, after obtaining the primitive morphological parameters, the primitive morphological parameters are updated through multiple transformation methods in the equivalent transformation set. By combining the equivalent transformation set with the primitive morphological parameters, the expression method of the primitive morphological parameters is determined and unified according to the combination result, thereby unifying the expression meaning of the primitive morphological parameter encoding result, improving the encoding accuracy, thereby correspondingly improving the decoding accuracy, improving the accuracy of primitive prediction and the accuracy of three-dimensional model generation.

[0265] The method for generating three-dimensional models provided in this application can be applied to game development and user-generated content (UGC) scenarios, industrial design, extended reality (XR) fields, etc., and is not limited thereto.

[0266] In some embodiments, taking game development and UGC scenarios as an example, the process is as follows: Model prompt information is obtained, such as text prompts, image prompts, point cloud prompts, etc., input by the player. This model prompt information indicates the requirement to generate a 3D scene model in the virtual scene using at least one primitive, where a primitive refers to a 3D geometric element. In the i-th prediction stage, primitive parameters corresponding to the primitives predicted in the previous i-1 prediction stages are obtained, where i > 1 and i is an integer. These primitive parameters indicate the geometric shape of the primitives. The model prompt information and the primitive parameters corresponding to the primitives predicted in the previous i-1 prediction stages are fused to obtain a fused feature representation. The fused feature representation is then predicted to obtain the primitive parameters corresponding to the primitives predicted in the i-th prediction stage. This process continues until the prediction stage ends, resulting in the 3D scene model in the game's virtual scene.

[0267] In some embodiments, a prompt information input area is displayed, which is used to indicate the generation of a three-dimensional scene model in the virtual scene; input operation on the model prompt information is received in the prompt information input area; then the fusion feature representation is predicted to obtain the primitive parameters corresponding to the primitives predicted in the i-th prediction stage, until the prediction stage ends, and a three-dimensional scene model is generated and displayed in the virtual scene, which is implemented as scene elements in the virtual scene.

[0268] The three-dimensional scene model includes three-dimensional character models, three-dimensional vehicle models, three-dimensional pet models, three-dimensional house models, three-dimensional plant models, etc., and this application embodiment does not limit this.

[0269] The information feature representation of the model prompt information is obtained, and the information feature representation is used to express the model prompt information in the form of feature vectors.

[0270] Obtain the parameter feature sequence corresponding to the primitive parameters predicted in the first i-1 prediction stages;

[0271] By fusing the information feature representation and the parameter feature sequence, a fused feature representation is obtained.

[0272] In an optional embodiment, the primitive parameters predicted in the first i-1 prediction stages are encoded to obtain the parameter feature representations corresponding to the first i-1 prediction stages respectively; the parameter feature representations corresponding to the first i-1 prediction stages are concatenated in sequence to obtain the parameter feature sequence.

[0273] In an optional embodiment, the predicted fusion feature representation is used to obtain the prediction result of the terminator in the i-th prediction stage, which indicates the end of the prediction process in the i-th prediction stage; in response to the generation of the terminator corresponding to the i-th prediction stage, a three-dimensional scene model is generated based on the primitives determined in the i-th prediction stage.

[0274] In an optional embodiment, the primitive parameters corresponding to the primitives predicted in the i-th prediction stage are obtained by decoding the fused feature representation through the first decoder; the terminator prediction result of the i-th prediction stage is obtained by decoding the fused feature representation through the second decoder.

[0275] In an optional embodiment, the primitive parameters include primitive type parameters and primitive shape parameters, wherein the primitive type parameters are used to indicate the geometry of the primitive, and the primitive shape parameters are used to indicate the transformation method of the basic shape of the primitive;

[0276] Obtain the equivalent transformation set corresponding to the primitive type parameter, wherein the equivalent transformation set includes at least one candidate transformation scheme, and the at least one candidate transformation scheme has the same transformation result on the basic form of the primitive;

[0277] The primitive morphological parameters are updated based on the at least one candidate transformation scheme.

[0278] In an optional embodiment, the primitive morphological parameters are combined with the at least one candidate transformation scheme to obtain at least one morphological transformation scheme, wherein the transformation result of the at least one morphological transformation scheme on the basic morphology of the primitive matches the primitive morphological parameters.

[0279] The target morphological transformation scheme that meets the preset requirements is obtained from the at least one morphological transformation scheme, and the updated primitive morphological parameters are obtained.

[0280] In an optional embodiment, the primitive rotation angle corresponding to the at least one morphological transformation scheme is obtained, wherein the primitive rotation angle refers to the rotation angle of the basic form of the primitive in the morphological transformation scheme.

[0281] Obtain the target shape transformation scheme whose primitive rotation angle meets the preset requirements, and obtain the updated primitive shape parameters.

[0282] In an optional embodiment, the at least one candidate transformation scheme includes at least one rotation scheme, the rotation scheme including a rotation axis and a rotation angle;

[0283] Obtain a first rotation scheme from the primitive shape parameters, wherein the first rotation scheme includes a rotation axis and a rotation angle;

[0284] The rotation angle in the first rotation scheme is added to the rotation angle in the at least one candidate transformation scheme according to the rotation axis to obtain the at least one shape transformation scheme.

[0285] In an optional embodiment, the rotation angle in the first rotation scheme is added to the rotation angle in the at least one candidate transformation scheme according to the rotation axis to obtain the rotation angle sum corresponding to at least one rotation axis;

[0286] By adjusting the rotation angle to a preset angle range, at least one shape transformation scheme is obtained.

[0287] In an optional embodiment, sample prompt information and a reference 3D scene model corresponding to the sample prompt information are obtained. The reference 3D scene model is a 3D scene model constructed by at least one first primitive. The sample prompt information is predicted by a primitive prediction network to obtain a predicted 3D scene model. The predicted 3D scene model is a 3D scene model constructed by at least one second primitive. The primitive prediction network is trained based on the difference between the predicted 3D scene model and the reference 3D scene model.

[0288] In an optional embodiment, a first primitive sequence consisting of at least one first primitive in a reference 3D scene model is obtained, the first primitive sequence being a sequence obtained by arranging the at least one first primitive in the generation order; a second primitive sequence consisting of at least one second primitive in a predicted 3D scene model is obtained, the second primitive sequence being a sequence obtained by arranging the second primitive in the prediction order; and a primitive prediction network is trained based on the difference between the first primitive sequence and the second primitive sequence.

[0289] In an optional embodiment, a first loss value is obtained based on the primitive parameters of the i-th first primitive in the first primitive sequence and the primitive parameters of the i-th second primitive in the second primitive sequence, where i is a positive integer; and the primitive prediction network is trained using the first loss value.

[0290] In an optional embodiment, a second loss value is obtained based on the termination position of the first primitive sequence and the termination position of the second primitive sequence; the primitive prediction network is trained using the second loss value.

[0291] In an optional embodiment, first point cloud data corresponding to a reference 3D scene model is obtained, and the first point cloud data is used to express the reference 3D scene model in a point cloud data format; the predicted 3D scene model is converted into second point cloud data, and the second point cloud data is used to express the predicted 3D scene model in a point cloud data format; a third loss value is obtained based on the distance between the first point cloud in the first point cloud data and the second point cloud in the second point cloud data; and a primitive prediction network is trained using the third loss value.

[0292] In summary, the method provided in this application, by iteratively determining the primitive parameters of at least two primitives, simulates the process of gradually building a 3D scene model by sequentially determining primitives in human cognition. By constraining the primitive parameters of subsequently determined primitives with the primitive parameters of primitives already predicted in the preceding prediction stage, the method improves the coordination between primitives, avoids the problems of misalignment and weak compatibility among multiple primitives caused by determining multiple primitives in parallel, improves the accuracy of primitive determination, and enhances the expressive power of primitives for model components.

[0293] In some embodiments, taking the field of industrial design as an example, the model prompt information is obtained, such as point cloud data of complex parts input by engineering designers, such as point cloud data obtained by capturing depth information of complex parts through a depth camera and converting it. The model prompt information is used to indicate the requirements for generating a three-dimensional part model in industrial design through at least one primitive, where a primitive refers to a three-dimensional geometric element.

[0294] In the i-th prediction stage, the primitive parameters corresponding to the primitives predicted in the previous i-1 prediction stages are obtained, where i > 1 and i is an integer. The primitive parameters are used to indicate the geometric shape of the primitives. The model prompt information and the primitive parameters corresponding to the primitives predicted in the previous i-1 prediction stages are fused to obtain the fused feature representation. The fused feature representation is predicted to obtain the primitive parameters corresponding to the primitives predicted in the i-th prediction stage. This process continues until the prediction stage ends to obtain the three-dimensional part model in industrial design.

[0295] In some embodiments, a prompt information input area is displayed, which is used to instruct the generation of a three-dimensional part model in industrial design; input operation for model prompt information is received in the prompt information input area; then the fusion feature representation is predicted to obtain the primitive parameters corresponding to the primitives predicted in the i-th prediction stage, until the prediction stage ends, and a three-dimensional part model corresponding to the industrial part is generated and displayed.

[0296] The three-dimensional part models include three-dimensional models of vehicle parts, three-dimensional models of switch parts, three-dimensional models of circuit elements, chip models, etc., and this application embodiment does not limit them.

[0297] The information feature representation of the model prompt information is obtained, and the information feature representation is used to express the model prompt information in the form of feature vectors.

[0298] Obtain the parameter feature sequence corresponding to the primitive parameters predicted in the first i-1 prediction stages;

[0299] By fusing the information feature representation and the parameter feature sequence, a fused feature representation is obtained.

[0300] In an optional embodiment, the primitive parameters predicted in the first i-1 prediction stages are encoded to obtain the parameter feature representations corresponding to the first i-1 prediction stages respectively; the parameter feature representations corresponding to the first i-1 prediction stages are concatenated in sequence to obtain the parameter feature sequence.

[0301] In an optional embodiment, the predicted fusion feature representation is used to obtain the prediction result of the terminator in the i-th prediction stage, which indicates the end of the prediction process in the i-th prediction stage; in response to the generation of the terminator corresponding to the i-th prediction stage, a three-dimensional part model is generated based on the primitives determined in the i-th prediction stage.

[0302] In an optional embodiment, the primitive parameters corresponding to the primitives predicted in the i-th prediction stage are obtained by decoding the fused feature representation through the first decoder; the terminator prediction result of the i-th prediction stage is obtained by decoding the fused feature representation through the second decoder.

[0303] In an optional embodiment, the primitive parameters include primitive type parameters and primitive shape parameters, wherein the primitive type parameters are used to indicate the geometry of the primitive, and the primitive shape parameters are used to indicate the transformation method of the basic shape of the primitive;

[0304] Obtain the equivalent transformation set corresponding to the primitive type parameter, wherein the equivalent transformation set includes at least one candidate transformation scheme, and the at least one candidate transformation scheme has the same transformation result on the basic form of the primitive;

[0305] The primitive morphological parameters are updated based on the at least one candidate transformation scheme.

[0306] In an optional embodiment, the primitive morphological parameters are combined with the at least one candidate transformation scheme to obtain at least one morphological transformation scheme, wherein the transformation result of the at least one morphological transformation scheme on the basic morphology of the primitive matches the primitive morphological parameters.

[0307] The target morphological transformation scheme that meets the preset requirements is obtained from the at least one morphological transformation scheme, and the updated primitive morphological parameters are obtained.

[0308] In an optional embodiment, the primitive rotation angle corresponding to the at least one morphological transformation scheme is obtained, wherein the primitive rotation angle refers to the rotation angle of the basic form of the primitive in the morphological transformation scheme.

[0309] Obtain the target shape transformation scheme whose primitive rotation angle meets the preset requirements, and obtain the updated primitive shape parameters.

[0310] In an optional embodiment, the at least one candidate transformation scheme includes at least one rotation scheme, the rotation scheme including a rotation axis and a rotation angle;

[0311] Obtain a first rotation scheme from the primitive shape parameters, wherein the first rotation scheme includes a rotation axis and a rotation angle;

[0312] The rotation angle in the first rotation scheme is added to the rotation angle in the at least one candidate transformation scheme according to the rotation axis to obtain the at least one shape transformation scheme.

[0313] In an optional embodiment, the rotation angle in the first rotation scheme is added to the rotation angle in the at least one candidate transformation scheme according to the rotation axis to obtain the rotation angle sum corresponding to at least one rotation axis;

[0314] By adjusting the rotation angle to a preset angle range, at least one shape transformation scheme is obtained.

[0315] In an optional embodiment, sample prompt information and a reference 3D part model corresponding to the sample prompt information are obtained. The reference 3D part model is a 3D part model constructed by at least one first primitive. The sample prompt information is predicted by a primitive prediction network to obtain a predicted 3D part model. The predicted 3D part model is a 3D part model constructed by at least one second primitive. The primitive prediction network is trained based on the difference between the predicted 3D part model and the reference 3D part model.

[0316] In an optional embodiment, a first primitive sequence consisting of at least one first primitive in a reference 3D part model is obtained, the first primitive sequence being a sequence obtained by arranging the at least one first primitive in the generation order; a second primitive sequence consisting of at least one second primitive in a predicted 3D part model is obtained, the second primitive sequence being a sequence obtained by arranging the second primitive in the prediction order; and a primitive prediction network is trained based on the difference between the first primitive sequence and the second primitive sequence.

[0317] In an optional embodiment, a first loss value is obtained based on the primitive parameters of the i-th first primitive in the first primitive sequence and the primitive parameters of the i-th second primitive in the second primitive sequence, where i is a positive integer; and the primitive prediction network is trained using the first loss value.

[0318] In an optional embodiment, a second loss value is obtained based on the termination position of the first primitive sequence and the termination position of the second primitive sequence; the primitive prediction network is trained using the second loss value.

[0319] In an optional embodiment, first point cloud data corresponding to a reference 3D part model is obtained, and the first point cloud data is used to represent the reference 3D part model in a point cloud data format; the predicted 3D part model is converted into second point cloud data, and the second point cloud data is used to represent the predicted 3D part model in a point cloud data format; a third loss value is obtained based on the distance between the first point cloud in the first point cloud data and the second point cloud in the second point cloud data; and a primitive prediction network is trained using the third loss value.

[0320] In summary, the method provided in this application, by iteratively determining the primitive parameters of at least two primitives, simulates the process of gradually constructing a three-dimensional part model by sequentially determining primitives in human cognition. By constraining the primitive parameters of subsequently determined primitives using the primitive parameters of primitives predicted in the preceding prediction stage, the method improves the coordination between primitives, avoids the problems of misalignment and weak compatibility among multiple primitives caused by determining multiple primitives in parallel, improves the accuracy of primitive determination, and enhances the expressive power of primitives for model components.

[0321] In some embodiments, taking the XR field as an example, such as in an AR navigation scenario, illustratively, model prompt information is obtained, such as point cloud data obtained by acquiring depth information of buildings in the physical environment through a depth camera and converting it. The model prompt information is used to indicate the requirement to generate a 3D building model in AR navigation through at least one primitive, where a primitive refers to a 3D geometric element.

[0322] In the i-th prediction stage, the primitive parameters corresponding to the primitives predicted in the previous i-1 prediction stages are obtained, where i > 1 and i is an integer. The primitive parameters are used to indicate the geometric shape of the primitives. The model prompt information and the primitive parameters corresponding to the primitives predicted in the previous i-1 prediction stages are fused to obtain the fused feature representation. The fused feature representation is predicted to obtain the primitive parameters corresponding to the primitives predicted in the i-th prediction stage. This process continues until the prediction stage ends and the three-dimensional building model in AR navigation is obtained.

[0323] In some embodiments, building point cloud data is received. The building point cloud data is point cloud data collected by a depth camera from buildings in the physical environment. The building point cloud data is used to indicate the generation of a three-dimensional building model corresponding to the building. Then, the prediction fusion feature representation is obtained to obtain the primitive parameters corresponding to the primitives predicted in the i-th prediction stage. Until the prediction stage ends, the three-dimensional building model corresponding to the building is generated and displayed.

[0324] The three-dimensional building models include three-dimensional building models, three-dimensional bridge models, three-dimensional signal tower models, etc., and this application embodiment does not limit them.

[0325] The information feature representation of the model prompt information is obtained, and the information feature representation is used to express the model prompt information in the form of feature vectors.

[0326] Obtain the parameter feature sequence corresponding to the primitive parameters predicted in the first i-1 prediction stages;

[0327] By fusing the information feature representation and the parameter feature sequence, a fused feature representation is obtained.

[0328] In an optional embodiment, the primitive parameters predicted in the first i-1 prediction stages are encoded to obtain the parameter feature representations corresponding to the first i-1 prediction stages respectively; the parameter feature representations corresponding to the first i-1 prediction stages are concatenated in sequence to obtain the parameter feature sequence.

[0329] In an optional embodiment, the predicted fusion feature representation is used to obtain the prediction result of the terminator in the i-th prediction stage, which indicates the end of the prediction process in the i-th prediction stage; in response to the generation of the terminator corresponding to the i-th prediction stage, a three-dimensional building model is generated based on the primitives determined in the i-th prediction stage.

[0330] In an optional embodiment, the primitive parameters corresponding to the primitives predicted in the i-th prediction stage are obtained by decoding the fused feature representation through the first decoder; the terminator prediction result of the i-th prediction stage is obtained by decoding the fused feature representation through the second decoder.

[0331] In an optional embodiment, the primitive parameters include primitive type parameters and primitive shape parameters, wherein the primitive type parameters are used to indicate the geometry of the primitive, and the primitive shape parameters are used to indicate the transformation method of the basic shape of the primitive;

[0332] Obtain the equivalent transformation set corresponding to the primitive type parameter, wherein the equivalent transformation set includes at least one candidate transformation scheme, and the at least one candidate transformation scheme has the same transformation result on the basic form of the primitive;

[0333] The primitive morphological parameters are updated based on the at least one candidate transformation scheme.

[0334] In an optional embodiment, the primitive morphological parameters are combined with the at least one candidate transformation scheme to obtain at least one morphological transformation scheme, wherein the transformation result of the at least one morphological transformation scheme on the basic morphology of the primitive matches the primitive morphological parameters.

[0335] The target morphological transformation scheme that meets the preset requirements is obtained from the at least one morphological transformation scheme, and the updated primitive morphological parameters are obtained.

[0336] In an optional embodiment, the primitive rotation angle corresponding to the at least one morphological transformation scheme is obtained, wherein the primitive rotation angle refers to the rotation angle of the basic form of the primitive in the morphological transformation scheme.

[0337] Obtain the target shape transformation scheme whose primitive rotation angle meets the preset requirements, and obtain the updated primitive shape parameters.

[0338] In an optional embodiment, the at least one candidate transformation scheme includes at least one rotation scheme, the rotation scheme including a rotation axis and a rotation angle;

[0339] Obtain a first rotation scheme from the primitive shape parameters, wherein the first rotation scheme includes a rotation axis and a rotation angle;

[0340] The rotation angle in the first rotation scheme is added to the rotation angle in the at least one candidate transformation scheme according to the rotation axis to obtain the at least one shape transformation scheme.

[0341] In an optional embodiment, the rotation angle in the first rotation scheme is added to the rotation angle in the at least one candidate transformation scheme according to the rotation axis to obtain the rotation angle sum corresponding to at least one rotation axis;

[0342] By adjusting the rotation angle to a preset angle range, at least one shape transformation scheme is obtained.

[0343] In an optional embodiment, sample prompt information and a reference 3D building model corresponding to the sample prompt information are obtained. The reference 3D building model is a 3D building model constructed by at least one first primitive. The sample prompt information is predicted by a primitive prediction network to obtain a predicted 3D building model. The predicted 3D building model is a 3D building model constructed by at least one second primitive. The primitive prediction network is trained based on the difference between the predicted 3D building model and the reference 3D building model.

[0344] In an optional embodiment, a first primitive sequence consisting of at least one first primitive in a reference 3D building model is obtained, the first primitive sequence being a sequence obtained by arranging the at least one first primitive in the generation order; a second primitive sequence consisting of at least one second primitive in a predicted 3D building model is obtained, the second primitive sequence being a sequence obtained by arranging the second primitive in the prediction order; and a primitive prediction network is trained based on the difference between the first primitive sequence and the second primitive sequence.

[0345] In an optional embodiment, a first loss value is obtained based on the primitive parameters of the i-th first primitive in the first primitive sequence and the primitive parameters of the i-th second primitive in the second primitive sequence, where i is a positive integer; and the primitive prediction network is trained using the first loss value.

[0346] In an optional embodiment, a second loss value is obtained based on the termination position of the first primitive sequence and the termination position of the second primitive sequence; the primitive prediction network is trained using the second loss value.

[0347] In an optional embodiment, first point cloud data corresponding to a reference 3D building model is obtained, and the first point cloud data is used to represent the reference 3D building model in a point cloud data format; the predicted 3D building model is converted into second point cloud data, and the second point cloud data is used to represent the predicted 3D building model in a point cloud data format; a third loss value is obtained based on the distance between the first point cloud in the first point cloud data and the second point cloud in the second point cloud data; and a primitive prediction network is trained using the third loss value.

[0348] In summary, the method provided in this application, by iteratively determining the primitive parameters of at least two primitives, simulates the process of gradually constructing a three-dimensional building model by sequentially determining primitives in human cognition. By constraining the primitive parameters of subsequently determined primitives with the primitive parameters of primitives already predicted in the preceding prediction stage, the method improves the coordination between primitives, avoids the problems of misalignment and weak compatibility among multiple primitives caused by determining multiple primitives in parallel, improves the accuracy of primitive determination, and enhances the expressive ability of primitives for model components.

[0349] Figure 11 This is a structural block diagram of a three-dimensional model generation apparatus provided in an exemplary embodiment of this application, such as... Figure 11 As shown, the device includes:

[0350] The acquisition module 1110 is used to acquire model prompt information, which indicates the requirement to generate the three-dimensional model through at least one primitive, where the primitive refers to a three-dimensional geometric element.

[0351] The acquisition module 1110 is also used to acquire the primitive parameters corresponding to the primitives predicted in the previous i-1 prediction stages in the i-th prediction stage, where i>1 and i is an integer, and the primitive parameters are used to indicate the geometric shape of the primitive.

[0352] The fusion module 1120 is used to fuse the model prompt information and the primitive parameters corresponding to the primitives predicted in the first i-1 prediction stages to obtain a fused feature representation;

[0353] The prediction module 1130 is used to predict the fused feature representation, obtain the primitive parameters corresponding to the primitives predicted in the i-th prediction stage, and obtain the three-dimensional model until the prediction stage ends.

[0354] In an optional embodiment, the acquisition module 1110 is further configured to acquire the information feature representation of the model prompt information, wherein the information feature representation is used to express the model prompt information in the form of feature vectors; and acquire the parameter feature sequence corresponding to the primitive parameters predicted in the first i-1 prediction stages;

[0355] The fusion module 1120 is also used to fuse the information feature representation and the parameter feature sequence to obtain the fused feature representation.

[0356] In an optional embodiment, the acquisition module 1110 is further configured to encode the primitive parameters predicted in the first i-1 prediction stages to obtain the parameter feature representations corresponding to the first i-1 prediction stages respectively;

[0357] The acquisition module 1110 is also used to sequentially concatenate the parameter feature representations corresponding to the first i-1 prediction stages to obtain the parameter feature sequence.

[0358] In an optional embodiment, the prediction module 1130 is further configured to predict the fused feature representation to obtain the terminator prediction result of the i-th prediction stage, wherein the terminator prediction result is used to indicate the situation where the prediction process ends in the i-th prediction stage.

[0359] like Figure 12 As shown, the device also includes:

[0360] The generation module 1140 is used to generate a generation terminator in response to the i-th prediction stage and to generate the three-dimensional model based on the primitives determined in the i-th prediction stage.

[0361] In an optional embodiment, the prediction module 1130 is further configured to decode the fused feature representation through a first decoder to obtain the primitive parameters corresponding to the primitives predicted in the i-th prediction stage;

[0362] The prediction module 1130 is also used to decode the fused feature representation through the second decoder to obtain the terminator prediction result of the i-th prediction stage.

[0363] In an optional embodiment, the primitive parameters include primitive type parameters and primitive shape parameters. The primitive type parameters are used to indicate the geometry of the primitive, and the primitive shape parameters are used to indicate the transformation method of the basic shape of the primitive.

[0364] The acquisition module 1110 is further configured to acquire an equivalent transformation set corresponding to the primitive type parameter, wherein the equivalent transformation set includes at least one candidate transformation scheme, and the at least one candidate transformation scheme has the same transformation result on the basic form of the primitive; and update the primitive form parameter based on the at least one candidate transformation scheme.

[0365] In an optional embodiment, the acquisition module 1110 is further configured to combine the primitive morphological parameters with the at least one candidate transformation scheme respectively to obtain at least one morphological transformation scheme, wherein the transformation result of the at least one morphological transformation scheme on the basic morphology of the primitive matches the primitive morphological parameters.

[0366] The acquisition module 1110 is further configured to acquire a target morphological transformation scheme that meets preset requirements from the at least one morphological transformation scheme, and obtain the updated primitive morphological parameters.

[0367] In an optional embodiment, the acquisition module 1110 is further configured to acquire the primitive rotation angles corresponding to the at least one morphological transformation scheme, wherein the primitive rotation angle refers to the rotation angle of the basic form of the primitive in the morphological transformation scheme.

[0368] The acquisition module 1110 is also used to acquire the target shape transformation scheme in which the rotation angle of the primitive meets the preset requirements, and to obtain the updated primitive shape parameters.

[0369] In an optional embodiment, the at least one candidate transformation scheme includes at least one rotation scheme, the rotation scheme including a rotation axis and a rotation angle;

[0370] The acquisition module 1110 is further configured to acquire a first rotation scheme in the primitive shape parameters, the first rotation scheme including a rotation axis and a rotation angle; and to add the rotation angle in the first rotation scheme to the rotation angle in the at least one candidate transformation scheme according to the rotation axis to obtain the at least one shape transformation scheme.

[0371] In an optional embodiment, the acquisition module 1110 is further configured to add the rotation angle in the first rotation scheme to the rotation angle in the at least one candidate transformation scheme according to the rotation axis to obtain the rotation angle sum corresponding to at least one rotation axis; and correct the rotation angle sum to a preset angle range to obtain the at least one shape transformation scheme.

[0372] In an optional embodiment, the acquisition module 1110 is further configured to acquire sample prompt information and a reference three-dimensional model corresponding to the sample prompt information, wherein the reference three-dimensional model is a three-dimensional model constructed by at least one first primitive.

[0373] The prediction module 1130 is further configured to predict the sample prompt information through the primitive prediction network to obtain a predicted three-dimensional model, wherein the predicted three-dimensional model is a three-dimensional model constructed by at least one second primitive.

[0374] The device also includes:

[0375] Training module 1150 is used to train the primitive prediction network based on the difference between the predicted 3D model and the reference 3D model.

[0376] In an optional embodiment, the acquisition module 1110 is further configured to acquire a first primitive sequence composed of at least one first primitive in the reference three-dimensional model, wherein the first primitive sequence is a sequence obtained by arranging the at least one first primitive in the generation order of the first primitive.

[0377] The acquisition module 1110 is further configured to acquire a second primitive sequence consisting of at least one second primitive in the predicted three-dimensional model, wherein the second primitive sequence is a sequence obtained by arranging the primitives in the prediction order;

[0378] The training module 1150 is also used to train the primitive prediction network based on the difference between the first primitive sequence and the second primitive sequence.

[0379] In an optional embodiment, the training module 1150 is further configured to obtain a first loss value based on the primitive parameters of the i-th first primitive in the first primitive sequence and the primitive parameters of the i-th second primitive in the second primitive sequence, where i is a positive integer;

[0380] The training module 1150 is also used to train the primitive prediction network using the first loss value.

[0381] In an optional embodiment, the training module 1150 is further configured to obtain a second loss value based on the termination position of the first primitive sequence and the termination position of the second primitive sequence; and train the primitive prediction network using the second loss value.

[0382] In an optional embodiment, the training module 1150 is further configured to: acquire first point cloud data corresponding to the reference 3D model, the first point cloud data being used to represent the reference 3D model in a point cloud data format; convert the predicted 3D model into second point cloud data, the second point cloud data being used to represent the predicted 3D model in the point cloud data format; obtain a third loss value based on the distance between the first point cloud in the first point cloud data and the second point cloud in the second point cloud data; and train the primitive prediction network using the third loss value.

[0383] In summary, the apparatus provided in this application, by iteratively determining the primitive parameters of at least two primitives, simulates the process of gradually building a three-dimensional model by sequentially determining primitives in human cognition. By constraining the primitive parameters of subsequently determined primitives using the primitive parameters of primitives predicted in the preceding prediction stage, the apparatus improves the coordination between primitives, avoids the problems of misalignment and weak compatibility among multiple primitives caused by determining multiple primitives in parallel, improves the accuracy of primitive determination, and enhances the expressive ability of primitives to represent the components of the model.

[0384] It should be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the content structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0385] Please refer to Figure 13 This diagram illustrates a structural block diagram of a computer device 1300 provided in one embodiment of this application. The computer device 1300 may be... Figure 2 The terminal or server in the computer system shown is used to implement the three-dimensional model generation method provided in the above embodiments. Specifically:

[0386] Typically, computer device 1300 includes a processor 1301 and a memory 1302.

[0387] Processor 1301 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 1301 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), and PLA (Programmable Logic Array). Processor 1301 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1301 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 1301 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0388] The memory 1302 may include one or more computer-readable storage media, which may be non-transitory. The memory 1302 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1302 are used to store a computer program configured to be executed by one or more processors to implement the above-described method for generating the three-dimensional model.

[0389] In some embodiments, the computer device 1300 may also optionally include other components 1303: a peripheral device interface and at least one peripheral device. The processor 1301, memory 1302, and peripheral device interface can be connected via a bus or signal lines. Each peripheral device can be connected to the peripheral device interface via a bus, signal lines, or a circuit board. Specifically, the peripheral device includes at least one of: radio frequency circuitry, a display screen, audio circuitry, and a power supply.

[0390] Those skilled in the art will understand that Figure 13 The structure shown does not constitute a limitation on the computer device 1300, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0391] In an exemplary embodiment, a computer-readable storage medium is also provided, wherein a computer program is stored in the storage medium, and the computer program, when executed by a processor, implements the above-described method for generating a three-dimensional model. Optionally, the computer-readable storage medium may include: ROM (Read-Only Memory), RAM (Random Access Memory), SSD (Solid State Drives), or optical disc, etc. The random access memory may include ReRAM (Resistance Random Access Memory) and DRAM (Dynamic Random Access Memory).

[0392] In an exemplary embodiment, a computer program product is also provided, the computer program product including a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, causing the computer device to perform the above-described method for generating a three-dimensional model.

[0393] It should be noted that the collection and processing of relevant data (such as virtual scenes, point cloud data, etc.) in this application should strictly comply with the requirements of relevant national laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0394] It should be understood that "multiple" as used herein refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. Furthermore, the step numbers described herein are merely illustrative of one possible execution order. In some other embodiments, the steps may not be executed in the order shown in the figures, such as two steps with different numbers being executed simultaneously, or two steps with different numbers being executed in the reverse order of the figures. This application does not limit this.

[0395] The above description is merely an exemplary embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.< / eos> < / sos> < / sos> < / sos> < / sos> < / sos> < / sos>

Claims

1. A method for generating a three-dimensional model, characterized in that, The method includes: Obtain model hint information, which indicates the requirement to generate the three-dimensional model using at least one primitive, where the primitive refers to a three-dimensional geometric element; In the i-th prediction stage, the primitive parameters corresponding to the primitives predicted in the previous i-1 prediction stages are obtained, where i > 1 and i is an integer. The primitive parameters are used to indicate the geometric shape of the primitive. By fusing the model prompt information and the primitive parameters corresponding to the primitives predicted in the first i-1 prediction stages, a fused feature representation is obtained; The fusion feature representation is predicted to obtain the primitive parameters corresponding to the primitives predicted in the i-th prediction stage, until the prediction stage ends and the three-dimensional model is obtained.

2. The method according to claim 1, characterized in that, The fusion of the model prompt information and the primitive parameters corresponding to the primitives predicted in the first i-1 prediction stages yields a fused feature representation, including: Obtain the information feature representation of the model prompt information, wherein the information feature representation is used to express the model prompt information in the form of feature vectors; Obtain the parameter feature sequence corresponding to the primitive parameters predicted in the first i-1 prediction stages; The fused feature representation is obtained by fusing the information feature representation and the parameter feature sequence.

3. The method according to claim 2, characterized in that, The step of obtaining the parameter feature sequence corresponding to the primitive parameters predicted in the first i-1 prediction stages includes: Encode the primitive parameters predicted in the first i-1 prediction stages to obtain the parameter feature representations corresponding to the first i-1 prediction stages respectively; The parameter feature sequence is obtained by sequentially concatenating the parameter feature representations corresponding to the first i-1 prediction stages.

4. The method according to any one of claims 1 to 3, characterized in that, The method further includes: The fused feature representation is predicted to obtain the terminator prediction result of the i-th prediction stage, which is used to indicate the situation where the prediction process ends in the i-th prediction stage; In response to the generation of a terminator corresponding to the i-th prediction stage, the three-dimensional model is generated based on the primitives determined in the i-th prediction stage.

5. The method according to claim 4, characterized in that, The prediction of the fused feature representation yields the primitive parameters corresponding to the primitives predicted in the i-th prediction stage, including: The fused feature representation is decoded by the first decoder to obtain the primitive parameters corresponding to the primitives predicted in the i-th prediction stage. The prediction of the fused feature representation to obtain the terminator prediction result of the i-th prediction stage includes: The fused feature representation is decoded by the second decoder to obtain the terminator prediction result of the i-th prediction stage.

6. The method according to any one of claims 1 to 3, characterized in that, The primitive parameters include primitive type parameters and primitive shape parameters. The primitive type parameters are used to indicate the geometric shape of the primitive, and the primitive shape parameters are used to indicate the transformation method of the basic shape of the primitive. The method further includes: Obtain the equivalent transformation set corresponding to the primitive type parameter, wherein the equivalent transformation set includes at least one candidate transformation scheme, and the at least one candidate transformation scheme has the same transformation result on the basic form of the primitive; The primitive morphological parameters are updated based on the at least one candidate transformation scheme.

7. The method according to claim 6, characterized in that, Updating the primitive morphological parameters based on the at least one candidate transformation scheme includes: The primitive morphological parameters are combined with the at least one candidate transformation scheme to obtain at least one morphological transformation scheme, wherein the transformation result of the at least one morphological transformation scheme on the basic morphology of the primitive matches the primitive morphological parameters. The target morphological transformation scheme that meets the preset requirements is obtained from the at least one morphological transformation scheme, and the updated primitive morphological parameters are obtained.

8. The method according to claim 7, characterized in that, The step of obtaining a target morphological transformation scheme that meets preset requirements from the at least one morphological transformation scheme to obtain the updated primitive morphological parameters includes: Obtain the primitive rotation angle corresponding to each of the at least one morphological transformation schemes, wherein the primitive rotation angle refers to the rotation angle of the basic form of the primitive in the morphological transformation scheme; Obtain the target shape transformation scheme whose primitive rotation angle meets the preset requirements, and obtain the updated primitive shape parameters.

9. The method according to claim 7, characterized in that, The at least one candidate transformation scheme includes at least one rotation scheme, the rotation scheme including a rotation axis and a rotation angle; The step of combining the primitive morphological parameters with the at least one candidate transformation scheme to obtain at least one morphological transformation scheme includes: Obtain a first rotation scheme from the primitive shape parameters, wherein the first rotation scheme includes a rotation axis and a rotation angle; The rotation angle in the first rotation scheme is added to the rotation angle in the at least one candidate transformation scheme according to the rotation axis to obtain the at least one shape transformation scheme.

10. The method according to claim 9, characterized in that, The step of adding the rotation angle in the first rotation scheme to the rotation angle in the at least one candidate transformation scheme according to the rotation axis to obtain the at least one shape transformation scheme includes: Add the rotation angle in the first rotation scheme to the rotation angle in the at least one candidate transformation scheme according to the rotation axis to obtain the sum of rotation angles corresponding to at least one rotation axis; The rotation angle is adjusted to a preset angle range to obtain at least one shape transformation scheme.

11. The method according to any one of claims 1 to 3, characterized in that, The method is performed by a primitive prediction network; The training process of the primitive prediction network includes: Obtain sample prompt information and a reference 3D model corresponding to the sample prompt information, wherein the reference 3D model is a 3D model constructed by at least one first primitive; The sample prompt information is predicted by the primitive prediction network to obtain a predicted three-dimensional model, which is a three-dimensional model constructed by at least one second primitive. The primitive prediction network is trained based on the differences between the predicted 3D model and the reference 3D model.

12. The method according to claim 11, characterized in that, The step of training the primitive prediction network based on the difference between the predicted 3D model and the reference 3D model includes: Obtain a sequence of first primitives consisting of at least one first primitive in the reference 3D model, wherein the first primitive sequence is a sequence obtained by arranging the at least one first primitive in the order of its generation; Obtain a sequence of second primitives consisting of at least one second primitive in the predicted 3D model, wherein the second primitive sequence is a sequence obtained by arranging them in the prediction order; The primitive prediction network is trained based on the differences between the first primitive sequence and the second primitive sequence.

13. The method according to claim 12, characterized in that, The step of training the primitive prediction network based on the difference between the first primitive sequence and the second primitive sequence includes: Based on the primitive parameters of the i-th first primitive in the first primitive sequence and the primitive parameters of the i-th second primitive in the second primitive sequence, a first loss value is obtained, where i is a positive integer; The primitive prediction network is trained using the first loss value.

14. The method according to claim 12, characterized in that, The step of training the primitive prediction network based on the difference between the first primitive sequence and the second primitive sequence includes: Based on the termination positions of the first primitive sequence and the second primitive sequence, a second loss value is obtained; The primitive prediction network is trained using the second loss value.

15. The method according to claim 11, characterized in that, The step of training the primitive prediction network based on the difference between the predicted 3D model and the reference 3D model includes: Obtain the first point cloud data corresponding to the reference 3D model, and the first point cloud data is used to express the reference 3D model in a point cloud data format; The predicted 3D model is converted into second point cloud data, which is used to express the predicted 3D model through the point cloud data format. Based on the distance between the first point cloud in the first point cloud data and the second point cloud in the second point cloud data, a third loss value is obtained; The primitive prediction network is trained using the third loss value.

16. The method according to any one of claims 1 to 3, characterized in that, The acquisition of model prompt information includes: Display prompt information input area, the prompt information input area is used to instruct the generation of a three-dimensional scene model in the virtual scene; Receive input operations for the model prompt information in the prompt information input area; The prediction of the fused feature representation, obtaining the primitive parameters corresponding to the primitives predicted in the i-th prediction stage, until the end of the prediction stage to obtain the three-dimensional model, includes: The fusion feature representation is predicted to obtain the primitive parameters corresponding to the primitives predicted in the i-th prediction stage. This process continues until the prediction stage ends. The three-dimensional scene model is then generated and displayed in the virtual scene, and the three-dimensional scene model is implemented as scene elements in the virtual scene.

17. A device for generating a three-dimensional model, characterized in that, The device includes: The acquisition module is used to acquire model prompt information, which indicates the requirement to generate the three-dimensional model through at least one primitive, where the primitive refers to a three-dimensional geometric element. The acquisition module is further configured to acquire the primitive parameters corresponding to the primitives predicted in the previous i-1 prediction stages in the i-th prediction stage, where i > 1 and i is an integer, and the primitive parameters are used to indicate the geometric shape of the primitive. The fusion module is used to fuse the model prompt information and the primitive parameters corresponding to the primitives predicted in the first i-1 prediction stages to obtain a fused feature representation; The prediction module is used to predict the fused feature representation, obtain the primitive parameters corresponding to the primitives predicted in the i-th prediction stage, and obtain the three-dimensional model at the end of the prediction stage.

18. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing a computer program that is loaded and executed by the processor to implement the method for generating a three-dimensional model as described in any one of claims 1 to 16.

19. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which is loaded and executed by a processor to implement the method for generating a three-dimensional model as described in any one of claims 1 to 16.

20. A computer program product, characterized in that, The computer program product includes a computer program stored in a computer-readable storage medium, and a processor reads from and executes the computer program to implement the method for generating a three-dimensional model as described in any one of claims 1 to 16.

Images