Three-dimensional model generation method and device, equipment, medium and program product

Abstract

The invention discloses a three-dimensional model generation method and device, equipment, a medium and a program product, and relates to the field of artificial intelligence. The method comprises the steps of obtaining object prompt information of a first object; performing a denoising process on the noise data by taking the object prompt information as a denoising condition to obtain a denoising result; and inputting the performance characteristic parameters into a generator, and generating three-dimensional model data corresponding to the first object through the generator. According to the method, the device and the system, the process of obtaining the performance characteristic parameters is modeled as the denoising process of the noise data, the performance characteristic parameters meeting the conditions are quickly generated, the adjustment process of the performance characteristic parameters is reduced, and therefore the generation efficiency of the three-dimensional model is improved. The method can be applied to various application scenes such as three-dimensional modeling, animation production and building design.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence, and in particular to a method, device, equipment, medium and program product for generating a three-dimensional model. Background Art

[0002] Procedural Content Generation (PCG) is a computer program that automatically generates three-dimensional content. Taking the generation of three-dimensional models as an example, different parameter combinations are input into the procedural generator, and various three-dimensional models of the target object will be output. The inverse PCG task is an important application in the field of PCG, which aims to generate a three-dimensional model from a two-dimensional image.

[0003] In the related art, the inverse PCG task is implemented by a random sampling method. It is assumed that the procedural generator contains n variables and the two-dimensional image contains object a. Multiple parameter combinations are obtained by random sampling, and each parameter combination includes n variable values. These multiple parameter combinations are respectively input into the procedural generator to obtain multiple candidate three-dimensional models. The technician needs to select the three-dimensional model that can most accurately restore the object a in the two-dimensional image from these multiple candidate three-dimensional models.

[0004] However, implementing the inverse PCG task based on random sampling usually requires multiple samplings, each of which involves the generation process of the procedural generator, resulting in low generation efficiency of the 3D model. Summary of the invention

[0005] The present application provides a method, device, equipment, medium and program product for generating a three-dimensional model. The technical solution is as follows:

[0006] In one aspect, a method for generating a three-dimensional model is provided, the method comprising:

[0007] Acquire object prompt information of a first object, where the object prompt information is used to describe the first object, and the object prompt information includes at least one of image information and text information;

[0008] Using the object prompt information as a denoising condition, performing a denoising process on the noise data to obtain a denoising result, wherein the denoising result includes a performance feature parameter corresponding to the appearance performance of the first object;

[0009] The performance characteristic parameters are input into a generator, and the three-dimensional model data corresponding to the first object is generated by the generator.

[0010] In another aspect, a three-dimensional model generation device is provided, the device comprising:

[0011] An acquisition module, used to acquire object prompt information of a first object, where the object prompt information is used to describe the first object, and the object prompt information includes at least one of image information and text information;

[0012] A denoising module, configured to perform a denoising process on the noise data using the object prompt information as a denoising condition to obtain a denoising result, wherein the denoising result includes a performance feature parameter corresponding to the appearance performance of the first object;

[0013] A generation module is used to input the performance characteristic parameters into a generator, and generate three-dimensional model data corresponding to the first object through the generator.

[0014] On the other hand, a computer device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement any of the above-mentioned three-dimensional model generation methods.

[0015] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction, at least one program, a code set or an instruction set is stored in the storage medium, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement any of the above-mentioned methods for generating a three-dimensional model.

[0016] On the other hand, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs any of the above-mentioned methods for generating a three-dimensional model.

[0017] The beneficial effects brought by the technical solution provided by the embodiment of the present application include at least:

[0018] Under the guidance of the object prompt information, the noise data is denoised to obtain the performance feature parameters that match the object appearance described in the object prompt information, and the performance feature parameters are input into the generator to generate the three-dimensional model data of the object. In this application, the process of obtaining the performance feature parameters is modeled as the denoising process of the noise data, and the performance feature parameters that meet the conditions are quickly generated, which reduces the adjustment process of the performance feature parameters, thereby improving the generation efficiency of the three-dimensional model. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 is a schematic diagram of a method for generating a three-dimensional model provided by an exemplary embodiment of the present application;

[0021] Figure 2 is a schematic diagram of a computer system provided by an exemplary embodiment of the present application;

[0022] Figure 3 is a flow chart of a method for generating a three-dimensional model provided by an exemplary embodiment of the present application;

[0023] Figure 4 is a flow chart of a method for generating a three-dimensional model provided by another exemplary embodiment of the present application;

[0024] Figure 5 is a schematic diagram of a generated result after editing provided by an exemplary embodiment of the present application;

[0025] Figure 6 is a flowchart of a method for generating a three-dimensional model provided by another exemplary embodiment of the present application;

[0026] Figure 7 is a schematic diagram of a method for generating a three-dimensional model provided by another exemplary embodiment of the present application;

[0027] Figure 8 This is a comparison chart of the generation results of this application and other baseline methods in various categories;

[0028] Fig. 9 is a schematic diagram of a generation result based on a sketch provided by an exemplary embodiment of the present application;

[0029] Fig.10 is a structural block diagram of a three-dimensional model generation device provided by an exemplary embodiment of the present application;

[0030] Fig.11 It is a structural block diagram of a computer device provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solution and advantages of the present application clearer, the implementation mode of the present application will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.

[0032] In the present application, the terms "first", "second", etc. are used to distinguish identical or similar items with substantially the same effects and functions. It should be understood that there is no logical or temporal dependency between "first" and "second", nor are there any limitations on quantity and execution order.

[0033] It should be noted that before collecting relevant data of users and during the process of collecting relevant data of users, this application can display a prompt interface, pop-up window or output voice prompt information, and the prompt interface, pop-up window or voice prompt information is used to prompt the user that its relevant data is currently being collected, so that this application only starts to execute the relevant steps of obtaining relevant data of users after obtaining the confirmation operation issued by the user to the prompt interface or pop-up window, otherwise (that is, when the confirmation operation issued by the user to the prompt interface or pop-up window is not obtained), the relevant steps of obtaining relevant data of users are terminated, that is, the relevant data of users are not obtained. In other words, all user data collected by this application are collected with the consent and authorization of the user, and the collection, use and processing of relevant user data need to comply with relevant laws, regulations and standards.

[0034] First, a brief introduction is given to the terms involved in the embodiments of the present application.

[0035] Procedural Content Generation (PCG): refers to a computer program that automatically generates three-dimensional content, including but not limited to three-dimensional models (e.g., character models, building models, terrain models, prop models, etc.), dynamic effects, lighting effects, etc. Its basic principle is to define a series of rules and variables, and let the computer automatically generate three-dimensional content according to these rules and variables, so as to achieve efficient and large-scale generation of three-dimensional content. Taking the generation of three-dimensional models as an example, different parameter combinations are input into the procedural generator, and various three-dimensional models of the target object will be output.

[0036] Inverse PCG task: It is an important application in the field of PCG, aiming to Generate 3D models from 2D images , that is, reconstructing a 3D model from a given 2D image information. For example, given a front photo of a chair, the inverse PCG task is to construct a complete 3D model of the chair, including the overall structure, decoration, texture, etc.

[0037] Diffusion Models (DM): is a generative model whose training phase includes a forward process and a reverse process. The forward process is a Markov chain, which is a process of gradually adding noise to the input value (input sample) until a pure noise is obtained. Each step here can be called a time step; the reverse process is a process of gradually denoising a noisy data until the input value (input sample) is obtained. That is, the forward process is a process of adding noise, and the reverse process is a process of denoising and inference. The diffusion model mainly learns the reverse process during the training phase, that is, learning how to infer the input value (input sample) from a noisy data.

[0038] In the related art, the inverse PCG task is implemented by random sampling. It is assumed that the procedural generator contains n variables and the two-dimensional image contains object a. Multiple parameter combinations are obtained by random sampling, and each parameter combination includes n variable values. These multiple parameter combinations are respectively input into the procedural generator to obtain multiple candidate three-dimensional models. The technician needs to select the three-dimensional model that can most accurately restore the object a in the two-dimensional image from these multiple candidate three-dimensional models. However, implementing the inverse PCG task based on random sampling usually requires multiple samplings, and each sampling involves the generation process of the procedural generator, resulting in low efficiency in generating three-dimensional models.

[0039] Based on this, the present application provides a method for generating a three-dimensional model, which denoises noise data under the guidance of object prompt information, thereby obtaining performance feature parameters of the object appearance performance that conforms to the description of the object prompt information, and then inputs the performance feature parameters into a generator to generate three-dimensional model data corresponding to the object. Below, taking the object prompt information implemented as image information as an example, the method for generating a three-dimensional model provided by the present application is explained.

[0040] For illustration, please refer to Figure 1 , which shows a schematic diagram of a method for generating a three-dimensional model, the method comprising the following steps:

[0041] Step 1: Input the image 110 into the image encoder 120, and encode the image 110 through the image encoder 120 to obtain image feature representation.

[0042] The image 110 refers to a two-dimensional image including the object a, wherein the form of the image 110 includes but is not limited to: a sketch (line drawing), a real picture (such as a photograph), a design drawing, a cartoon picture, an oil painting, etc.

[0043] The image feature representation is used to characterize the visual features of the object a in the image 110, such as color features, texture features, shape features, spatial relationship features, etc.

[0044] The image encoder 120 can be implemented as an encoder based on a convolutional neural network, an encoder based on a visual Transformer, etc., which is not limited here.

[0045] Step 2: Input the image feature representation and the noise data 130 into the denoising unit 140, use the image feature representation as the denoising condition, perform the denoising process on the noise data 130 through the denoising unit 140, and obtain the performance feature parameter 150 as the denoising result.

[0046] The noise data can be implemented as n random noise values, where n is a positive integer, and n refers to the number of variables in the generator 160 described below. The n random noise values ​​correspond one-to-one to the n variables in the generator 160. It should be noted that the random noise value refers to the "variable value" of the variable in the generator, but this "variable value" contains "noise", which needs to be removed by the denoising unit 140.

[0047] The denoising unit 140 will gradually remove the "noise" in the noise data under the guidance of the image feature representation, thereby obtaining the performance feature parameter 150. Optionally, there are at least two denoising units 140 ( Figure 1 Only one denoising unit 140 is shown as an illustration), and at least two denoising units 140 denoise the noise data in a certain order. Among them, the i+1 denoising unit will obtain the image feature representation and the output result of the i denoising unit, and under the guidance of the image feature representation, remove the "noise" in the output result of the i denoising unit to obtain the output result of the i+1 denoising unit, where i is a positive integer; then, the output result of the last denoising unit is also the performance feature parameter 150.

[0048] The performance characteristic parameter 150 is also the "variable value" for removing "noise", and the performance characteristic parameter 150 includes n characteristic parameters, and these n characteristic parameters correspond one-to-one to the n variables in the generator 160. For example, if the first variable in the generator 160 is the color variable of the three-dimensional chair model, then the first characteristic parameter refers to the color parameter of the three-dimensional model chair, such as: the chair parameter is 1, indicating that the chair is red.

[0049] Step 3: Input the performance characteristic parameters 150 into the generator 160, and generate the three-dimensional model data 170 corresponding to the object a through the generator 160.

[0050] The generator 160 refers to a program for generating a three-dimensional model of the object a. The generator 160 is a program pre-written by a technician, wherein n variables are defined, where n is a positive integer, and the n variables are used to control the shape, size, color, texture, etc. of the generated three-dimensional model. By assigning different variable values ​​to the n variables, the generator 160 generates three-dimensional models with different appearances.

[0051] The performance characteristic parameters 150 are input into the generator 160 , that is, n characteristic parameters in the performance characteristic parameters 150 are used as variable values ​​and assigned one by one to corresponding variables, so that the generator 160 generates three-dimensional model data 170 .

[0052] Among them, since the performance feature parameters 150 are generated by the denoising unit 140 under the guidance of the image feature representation, the performance feature parameters 150 match the appearance of the object a presented in the image 110. Then, the three-dimensional model data 170 generated by the generator 160 under the assignment of the performance feature parameters 150 is consistent with the appearance of the object a in the image 110, that is, the three-dimensional model data 170 can accurately reflect the appearance features of the object a, such as shape, color, and texture.

[0053] In summary, the method for generating a three-dimensional model provided in the embodiment of the present application can, under the guidance of the prompt image, denoise the noise data through the diffusion model, thereby obtaining the performance feature parameters that match the appearance of the object in the prompt image, and inputting the performance feature parameters into the generator can generate the three-dimensional model data of the object. In the present application, the process of obtaining the performance feature parameters is modeled as the denoising process of the noise data, and the performance feature parameters that meet the conditions are quickly generated, which reduces the adjustment process of the performance feature parameters, thereby improving the generation efficiency of the three-dimensional model.

[0054] Secondly, a computer system for implementing the method for generating a three-dimensional model provided in an embodiment of the present application is described. introduce.

[0055] Figure 2 1 is a block diagram of a computer system provided by an exemplary embodiment of the present application. The computer system can be implemented as a system architecture of a method for generating a three-dimensional model. The computer system includes: a terminal 210 and a server 220. The terminal 210 is connected to the server 220 via a wireless network or a wired network.

[0056] The terminal 210 may be an electronic device such as a mobile phone, a tablet computer, a vehicle-mounted terminal (vehicle computer), a wearable device, a PC (Personal Computer), etc. The terminal 210 may be installed with a client that runs a target application, and the target application may be an application for generating a three-dimensional model, or an application that provides a three-dimensional model generation function, which is not limited in this application. In addition, this application does not limit the form of the target application, including but not limited to App (Application) installed in the terminal 210, applets, etc., and may also be in the form of a web page.

[0057] Server 220 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, a cloud server that provides basic cloud computing services, or a node in a blockchain system. Server 220 can be the background server of the above-mentioned target application, used to provide background services for the client of the target application.

[0058] In the method for generating a three-dimensional model provided in the embodiment of the present application, the execution subject of each step may be a computer device, which refers to an electronic device with data calculation, processing and storage capabilities. Figure 1 As shown, the method for generating a three-dimensional model can be executed by the terminal 210 (for example, a client of a target application installed and running in the terminal 210 executes the method for generating a three-dimensional model), or the method for generating a three-dimensional model can be executed by the server 220, or the terminal 210 and the server 220 can interact and cooperate to execute the method for generating a three-dimensional model, and this application does not limit this.

[0059] Next, the process of the method for generating a three-dimensional model provided in an embodiment of the present application is introduced.

[0060] Figure 3 is a flowchart of a method for generating a three-dimensional model provided in an embodiment of the present application. The method is executed by a computer device, which may be Figure 2 The terminal 210 and / or the server 220 are shown. The method includes the following steps 310 to 330.

[0061] Step 310: Obtain object prompt information of the first object.

[0062] The object prompt information is used to describe the first object. The object prompt information includes at least one of image information and text information. The image information refers to a two-dimensional image containing the first object; the text information is used to describe the first object in text form.

[0063] Illustratively, the object prompt information refers to image information; or the object prompt information refers to text information; or the object prompt information includes text information and image information.

[0064] Optionally, the object prompt information includes at least one of the following description information:

[0065] (1) Appearance of the first object.

[0066] The appearance of the first object refers to external features of the first object that can be directly perceived by vision, such as shape, size, color, material, texture, decorative elements, etc.

[0067] When the object prompt information includes image information, the appearance of the first object may be presented through the image information. When the object prompt information includes text information, the appearance of the first object may be described through the text information.

[0068] (2) Dynamic effect of the first object.

[0069] The dynamic effect of the first object is used to display the characteristics of the first object during the motion or change process, such as: motion trajectory, speed, acceleration, deformation mode, light and shadow changes, etc.

[0070] When the object prompt information includes image information, the image information can be implemented as a dynamic picture or a continuous frame picture, so as to intuitively show the dynamic effect of the first object. When the object prompt information is text information, the dynamic effect of the first object can be described by the text information.

[0071] It should be noted that the above examples of description information are only for illustrative purposes and are not limited to this in the embodiments of the present application. For example, the description information contained in the object prompt information may also include scene information in which the first object is located. The scene information is used to indicate the specific spatial environment in which the first object is located, such as a street, park, playground, etc.

[0072] Optionally, the number of first objects described by the object prompt information may be one or more. In the embodiment of the present application, one first object is mainly used as an example for description.

[0073] Step 320, using the object prompt information as a denoising condition, performs a denoising process on the noise data to obtain a denoising result.

[0074] The denoising condition is used to indicate the guiding information in the denoising process of the noise data, and the denoising result generated based on the denoising condition meets the conditional content indicated by the denoising condition. In the present application, the object prompt information is used as the denoising condition, that is, the object prompt information is used as the guiding information in the denoising process of the noise data, and the denoising result generated based on the object prompt information meets the description of the object prompt information (or, the denoising result matches the object prompt information). For example: the denoising result meets the appearance of the first object described by the object prompt information; or, the denoising result meets the dynamic effect of the first object described by the object prompt information.

[0075] Optionally, the noise data refers to random noise. Schematically, this random noise refers to the variable value in the generator, but this variable value contains "noise". After removing the "noise" in this variable value, a "clean" variable value to be input into the generator can be obtained, which is the performance characteristic parameter in the denoising result described below.

[0076] In some embodiments, the noise data is obtained by random sampling from a preset noise distribution.

[0077] Optionally, the noise distribution can be implemented as at least one of Gaussian distribution, Laplace distribution, uniform distribution, etc., which is not limited here. Schematically, at least one noise value is randomly extracted from the Gaussian distribution, and the at least one noise value is used as noise data.

[0078] Optionally, the denoising process refers to the process of removing "noise" in noise data, or it can be understood as the process of adjusting the noise data according to the object prompt information. Schematically, in the denoising process of noise data, the noise data will be gradually adjusted according to the object prompt information to finally obtain the denoising result.

[0079] The denoising result includes a performance characteristic parameter corresponding to the appearance performance of the first object.

[0080] Optionally, the performance characteristic parameters match the appearance of the first object described in the object prompt information, that is, the characteristics of the first object reflected by the performance characteristic parameters are consistent with the appearance of the first object described in the object prompt information. For example, if the object prompt information describes the first object as "a wooden chair with a dark brown color and an arc-shaped backrest", then the corresponding performance characteristic parameters include: material parameters (wood), color parameters (brown), and shape parameters (the backrest is arc-shaped).

[0081] Optionally, the performance characteristic parameters are used to characterize model parameters of the three-dimensional model corresponding to the first object. Schematically, the model parameters include but are not limited to: shape-related parameters, material texture-related parameters, decoration parameters, etc., which are not limited here.

[0082] Optionally, the generator described below includes n variables, the noise data includes n sub-noise data, and n is a positive integer.

[0083] Taking the object prompt information as the denoising condition, a denoising process is performed on the n sub-noise data to obtain a denoising result, wherein the performance feature parameters included in the denoising result include n feature parameters, and the n feature parameters correspond to the n variables one by one.

[0084] In an illustrative manner, n noise values ​​are randomly sampled from a preset noise distribution as n sub-noise data. The n sub-noise data are used as noise data, and under the guidance of the object prompt information, the "noise" in the n sub-noise data is removed to obtain feature parameters corresponding to the n sub-noise data, that is, n feature parameters. These n feature parameters correspond to the n variables in the generator one by one, for example: the first feature parameter is used as the variable value assigned to the first variable in the generator.

[0085] In some embodiments, the above noise data can also be implemented as a performance characteristic parameter to be adjusted. Schematically, the performance characteristic parameter to be adjusted can be a characteristic parameter generated by applying the method for generating a three-dimensional model provided by the present application; or, the performance characteristic parameter to be adjusted is a variable value to be adjusted set by a relevant person for n variables in the generator with reference to the object prompt information. The denoising process of the performance characteristic parameter to be adjusted is performed with the object prompt information as the denoising condition, and the adjusted performance characteristic parameter can be obtained.

[0086] In the above embodiment, the n characteristic parameters correspond one-to-one to the n variables in the generator, so that the data input to the generator can accurately match its corresponding variables, thereby improving the accuracy of the three-dimensional model data generated by the generator.

[0087] In some embodiments, the object feature representation of the object prompt information is extracted, and the noise data is denoised using the object feature representation. The above step 320 also includes the following steps:

[0088] Step 1: Extract object feature representation corresponding to object prompt information.

[0089] The object feature representation is used to characterize the features of the first object in the object prompt information, such as appearance features, dynamic effect features, etc. of the first object. Optionally, the object feature representation may be implemented in a vector form.

[0090] When the object prompt information includes image information, the object feature representation is used to represent the visual features of the first object; when the object prompt information includes text information, the object feature representation is used to represent the semantics of the text information.

[0091] Illustratively, if the object prompt information refers to image information, the image feature representation corresponding to the image information is extracted as the object feature representation; if the object prompt information refers to text information, the text feature representation corresponding to the text information is extracted as the object feature representation; if the object prompt information includes image information and text information, the image feature representation corresponding to the image information is extracted, and the text feature representation corresponding to the text information is extracted, and the image feature representation and the text feature representation are fused to obtain the object feature representation.

[0092] Optionally, a specific method of fusing the image feature representation and the text feature representation to obtain the object feature representation includes:

[0093] The text feature representation and the image feature representation are concatenated to obtain the object feature representation. Schematically, assuming that the text feature representation is a vector with a length of O and the image feature representation is a vector with a length of I, then the concatenated object feature representation is a vector with a length of O+I. The feature concatenation method is simple and efficient to fuse text and image features, which improves the feature fusion efficiency. O and I are positive integers.

[0094] Alternatively, the text feature representation and the image feature representation are fused through an attention mechanism to obtain an object feature representation. Schematically, the attention mechanism is used to determine the attention weight between the text feature representation and the image feature representation; the text feature representation and the image feature representation are fused according to the attention weight to obtain the object feature representation. Among them, the similarity between the text feature representation and the image feature representation is calculated through the attention mechanism. Commonly used similarity metrics include dot product, cosine similarity, etc. The similarity is normalized by the softmax function to obtain the attention weight, and the image feature representation is weighted according to the attention weight to obtain the weighted image feature representation. The weighted image feature representation is concatenated or added with the text feature representation to obtain the object feature representation. The fusion of text and image features using the attention mechanism can give different importance to different image regions, so that the subtle relationship between the image and the text can be better captured, which is conducive to improving the quality of the fused object feature representation.

[0095] Step 2: Perform a denoising process on the noisy data based on the object feature representation to obtain the denoising result.

[0096] That is, the object feature representation is used as the denoising condition, and the denoising process is performed on the noise data to obtain the denoising result.

[0097] In some embodiments, the object prompt information is used as a denoising condition, and a denoising process is performed on the noise data through a diffusion model to obtain a denoising result.

[0098] Optionally, the object cue information and noise data are input into a diffusion model, and a denoised result is output.

[0099] The diffusion model is a machine learning model that learns the correlation between input prompt information and performance feature parameters during the training phase. The input prompt information includes at least one of image information and text information, and the performance feature parameters are used to characterize the model parameters of the three-dimensional model corresponding to the first object, and the performance feature parameters correspond one-to-one to the variables in the generator described below.

[0100] Diffusion models are machine learning models used to generate data. The training phase of diffusion models includes forward and reverse processes. Taking the prompt information as image information as an example, in the forward process, the model will gradually convert the sample performance characteristic parameters into noise data, where the sample performance characteristic parameters are input into the generator described below to generate a sample three-dimensional model, and the sample three-dimensional model is rendered to obtain a sample image. The reverse process is the process of reconstructing the sample performance characteristic parameters from the noise data, that is, the denoising process of the noise data is performed with the sample image as the denoising condition. In the reverse process, the diffusion model will learn the association between the sample image (as the input prompt information) and the sample performance characteristic parameters (as the performance characteristic parameters), where the sample image is used as a conditional guide information to guide the diffusion model to gradually restore the noise data to the sample performance characteristic parameters.

[0101] After the diffusion model training is completed (after the association relationship is learned), the noise data can be denoised to obtain the performance feature parameters based on the learned association relationship and the object prompt information as the denoising condition.

[0102] In an embodiment of the present application, the task of reversing the model parameters (i.e., performance feature parameters) of the three-dimensional model from object prompt information is modeled as a type of generation task, and it is proposed to use a diffusion model to model the parameter generation process of reversing the performance feature parameters from object prompt information, so as to quickly and high-quality generate the performance feature parameters, thereby generating a high-quality three-dimensional model.

[0103] Optionally, the diffusion model includes but is not limited to: a diffusion model based on a Transformer (converter) architecture, a diffusion model based on a U-net (convolutional neural network) architecture, etc.

[0104] Taking the diffusion model based on the Transformer architecture as an example, the diffusion model optionally includes an object encoder and a denoising unit. The process of inputting object prompt information and noise data into the diffusion model and outputting the denoising result includes the following steps:

[0105] Step 1: Input the object prompt information into the object encoder and output the object feature representation corresponding to the object prompt information.

[0106] Illustratively, if the object prompt information refers to image information, the above-mentioned object encoder is an image encoder, the image information is input into the image encoder, and the image feature representation is output as the object feature representation; if the object prompt information refers to text information, the above-mentioned object encoder is a text encoder, the text information is input into the text encoder, and the text feature representation is output as the object feature representation.

[0107] Optionally, if the object prompt information includes image information and text information, the image information and the text information are input into the object encoder, and an object feature representation is output, where the object feature representation is used to represent a fused feature representation of the image feature representation corresponding to the image information and the text feature representation corresponding to the text information.

[0108] Schematically, the object encoder is a multimodal encoder, and image information and text information are input into the multimodal encoder. The image feature representation corresponding to the image information and the text feature representation corresponding to the text information are extracted through the multimodal encoder, and the image feature representation and the text feature representation are fused to obtain the object feature representation. The specific method of fusing the image feature representation and the text feature representation to obtain the object feature representation has been described above and will not be repeated here.

[0109] In the above scheme, when the object hint information contains both image information and text information, the object encoder can output a fusion feature representation of the two, so that the object feature representation can integrate the intuitive visual content of the image and the semantic connotation contained in the text, thereby generating more accurate feature parameters. For example: in some scenarios, the image information shows the appearance of the first object but the text information describes its additional attributes such as function. The object feature representation can combine these two aspects, enhance the understanding and characterization of the entire first object, and generate three-dimensional model data that is more in line with actual needs.

[0110] Step 2: Input the object feature representation and noise data into the denoising unit and output the denoising result.

[0111] The denoising unit is used to perform a denoising process on the noise data based on the object feature representation.

[0112] In some embodiments, the denoising unit includes a self-attention layer, a cross-attention layer, and a feature transformation layer. The denoising process performed in the denoising unit is described below:

[0113] (1) The noise feature representation corresponding to the noise data is input into the self-attention layer, and the self-attention feature representation is output.

[0114] Optionally, the noise data is converted into a vector form. The noise vector is the noise feature representation corresponding to the noise data. The noise vector representation is input into the self-attention layer, and the self-attention feature representation is output.

[0115] The self-attention feature representation is used to characterize the association relationship between at least two noise feature elements contained in the noise feature representation.

[0116] Optionally, at least two noise feature elements in the noise feature representation correspond to different dimensions or components in the noise vector.

[0117] Optionally, the self-attention layer mainly determines the association between different noise feature elements based on the query, key and value mechanism. Schematically, the self-attention weight of each noise feature element is determined according to the association between each noise feature element and other noise feature elements, and the self-attention weight reflects the importance of each noise feature element relative to other noise feature elements. Each noise feature element is weighted according to its self-attention weight to obtain a self-attention feature representation.

[0118] In some embodiments, the self-attention layer can be implemented as a multi-head self-attention layer, where different heads focus on the correlation between different aspects of the input noise feature representation. The multi-head self-attention layer concatenates or performs other fusion operations (such as linearly transforming the feature representations output by each head and then adding them together) the feature representation output by each head to obtain a self-attention feature representation.

[0119] (2) Input the object feature representation and the self-attention feature representation into the cross-attention layer, and output the cross-attention feature representation. Sign indicates.

[0120] The cross-attention feature representation is used to characterize the correlation between the self-attention feature representation and the object feature representation.

[0121] Optionally, the cross-attention layer mainly determines the association between the self-attention feature representation and the object feature representation based on the query, key and value mechanism. Schematically, the cross-attention weight of each feature element in the self-attention feature representation is determined according to the association between each feature element in the self-attention feature representation and all object feature elements in the object feature representation. Each feature element in the self-attention feature representation is weighted according to the cross-attention weight of each feature element to obtain a cross-attention feature representation.

[0122] In some embodiments, the cross-attention layer can be implemented as a multi-head cross-attention layer, where different heads pay attention to the correlation between the input self-attention feature representation and the associated feature representation in different aspects. The multi-head cross-attention layer concatenates or performs other fusion operations (such as linearly transforming the feature representations output by each head and then adding them together) the feature representations output by each head to obtain a cross-attention feature representation.

[0123] (3) The cross-attention feature representation is input into the feature transformation layer and the denoising result is output.

[0124] Optionally, the feature transformation layer can be implemented as a pointwise feedforward network, which performs the same linear transformation and nonlinear activation operations on each element in the input cross-attention feature representation. The pointwise feedforward layer usually contains two linear transformation layers and one nonlinear activation function. Assuming that the dimension of the cross-attention feature representation x is d1, the first linear transformation layer maps the input cross-attention feature representation x to an intermediate feature representation h with a dimension of d2, that is, h = w1×x+b1, where w1 is the weight matrix and b1 is the bias vector. Then, the nonlinear activation function ReLU is applied to the intermediate feature representation h to obtain h ReLU , that is, h ReLU =ReLU(h). Finally, the second linear transformation layer transforms h ReLU The output feature representation y mapped back to dimension d1 is y = w2 × h ReLU +b2, where w2 is the weight matrix and b2 is the bias vector. It should be noted that the above w1, b1, w2 and b2 are all parameters learned by the diffusion model during the training phase.

[0125] In some embodiments, a self-attention layer, a cross-attention layer, and a feature transformation layer may constitute a denoising unit. Optionally, the diffusion model includes at least two denoising units, wherein the object feature representation and the output result of the i-th denoising unit (for example, the above-mentioned output feature representation y) are input into the i+1-th denoising unit, and the output result of the i+1-th denoising unit is output, where i is a positive integer. Then, the output result of the last denoising unit is also the denoising result output by the final diffusion model.

[0126] In the above process, the self-attention layer can capture the correlation between different noise feature elements in the noise feature representation, and explore the internal connection of the noise data itself, which helps to more accurately determine the parts of the noise data that need to be focused on and the mutual influence between the parts; the cross-attention layer associates the object feature representation with the self-attention feature representation, so that the denoising process can fully consider the relationship between the characteristics of the first object itself and the internal correlation of the noise, so that the denoising operation can remove those noise components that do not meet the object feature requirements in a targeted manner according to the characteristics of the first object, further improving the effectiveness of denoising.

[0127] In the above embodiment, the denoising unit uses the object feature representation as the guiding information to perform denoising on the noise data. Since the object feature representation accurately reflects the core information of the object prompt information, it can more effectively identify and remove noise, thereby improving the denoising effect.

[0128] Step 330: input the characteristic parameters into a generator, and generate three-dimensional model data corresponding to the first object through the generator.

[0129] The generator is used to instruct a computer program for automatically generating a three-dimensional model of a first object, wherein the three-dimensional model includes three-dimensional model data; the generator may also be referred to as a procedural content generation (PCG) generator. Schematically, the generator is a program pre-written by a technician, in which generation rules and variables are defined. By changing the values ​​of the variables, different three-dimensional model instances can be generated based on the same generation rules.

[0130] For the variables in the generator, the variables correspond to the model parameters of the three-dimensional model of the first object, that is, the variables in the generator correspond to the performance characteristic parameters. For example: the shape-related variables in the generator correspond to the shape-related parameters; the material texture-related variables in the generator correspond to the material texture-related parameters; the decoration variables in the generator correspond to the decoration parameters, which are not limited here.

[0131] The above-mentioned inputting the performance characteristic parameters into the generator, that is, the performance characteristic parameters are used as variable values ​​and assigned to the variables in the generator, so as to generate the three-dimensional model data corresponding to the first object.

[0132] Optionally, the generator includes n variables, the performance characteristic parameters include n characteristic parameters, and the n characteristic parameters correspond to the n variables one by one. Schematically, the n characteristic parameters are used as variable values ​​and assigned to corresponding variables respectively to generate the three-dimensional model data corresponding to the first object.

[0133] In summary, the method for generating a three-dimensional model provided in the embodiment of the present application performs denoising on noise data under the guidance of object prompt information, thereby obtaining performance feature parameters that match the appearance of the object described in the object prompt information, and inputting the performance feature parameters into a generator to generate three-dimensional model data of the object. In the present application, the process of obtaining the performance feature parameters is modeled as a denoising process for noise data, and performance feature parameters that meet the conditions are quickly generated, which reduces the adjustment process of the performance feature parameters, thereby improving the generation efficiency of the three-dimensional model.

[0134] In some embodiments, in order to simplify the denoising process and improve the denoising efficiency, the processed data in the denoising process can be limited to a preset normalized data range. After the denoising is completed, the normalized data is subjected to an inverse normalization operation to obtain a denoising result. The characteristic parameters in the denoising result are within a range that can reflect their true physical meaning. For illustration, please refer to Figure 4 , the above Figure 3 The illustrated embodiment may also be implemented as steps 410 to 430 described below.

[0135] Step 410: Obtain object prompt information of the first object.

[0136] The object prompt information is used to describe the first object, and the object prompt information includes at least one of image information and text information.

[0137] Optionally, the image information refers to a two-dimensional image containing the first object, and the text information is a prompt text describing the first object in text form, and the text information is also a two-dimensional form of information. That is, in this application, the object prompt information is two-dimensional data used to describe the first object.

[0138] Step 421, using the object prompt information as a denoising condition, performs a denoising process on the noise data to obtain candidate feature parameters.

[0139] The denoising process is performed on the noise data using the object prompt information as the denoising condition. The specific process of obtaining the candidate feature parameters can be referred to step 320 and will not be described in detail here.

[0140] In some embodiments, the noise data is obtained by random sampling from a preset noise distribution.

[0141] Optionally, the noise data is data within a preset normalized data range. Taking the preset normalized data range as [-1, 1] as an example, the noise data needs to be limited to [-1, 1], then the noise value randomly drawn from the preset noise distribution is between [-1, 1].

[0142] After sampling to obtain noise data within a preset normalized data range, the object prompt information is used as a denoising condition, and a denoising process is performed on the noise data within the preset normalized data range to obtain candidate feature parameters.

[0143] The candidate feature parameters refer to feature parameters that characterize the appearance of the first object using data within a preset normalized data range.

[0144] Schematically, the denoising process can be regarded as a process of converting noise data into candidate feature parameters. During the denoising process, the processed data can be limited to a preset normalized data range, thereby obtaining candidate feature parameters within the preset normalized data range.

[0145] Step 422, perform an inverse normalization operation on the candidate feature parameters to obtain performance feature parameters as denoising results.

[0146] The inverse normalization operation refers to the process of converting candidate feature parameters within the preset normalized data range into performance feature parameters within the original data range. The original data range is the value range that can reflect the physical quantity corresponding to the feature parameter. For example, when describing the length of an object (i.e., an object), before normalization, the actual measured length value varies within the range of 0 cm to 100 cm. Then this range of 0 cm to 100 cm is the original data range corresponding to the length data.

[0147] Optionally, the candidate feature parameters include at least one of a continuous parameter and a discrete parameter.

[0148] Continuous parameters are parameters that can take any real value within a certain interval. The values ​​of these parameters are continuous and there are no gaps. For example, when describing the appearance of a three-dimensional object, the length, width, height, surface curvature, etc. of the object can all be continuous parameters. Discrete parameters are parameters whose values ​​are discontinuous and can only take specific values. For example, when describing the appearance of an object, the texture pattern category and color category of the object surface can all be discrete parameters. Taking the color category of the object surface as an example, if there are only three color options: red, green, and blue, then the value of this color parameter can only be these three specific values.

[0149] For continuous parameters, the preset normalized data range corresponds to a first mapping rule, and the first mapping rule can be implemented as at least one of a minimum-maximum normalization rule, a standard deviation normalization rule, etc., which is not limited here. According to the first mapping rule, an inverse normalization operation is performed on the continuous parameter to obtain the inverse normalized continuity parameter as the performance feature parameter.

[0150] Taking the first mapping rule as an example, the minimum-maximum normalization rule that normalizes the data to [-1, 1] (that is, the preset normalized data range) is implemented as follows: let the original continuity parameter be s, and its minimum value be s min , the maximum value is s max , the normalized data s norm The calculation formula is: This formula can normalize the original continuity parameter s to the range of [-1, 1]. Correspondingly, the inverse normalization operation is: Given the normalized value s norm (can be realized as the above candidate feature parameters), the original continuity parameter to be obtained Among them, s max and min It can be realized as a preset maximum and minimum value, and the original continuity parameter s is also the continuity parameter after inverse normalization.

[0151] For discrete parameters, the preset normalized data range corresponds to a second mapping rule, and the second mapping rule can be implemented as at least one of a uniform segmentation mapping rule, etc., which is not limited here. According to the second mapping rule, an inverse normalization operation is performed on the discrete parameter to obtain the inverse normalized discrete parameter as the performance feature parameter.

[0152] Taking the second mapping rule as an example of a uniform partition mapping rule that normalizes the data to [-1, 1] (that is, the preset normalized data range), the normalization process is as follows: determine the number of uniform partitions of [-1, 1] according to the number of discrete parameter values. For example, if the discrete parameter has 5 different values, then the interval [-1, 1] needs to be evenly partitioned into 5 parts. Starting from the minimum value of the discrete parameter, it is matched with the partitioned intervals in turn. For example, the first value corresponds to the interval [-1, -1+0.4) (here, left closed and right open is just an example, and the boundary opening and closing rules can also be set according to the actual situation), the second value corresponds to [-1+0.4, -1+0.4×2), and so on. In this way, each value of the discrete parameter is mapped to a subinterval in the interval [-1, 1], thereby realizing the normalization of the discrete parameter. Correspondingly, the inverse normalization operation is: given a normalized value (which can be realized as the above-mentioned candidate feature parameter), determine the subinterval where the normalized value is located, and determine the original value of the corresponding discrete parameter according to the corresponding relationship established during the normalization process. The original value is the discrete parameter after inverse normalization. For example; if a normalized value falls within the subinterval [-1+0.4, -1+0.4×2), then the original value of the corresponding discrete parameter is the second value of the discrete parameter.

[0153] Step 430: input the performance characteristic parameters into a generator, and generate three-dimensional model data corresponding to the first object through the generator.

[0154] The generator is used to instruct a computer program to automatically generate a three-dimensional model of the first object.

[0155] Optionally, the generator includes n variables, the performance characteristic parameters include n characteristic parameters, and the n characteristic parameters correspond to the n variables one by one. Schematically, the n characteristic parameters are used as variable values ​​and assigned to corresponding variables respectively to generate the three-dimensional model data corresponding to the first object.

[0156] Optionally, the three-dimensional model data can be implemented as at least one of the following data types: (1) Point cloud. A point cloud is a set of vector data in a three-dimensional coordinate system. These data points represent the surface features of a three-dimensional object. Point cloud data can record the geometric shape of the object very accurately. (2) Polygonal mesh. A polygonal mesh is composed of a series of vertices, edges, and faces, such as a triangular mesh. It divides the surface of a three-dimensional model into many triangles. These triangles construct the shape of the object by sharing vertices and edges. (3) Voxel model. A voxel is similar to a pixel in a two-dimensional image, but it is a volume element in a three-dimensional space. The voxel model divides the three-dimensional space into small cubic units (voxels) and represents a three-dimensional object by describing the combination and state of these voxels.

[0157] In some embodiments, a parameter editing operation is performed on the performance characteristic parameters to obtain edited performance characteristic parameters; wherein the parameter editing operation includes at least one of parameter modification, parameter deletion, and parameter addition; the edited performance characteristic parameters are input into a generator, and three-dimensional model data corresponding to the first object is generated through the generator.

[0158] Parameter modification refers to changing the value indicated by the performance characteristic parameter. For example, length is a continuous performance characteristic parameter. The original length is 5 meters. Through parameter modification, it can be changed to 6 meters, thus affecting the final shape of the object. The object surface color category is a discrete performance characteristic parameter (assuming the original value is red). It can be modified to blue, which will change the color display of the object surface.

[0159] Parameter deletion refers to deleting certain characteristic parameters in the characteristic parameters. For example, if the parameter representing the height of an object is deleted, the default height value (if there is a relevant default setting) can be used when generating a 3D model. If the parameter of the decorative pattern category on the surface of an object is deleted, the generated 3D model will not take the pattern into account, and the model surface may appear without a decorative pattern or present other default decorative images.

[0160] Parameter addition is to add new characteristic parameters to the performance characteristic parameters. For example, adding a new surface material category parameter can enrich the appearance properties of the model and enable it to present a unique material effect.

[0161] In the above embodiment, by editing the generated performance characteristic parameters, the performance characteristic parameters can be flexibly adjusted according to specific needs. For example, by adjusting the parameters to meet the requirements of a specific rendering software for the model data format and characteristics, the versatility and compatibility of the performance characteristic parameters in multi-platform and multi-software environments are improved, facilitating the wide application of the performance characteristic parameters.

[0162] For illustration, please refer to Figure 5 , which shows a schematic diagram of the generated result after editing, such as Figure 5 As shown, image 510 is a two-dimensional image containing a chair object. The original performance characteristic parameters corresponding to the chair object can be generated by the three-dimensional model generation method described above; the original performance characteristic parameters (i.e., the unedited performance characteristic parameters) are input into the generator to generate model 521, i.e., the three-dimensional model of the chair. The technician can edit the original performance characteristic parameters as required, and input the edited parameters into the generator to generate the edited model, for example Figure 5 Model 522 to Model 527 are shown:

[0163] Model 522: The original characteristic parameters corresponding to the chair object include an armrest type parameter, and the armrest type parameter can be set to "0". "0" means no armrest. Then, the three-dimensional chair model in the generated model 522 is presented as having no armrest.

[0164] Model 523: The armrest type parameter can be set to "1", "1" means short armrests, and the generated model 523 chair 3D model will be in a state of short armrests.

[0165] Model 524: The original characteristic parameters corresponding to the chair object include a chair leg type parameter, and the chair leg type parameter can be set to "1". "1" represents a thick chair leg. Then the three-dimensional chair model in the generated model 524 appears to be in a state of a thick chair leg.

[0166] Model 525: The chair leg type parameter can be set to "2", where "2" represents a long bench leg. Then the three-dimensional model of the chair in the generated model 525 appears to be a long bench leg.

[0167] Model 526: The original characteristic parameters corresponding to the chair object include a chair back type parameter, which can be set to "1". "1" represents the entire chair back. The three-dimensional chair model in the generated model 526 then presents the entire chair back.

[0168] Model 527: The backrest type parameter can be set to "2", where "2" represents a wide backrest. The three-dimensional model of the chair in the generated model 527 will then be presented as a wide backrest.

[0169] In some embodiments, the object prompt information is used to describe the appearance of the first object and the dynamic effect of the first object, and the denoising result also includes effect feature parameters corresponding to the dynamic effect of the first object.

[0170] Optionally, the generator includes a first variable and a second variable, and uses the object prompt information as a denoising condition to perform a denoising process on the noise data to obtain performance characteristic parameters and effect characteristic parameters as denoising results, wherein the performance characteristic parameters included in the denoising result correspond to the first variable, and the effect characteristic parameters included in the denoising result correspond to the second variable.

[0171] Optionally, the number of first variables may be one or more (i.e., the n variables mentioned above). When there are multiple first variables, the performance characteristic parameter includes multiple characteristic parameters, and the multiple characteristic parameters correspond one-to-one with the multiple first variables. Optionally, the number of second variables may be one or more. When there are multiple second variables, the effect characteristic parameter includes multiple characteristic parameters, and the multiple characteristic parameters correspond one-to-one with the multiple second variables.

[0172] Optionally, the performance characteristic parameters and the effect characteristic parameters are input into a generator, and animation data corresponding to the first object is generated by the generator, the animation data including three-dimensional model data, and the animation data is used to characterize the dynamic performance of the first object in three-dimensional space.

[0173] Illustratively, the generator is used to generate three-dimensional animation data of a first object, a first variable in the generator is used to construct a three-dimensional model of the first object, and a second variable is used to construct a dynamic effect of the three-dimensional model of the first object.

[0174] Optionally, the performance characteristic parameter is assigned to the first variable in the generator, and the effect characteristic parameter is assigned to the second variable in the generator, so as to generate animation data corresponding to the first object. Schematically, within the generator, a three-dimensional model is constructed based on the performance characteristic parameter assigned to the first variable; a dynamic effect is given to the constructed three-dimensional model based on the effect characteristic parameter assigned to the second variable. The dynamic effect includes various parameters used to describe how the first object moves and changes in three-dimensional space, such as: the type of movement (such as basic movement types such as translation, rotation, and scaling, or more complex combined movement forms, such as rotating while translating along a specific trajectory, etc.), the speed and acceleration of the movement (it can be uniform motion or variable speed motion according to a specific function, and the acceleration determines the speed of the speed change), the trajectory of the movement (such as a straight trajectory, a curved trajectory), etc., which are not limited here.

[0175] In the above embodiment, the object prompt information covers both the appearance and dynamic effects of the first object, and the animation data finally generated can simultaneously reflect the three-dimensional model data and the corresponding dynamic performance, so that the portrayal of the first object is no longer limited to the static appearance, but can fully display how it moves and changes in three-dimensional space, and can more realistically and comprehensively restore the characteristics that the first object should have.

[0176] To summarize, the three-dimensional model generation method provided in the embodiment of the present application limits the processed data in the denoising process to a preset normalized data range. After the denoising is completed, an inverse normalization operation is performed on the obtained normalized data to obtain a denoising result within the original data range. Performing the denoising operation on the normalized data helps to more accurately identify and remove noise components, improve the efficiency of the entire denoising process, and reduce the consumption of computing resources.

[0177] The following is an introduction to the training process of the diffusion model involved in the above. Optionally, the training process of the diffusion model includes the following steps:

[0178] Step 1: Obtain sample prompt information and sample performance characteristic parameters.

[0179] The sample prompt information includes at least one of a sample image and a sample text. The sample image is a two-dimensional image containing a first object, and the sample text is used to describe the first object in text form.

[0180] (1) For sample images.

[0181] After obtaining a generator for generating three-dimensional model data of a first object, a variable combination is randomly sampled for each variable in the generator, and the variable combination is input into the generator to obtain sample three-dimensional model data, wherein the variable combination can be used as a sample performance characteristic parameter.

[0182] A sample three-dimensional model of the first object is constructed according to the sample three-dimensional model data, and the sample three-dimensional model is rendered to obtain a sample rendered image, which can be used as a sample image.

[0183] In some embodiments, the sample three-dimensional model is rendered at a sample perspective to obtain a sample rendered image as a sample image, and the sample rendered image is used to present the rendering result of the sample three-dimensional model at the sample perspective (for example, the sample rendered image can be implemented as an isometric view of the sample three-dimensional model). Subsequently, the sample diffusion model described below can be trained based on the sample image corresponding to the sample perspective, so that the model can focus on learning the feature information presented by the three-dimensional model at this specific perspective, thereby obtaining a trained diffusion model. The diffusion model is trained based on the sample image at the same sample perspective, and the model can be familiar with and utilize the inherent consistency characteristics of the image data at this perspective. For example, in an isometric view, the relative positions and proportional relationships of the various parts of an object have specific presentation rules. After the model masters these rules during the learning process, when denoising the noise data, it can restore the real information that conforms to the perspective characteristics based on these perspective characteristics, effectively reducing the uncertainty caused by perspective differences, thereby improving the accuracy of denoising.

[0184] In some embodiments, after obtaining the sample image, the sample image can also be enhanced to expand the sample image used for training, wherein the image enhancement processing includes at least one of: color transformation, random cropping, image flipping, binarization operation, etc., which is not limited here.

[0185] (2) For sample text.

[0186] After obtaining a generator for generating three-dimensional model data of a first object, a variable combination is randomly sampled for each variable in the generator, and the variable combination is input into the generator to obtain sample three-dimensional model data, wherein the variable combination can be used as a sample performance characteristic parameter.

[0187] The three-dimensional model data is input into a text generation model, and a sample text is output, where the sample text is used to describe the first object indicated by the three-dimensional model data in text form.

[0188] Among them, the text generation model is trained with a large amount of text data to learn the language patterns, semantic relationships, and grammatical rules in the text. For example, in the pre-training stage of unsupervised learning, the model will be exposed to a large amount of text materials with different themes and styles to learn the common rules; then in the supervised learning stage, according to the task of describing the 3D model data, the model parameters will be further optimized using the labeled related data (such as the accurate description text corresponding to the known 3D model as the label), so that it can accurately output the sample text that meets the requirements based on the input 3D model data.

[0189] Step 2: Obtain the sample diffusion model.

[0190] Optionally, the sample diffusion model includes a sample encoder, a sample denoising unit and a sample denoising unit.

[0191] The sample encoder is used to extract the sample feature representation corresponding to the sample prompt information. Schematically, if the sample prompt information refers to image information, the sample encoder is an image encoder, the image information is input into the image encoder, and the image feature representation is output as the sample feature representation; if the sample prompt information refers to text information, the sample encoder is a text encoder, the text information is input into the text encoder, and the text feature representation is output as the sample feature representation; if the sample prompt information includes image information and text information, the sample encoder is a multimodal encoder, the image information and the text information are input into the multimodal encoder, the image feature representation corresponding to the image information is extracted through the multimodal encoder, and the text feature representation corresponding to the text information is extracted, and the image feature representation and the text feature representation are fused to obtain the sample feature representation. Optionally, the sample encoder can be implemented as an encoder that has been trained, that is, the model parameters of the sample encoder do not need to be updated during the training of the sample diffusion model; or the sample encoder is an encoder to be trained, that is, the model parameters of the sample encoder need to be updated during the training of the sample diffusion model.

[0192] The sample denoising unit is used to add noise to the sample performance characteristic parameters. The sample denoising unit is used to remove the noise added in the sample performance characteristic parameters. Optionally, the sample denoising unit includes a sample self-attention layer, a sample cross-attention layer and a sample feature conversion layer. Optionally, the number of sample denoising units included in the diffusion model can be one or more. When the number of sample denoising units is multiple, the multiple sample denoising units can process the sample performance characteristic parameters after adding noise in a certain order. Each time a sample denoising unit passes through, a part of the noise can be removed, and the sample performance characteristic parameters before denoising can be gradually restored.

[0193] Step 3: Define M time steps. In the M time steps, gradually add noise to the sample performance characteristic parameters through the sample denoising unit to obtain the sample noise corresponding to the M time steps.

[0194] M is an integer greater than 1.

[0195] Schematically, at each time step, Gaussian noise is added to the sample performance characteristic parameters, and the noise level increases with the increase of time steps until the sample performance characteristic parameters completely become pure noise.

[0196] In each time step, the sample performance characteristic parameter after adding noise is the sample noise corresponding to the time step. For example, in the tth time step, noise t is added to the sample performance characteristic parameter, and the sample performance characteristic parameter after adding noise t is the sample noise corresponding to the tth time step, where t is a positive integer, t≤M.

[0197] Step 4: For the t-th time step, the sample feature corresponding to the sample prompt information is expressed as the denoising condition, and the sample noise corresponding to the t-th time step is denoised by the sample denoising unit to obtain the t-th sample denoising result.

[0198] The sample prompt information is input into the sample encoder and the sample feature representation is output.

[0199] The noise feature representation corresponding to the sample noise corresponding to the t-th time step is input into the sample self-attention layer, and the sample self-attention feature representation is output; the sample feature representation and the sample self-attention feature representation are input into the sample cross-attention layer, and the sample cross-attention feature representation is output; the sample cross-attention feature representation is input into the sample feature transformation layer, and the t-th sample denoising result is output.

[0200] Optionally, the diffusion model includes at least two sample denoising units, wherein the sample feature representation and the output result of the i-th denoising unit are input into the i+1-th denoising unit, and the output result of the i+1-th denoising unit is output, where i is a positive integer. Then, the output result of the last denoising unit is also the t-th sample denoising result.

[0201] The t-th sample denoising result includes the t-th predicted performance feature parameter corresponding to the appearance performance of the first object.

[0202] Step 5: Based on the difference between the t-th predicted performance feature parameter and the sample performance feature parameter, determine the target loss corresponding to the t-th time step.

[0203] Optionally, the function used to calculate the target loss includes but is not limited to: a mean square error loss function, a mean absolute error loss function, and a cross entropy loss function.

[0204] Step 6: Train the sample diffusion model based on the target loss corresponding to each of the T time steps to obtain the diffusion model.

[0205] Optionally, the model parameters of the sample diffusion model are adjusted based on the target loss of each time step to obtain the diffusion model; or, the target losses corresponding to T time steps are fused to obtain the total loss, and the model parameters of the sample diffusion model are adjusted based on the total loss to obtain the diffusion model.

[0206] Among them, the target losses corresponding to the fusion T time steps can be realized as follows: weighted sum of the target losses corresponding to the T time steps to obtain the total loss.

[0207] Optionally, the model parameters to be adjusted in the sample diffusion model include parameters of a denoising unit; optionally, the model parameters to be adjusted in the sample diffusion model may also include parameters of a sample encoder.

[0208] Optionally, when the trained model meets the training requirements, the training is stopped. The model obtained at this time is the diffusion model described above. Optionally, the training requirements refer to: the target loss output by the model is less than or equal to the preset loss value, or the model training times reaches a preset number of times.

[0209] In some embodiments, a normalization operation is performed on the sample performance characteristic parameters so that the performance characteristic parameters are within a preset normalized data range, that is, the sample diffusion model is trained using the normalized data.

[0210] Optionally, the sample characteristic parameters include at least one of continuous parameters and discrete parameters.

[0211] For the continuous parameter, a normalization operation is performed on the continuous parameter according to the first mapping rule to obtain a normalized sample feature parameter. The first mapping rule can be implemented as at least one of a minimum-maximum normalization rule, a standard deviation normalization rule, and the like.

[0212] For discrete parameters, a normalization operation is performed on the discrete parameters according to the second mapping rule to obtain normalized sample characteristic parameters. The second mapping rule can be implemented as at least one of a uniform segmentation mapping rule and the like, which is not limited here.

[0213] Optionally, after obtaining the normalized sample feature parameters, the normalized sample feature parameters are used as the sample feature parameters described in the above step. Then, the t-th predicted performance feature parameter included in the t-th sample denoising result obtained in the above step 5 is within the preset normalized data range. Therefore, before calculating the target loss, the t-th predicted performance feature parameter can also be subjected to an inverse normalization operation to obtain the inverse normalized t-th predicted performance feature parameter. Then step 6 is: based on the difference between the inverse normalized t-th predicted performance feature parameter and the sample performance feature parameter, determine the target loss corresponding to the t-th time step. Alternatively, step 6 is implemented as: based on the difference between the t-th predicted performance feature parameter and the normalized sample performance feature parameter, determine the target loss corresponding to the t-th time step.

[0214] The specific implementation details of the inverse normalization operation have been described in step 422 and will not be repeated here.

[0215] In some embodiments, the object prompt information refers to image information including the first object. For example, please refer to Figure 6 , the above Figure 3 or Figure 4 The illustrated embodiment may also be implemented as steps 610 to 630 described below.

[0216] Step 610: Acquire image information of a first object.

[0217] The image information refers to a two-dimensional image containing the first object. The form of the image information includes, but is not limited to: a sketch (line drawing), a real picture (such as a photograph), a design drawing, a cartoon picture, an oil painting, etc.

[0218] Step 621, extracting image feature representation corresponding to the image information.

[0219] The image feature represents a visual feature used to characterize the first object.

[0220] Optionally, an image feature representation corresponding to the image information is extracted through an image encoder in the diffusion model.

[0221] Optionally, the image feature representation includes at least one of a color feature representation, a texture feature representation, a shape feature representation, a spatial relationship feature representation, etc. The color feature representation is used to represent the color information of the first object contained in the image information; the texture feature representation is used to represent the texture information of the first object contained in the image information; the shape feature representation is used to represent the shape information of the first object contained in the image information (e.g., the outline or shape of the first object); the spatial relationship feature representation is used to represent the spatial relationship information of the first object contained in the image information (e.g., the spatial position relationship between the first object and other objects or the background).

[0222] Among them, the image encoder can be implemented as at least one of an encoder based on a convolutional neural network, an encoder based on a visual Transformer, etc.

[0223] In this example, the encoder of the visual transformer is used to divide the image information into a series of image patches, which are converted into vector representations and then input into the encoder of the visual transformer. The image feature representation is obtained through the self-attention mechanism and multi-layer encoding of the encoder of the visual transformer.

[0224] Step 622, performing a denoising process on the noise data based on the image feature representation to obtain a denoising result.

[0225] Optionally, the image feature is represented as a denoising condition, and a denoising process is performed on the noise data based on a denoising unit in a diffusion model to obtain a denoising result.

[0226] In some embodiments, the denoising unit includes a cross self-attention layer, a self-attention layer, and a feature transformation layer.

[0227] Optionally, the noise feature representation corresponding to the noise data is input into the self-attention layer in the diffusion model, and the self-attention feature representation is output; the image feature representation and the self-attention feature representation are input into the cross-attention layer in the diffusion model, and the cross-attention feature representation is output; the cross-attention feature representation is input into the feature transformation layer in the diffusion model, and the denoising result is output.

[0228] The denoising result includes a performance characteristic parameter corresponding to the appearance performance of the first object.

[0229] In some embodiments, a denoising process is performed on noise data based on image feature representation to obtain candidate feature parameters, where the candidate feature parameters refer to feature parameters that characterize the appearance of the first object using data within a preset normalized data range; an inverse normalization operation is performed on the candidate feature parameters to obtain performance feature parameters as denoising results.

[0230] Step 630: input the performance characteristic parameters into a generator, and generate three-dimensional model data corresponding to the first object through the generator.

[0231] Optionally, the generator includes n variables, the performance characteristic parameters include n characteristic parameters, and the n characteristic parameters correspond to the n variables one by one. Schematically, the n characteristic parameters are used as variable values ​​and assigned to corresponding variables respectively to generate the three-dimensional model data corresponding to the first object.

[0232] For illustration, please refer to Figure 7 , which shows a schematic diagram of a method for generating a three-dimensional model, such as Figure 7 As shown, the method for generating a three-dimensional model provided in an embodiment of the present application is executed through a diffusion model, and the diffusion model includes an image encoder 701 and R denoising units, where R is an integer greater than 1.

[0233] The image 702 containing the chair object is input into the image encoder 701 to obtain the image feature representation (in vector form); the noise data (in vector form, which can be regarded as the noise feature representation described above) 703 and the image feature representation are input into R denoising units, each of which includes a multi-head self-attention layer 704, a multi-head cross-attention layer 705 and a point-by-point feedforward layer 706, wherein the image feature data is input into the multi-head cross-attention layer 704 in each denoising unit.

[0234] Taking the i+1th denoising unit as an example: the output result of the i+1th denoising unit is input into the multi-head self-attention layer 704, and the self-attention feature representation is output; the image feature representation and the self-attention feature representation are input into the multi-head cross-attention layer 705, and the cross-attention feature representation is output; the cross-attention feature representation is input into the point-by-point feedforward layer 706 to obtain the output result of the i+1th denoising unit.

[0235] Then, the output result of the last denoising unit is Figure 7 The denoising result 707 is shown, wherein the denoising result 707 is data within a preset normalized data range, and an inverse normalization operation is performed on the denoising result 707 to obtain a performance characteristic parameter 708 as the final denoising result.

[0236] Finally, the performance characteristic parameters 708 are input into the generator 709 to generate a three-dimensional model 710 corresponding to the chair.

[0237] In some embodiments, the image information is used to display the first object at a specified viewing angle. The viewing angle refers to the angle at which the first object is viewed. The appearance and features of the object seen at different viewing angles are different.

[0238] Optionally, the three-dimensional model data corresponding to the first object is rendered based on a specified viewing angle to obtain a rendered image, and the rendered image is used to present a rendering result of the three-dimensional model data at the specified viewing angle; based on the difference between the image information and the rendered image, the performance feature parameters are adjusted.

[0239] In an illustrative manner, specific parameters of the viewing angle are clearly specified, including parameters such as azimuth (horizontal angle), pitch angle (vertical angle), etc. The three-dimensional model data is rendered using the parameters such as azimuth and pitch angle to obtain a rendered image, which is a two-dimensional image containing the first object.

[0240] Optionally, after rendering to obtain a rendered image, determine the similarity between the image information and the rendered image; when the similarity is less than or equal to a preset similarity, adjust the expression feature parameters according to the difference in expression features between the first object contained in the image information and the first object contained in the rendered image.

[0241] The above method for determining the similarity between the image information and the rendered image can be implemented as a pixel-based similarity calculation method: calculate the average of the squares of the difference values ​​of the corresponding pixels of the two images, the smaller the average value, the more similar the two images are, and theoretically the two images are exactly the same when the average value is 0; determine the similarity between the image information and the rendered image based on the average value, wherein the average value and the similarity are negatively correlated. Alternatively, the above method for determining the similarity between the image information and the rendered image can be implemented as a feature-based similarity calculation method: extract the feature vectors of the two images (image features can be extracted by methods such as convolutional neural networks), and then use cosine similarity to measure the cosine value of the angle between the two feature vectors, the closer the cosine value of the angle is to 1, the more similar the two images are; determine the similarity between the image information and the rendered image based on the cosine value of the angle, wherein the cosine value of the angle is positively correlated with the similarity.

[0242] If the calculated similarity is less than or equal to the preset similarity, it means that there are certain differences in the appearance representation of the first object presented by the currently generated three-dimensional model data and the first object displayed by the image information. In order to improve the accuracy of the finally generated three-dimensional model data, the current performance feature parameters may be adjusted according to the differences in performance features between the first object contained in the image information and the first object contained in the rendered image.

[0243] Optionally, based on the difference in expression characteristics between the first object contained in the image information and the first object contained in the rendered image, the method for adjusting the current expression characteristic parameters includes: extracting a first image feature representation of the first object contained in the image information; extracting a second image feature representation of the first object contained in the rendered image; and adjusting the current expression characteristic parameters with the goal of reducing the difference between the first image feature representation and the second image feature representation.

[0244] Optionally, the first image feature representation includes at least one of color feature representation, texture feature representation, shape feature representation, spatial relationship feature representation, etc. Correspondingly, the second image feature representation includes at least one of color feature representation, texture feature representation, shape feature representation, spatial relationship feature representation, etc.

[0245] Take the first image feature representation and the second image feature representation as color feature representation as an example for explanation. Extract the first color feature representation of the first object in the image information, such as the representation in the color space such as RGB (Red-Green-Blue) value and HSV (Hue-Saturation-Value) value. Similarly, extract the second color feature representation of the first object in the rendered image. The first color feature representation is consistent with the second color feature representation in type and dimension. Calculate the difference between the first color feature representation and the second color feature representation to obtain the color difference. According to the calculated color difference, determine the performance feature parameters that need to be adjusted (such as brightness, contrast, saturation, etc.) and the direction of adjustment (increase or decrease or change type).

[0246] Take the first image feature representation and the second image feature representation as an example to illustrate. Extract the first shape feature representation of the first object in the image information, for example, determine the contour lines of the first object through edge detection algorithms, and analyze the geometric shape characteristics formed by the contour. Similarly, extract the second shape feature representation of the first object in the rendered image. The first shape feature representation and the second shape feature representation are consistent in type and dimension. Compare the first shape feature representation and the second shape feature representation of the two to determine the shape difference, for example: a local structure presents a sharper angle in the image information, but is relatively rounded in the rendered image. According to the calculated shape difference, determine the performance feature parameters that need to be adjusted (such as angle, length, height, etc.) and the direction of adjustment (increase or decrease or change type).

[0247] In some embodiments, the parameter generation, image comparison and parameter adjustment processes involved above are performed multiple times. Through continuous iterative optimization, the difference between the rendered image and the image information is gradually reduced, thereby improving the accuracy of the three-dimensional model data.

[0248] Similarly, for other types of image feature representations such as texture feature representation, spatial relationship feature representation, etc., corresponding operations can also be performed according to the logic of their corresponding feature extraction, difference analysis and parameter adjustment, which will not be elaborated here.

[0249] In the above scheme, the performance characteristic parameters are adjusted based on the difference between the image information and the rendered image, so that the image rendered by the finally generated 3D model data is closer to the first object under the specified viewing angle that is actually desired to be presented. This helps to continuously optimize the 3D model so that it can restore the real object more accurately and realistically in terms of performance characteristics such as appearance and form, thereby improving the generation quality of the 3D model and meeting related requirements such as accurate 3D presentation of the object, such as building accurate scene elements in virtual reality and game development.

[0250] In summary, the method for generating a three-dimensional model provided in the embodiment of the present application performs denoising on noise data under the guidance of image information containing the first object, thereby obtaining performance feature parameters that match the appearance of the object presented in the image information, and inputting the performance feature parameters into a generator to generate three-dimensional model data of the object. In the present application, the process of obtaining the performance feature parameters is modeled as a denoising process for noise data, and performance feature parameters that meet the conditions are quickly generated, which reduces the adjustment process of the performance feature parameters, thereby improving the generation efficiency of the three-dimensional model.

[0251] The method for generating a three-dimensional model provided in this application realizes efficient and accurate reverse procedural generation (i.e., reverse PCG, for example: reversing the model parameters of a three-dimensional model from a two-dimensional image), thereby realizing high-quality three-dimensional resource generation. Figure 8 and Fig. 9 Compare the generated results of the method provided by this application with other baseline methods.

[0252] Table 1 shows the quantitative test results of the generation results of this application and other baseline methods on the chair category.

[0253] Table 1

[0254]

[0255] As shown in Table 1, the indicators of quantitative testing include the following:

[0256] CD (Chamfer Distance): is a metric used to measure the difference between two point cloud sets. CD can be used to quantify the degree of geometric difference between the generated chair shape (represented by point cloud) and the reference shape (represented by point cloud). The smaller the CD value, the more similar the generated chair shape is to the reference shape in terms of geometry, which means the generated 3D model is of higher quality.

[0257] EMD (Earth Mover's Distance): Also known as Wasserstein distance, EMD is used to evaluate the similarity between the generated chair shape and the reference shape in distribution (e.g., point cloud distribution). A lower EMD value means that the generated chair shape is closer to the reference shape in terms of overall structure and point distribution, i.e., the generated 3D model is of higher quality.

[0258] F-Score (F value): is a comprehensive measure of precision and recall. Precision can be understood as the proportion of generated chair shapes that are judged to be correct (for example, they meet certain geometric and structural standards) to all generated chair shapes; recall refers to the proportion of generated correct chair shapes to chairs in the reference dataset (chair dataset). F-Score provides a comprehensive evaluation metric for balancing the accuracy and completeness of generated shapes. A high F-Score indicates that the generated chair shapes perform well in both correct generation (precision) and generating as many correct shapes as possible (recall).

[0259] From the data shown in Table 1, it can be seen that the three-dimensional model generation method provided by the present application is superior to other baseline methods in all test indicators.

[0260] Figure 8 The figure shows the comparison of the generated results of this application and other baseline methods in various categories. Figure 8As shown, three-dimensional models are generated based on various types of two-dimensional images 801 using baseline methods 1-4 and the three-dimensional model generation method provided in the present application, wherein the three-dimensional model 806 generated by the present application has a significantly better match with the two-dimensional image 801 in terms of appearance than the three-dimensional models 802-805 generated by baseline methods 1-4.

[0261] Fig. 9 FIG. 1 shows a schematic diagram of the generation result based on the sketch of the present application. Fig. 9 As shown, the three-dimensional model 902 generated based on the sketch 901 of various objects using the three-dimensional model generation method provided by the present application has high accuracy and better generation effect.

[0262] Next, the application scenarios of the three-dimensional model generation method provided in the embodiments of the present application are introduced.

[0263] The method for generating a three-dimensional model provided in this application can be applied to various application scenarios such as three-dimensional modeling, application development, animation production, architectural design, etc. The following is an explanation using three-dimensional modeling scenarios and application development scenarios as examples.

[0264] (I) 3D modeling scene

[0265] Step 1: Display the application interface of the 3D modeling application.

[0266] The application interface includes a viewing area and content generation tools.

[0267] The view area is a space for displaying visualization content such as 3D models. In subsequent steps, if the first target 3D model is generated, the first target 3D model will be presented in this area, so that users can intuitively view the morphology, structure and other details of the first target 3D model.

[0268] The content generation tool is a plug-in tool generated based on the three-dimensional model generation method provided in the embodiment of the present application. The content generation tool is used to generate a three-dimensional model based on a two-dimensional image.

[0269] Step 2: Based on the content generation tool, receiving a configuration operation on the first target image.

[0270] The first target image is a two-dimensional image containing a first target object.

[0271] Schematically, the content generation tool can be implemented as a content generation control. After the user clicks the content generation control, an image configuration area is displayed. The user can configure a two-dimensional image containing the first target object, that is, the first target image, in the image configuration area through drawing operations, uploading operations, etc.

[0272] Step 3: In response to the configuration operation, display the first target three-dimensional model corresponding to the first target image in the viewing area.

[0273] The first target three-dimensional model refers to a three-dimensional model corresponding to the first target object.

[0274] The configuration operation is used to generate a first target three-dimensional model based on the first target image. The method for generating the first target three-dimensional model (as the above-mentioned three-dimensional model data) based on the first target image (as the above-mentioned object prompt information) can use any of the above-mentioned three-dimensional model generation methods.

[0275] Schematically, a first target three-dimensional model is generated based on a first target image through a diffusion model, and the diffusion model includes an image encoder and a denoising unit; the first target image is input into the image encoder to obtain an image feature representation; the noise data and the image feature representation are input into the denoising unit, and the denoising unit includes a multi-head self-attention layer, a multi-head cross-attention layer, and a point-by-point feedforward layer: the noise data is input into the multi-head self-attention layer, and the self-attention feature representation is output; the image feature representation and the self-attention feature representation are input into the multi-head cross-attention layer, and the cross-attention feature representation is output; the cross-attention feature representation is input into the point-by-point feedforward layer to obtain candidate feature parameters. The candidate feature parameters are data within a preset normalized data range, and an inverse normalization operation is performed on the candidate feature parameters to obtain performance feature parameters; the performance feature parameters are input into the generator to generate a three-dimensional model corresponding to the first object, that is, the first target three-dimensional model.

[0276] (II) Application Development Scenarios

[0277] Step 1: Display the resource management interface of the application development software.

[0278] The resource management interface includes resource import controls and resource lists.

[0279] The resource import control is used to import various external resources into the application development project. For example, developers can use the resource import control to import pictures, audio, models and other files, and import them into the application development environment, so that they can be used in the subsequent application production.

[0280] The resource list is an area for centralized display and management of various resources that have been imported or built into the application development software. Developers can use the resource list to understand the basic information of each resource, such as name, format, type, etc., to facilitate quick search, selection, editing, deletion and other further operations on resources.

[0281] Step 2: Receive an import operation on the second target image on the resource import control.

[0282] The second target image is a two-dimensional image containing a second target object.

[0283] Illustratively, after the user clicks the resource import control, an import area is displayed, and the user can upload a two-dimensional image containing the second target object, that is, the second target image, in the import area.

[0284] Step 3: In response to the import operation, the second target three-dimensional model corresponding to the second target image is added to the resource list of the resource management interface.

[0285] The import operation is used to generate a second target three-dimensional model based on the second target image. The method for generating a second target three-dimensional model (as the above-mentioned three-dimensional model data) based on the second target image (as the above-mentioned object prompt information) can use any of the above-mentioned three-dimensional model generation methods.

[0286] Schematically, a second target three-dimensional model is generated based on a second target image through a diffusion model, and the diffusion model includes an image encoder and a denoising unit; the second target image is input into the image encoder to obtain an image feature representation; the noise data and the image feature representation are input into the denoising unit, and the denoising unit includes a multi-head self-attention layer, a multi-head cross-attention layer, and a point-by-point feedforward layer: the noise data is input into the multi-head self-attention layer, and the self-attention feature representation is output; the image feature representation and the self-attention feature representation are input into the multi-head cross-attention layer, and the cross-attention feature representation is output; the cross-attention feature representation is input into the point-by-point feedforward layer to obtain candidate feature parameters. The candidate feature parameters are data within a preset normalized data range, and an inverse normalization operation is performed on the candidate feature parameters to obtain performance feature parameters; the performance feature parameters are input into the generator to generate a three-dimensional model corresponding to the first object, that is, the second target three-dimensional model.

[0287] For illustration, please refer to Fig.10 , which shows a structural block diagram of a three-dimensional model generation device, such as Fig.10 As shown, the device comprises:

[0288] An acquisition module 1010 is used to acquire object prompt information of a first object, where the object prompt information is used to describe the first object, and the object prompt information includes at least one of image information and text information;

[0289] A denoising module 1020, configured to perform a denoising process on the noise data using the object prompt information as a denoising condition to obtain a denoising result, wherein the denoising result includes a performance feature parameter corresponding to the appearance performance of the first object;

[0290] The generating module 1030 is used to input the performance characteristic parameters into a generator, and generate three-dimensional model data corresponding to the first object through the generator.

[0291] In some embodiments, the denoising module 1020 is used to:

[0292] Inputting the object prompt information and the noise data into a diffusion model, and outputting the denoising result;

[0293] The diffusion model is a machine learning model that learns the correlation between input prompt information and performance feature parameters during the training phase.

[0294] In some embodiments, the diffusion model includes an object encoder and a denoising unit; the denoising module 1020 is used to:

[0295] Inputting the object prompt information into the object encoder, and outputting an object feature representation corresponding to the object prompt information; wherein, when the object prompt information includes the image information, the object feature representation is used to represent the visual features of the first object, and when the object prompt information includes the text information, the object feature representation is used to represent the semantics of the text information;

[0296] The object feature representation and the noise data are input into the denoising unit, and the denoising result is output. The denoising unit is used to perform a denoising process on the noise data based on the object feature representation.

[0297] In some embodiments, the object prompt information includes the image information and the text information; the denoising module 1020 is used to:

[0298] The image information and the text information are input into the object encoder, and the object feature representation is output, where the object feature representation is used to represent a fused feature representation of the image feature representation corresponding to the image information and the text feature representation corresponding to the text information.

[0299] In some embodiments, the denoising unit includes a self-attention layer, a cross-attention layer and a feature transformation layer; the denoising module 1020 is used to:

[0300] Inputting the noise feature representation corresponding to the noise data into the self-attention layer, and outputting the self-attention feature representation, wherein the self-attention feature representation is used to characterize the association relationship between at least two noise feature elements included in the noise feature representation;

[0301] Inputting the object feature representation and the self-attention feature representation into the cross-attention layer, and outputting the cross-attention feature representation, wherein the cross-attention feature representation is used to characterize the association relationship between the self-attention feature representation and the object feature representation;

[0302] The cross-attention feature representation is input into the feature transformation layer, and the denoising result is output.

[0303] In some embodiments, the denoising module 1020 is used to:

[0304] Using the object prompt information as a denoising condition, performing a denoising process on the noise data to obtain candidate feature parameters, where the candidate feature parameters refer to feature parameters that characterize the appearance of the first object using data within a preset normalized data range;

[0305] An inverse normalization operation is performed on the candidate feature parameters to obtain the performance feature parameters as the denoising result.

[0306] In some embodiments, the generator includes n variables, the noise data includes n sub-noise data, and n is a positive integer; the denoising module 1020 is used to:

[0307] Taking the object prompt information as a denoising condition, a denoising process is performed on the n sub-noise data to obtain the denoising result, wherein the performance characteristic parameters included in the denoising result include n characteristic parameters, and the n characteristic parameters correspond one-to-one to the n variables.

[0308] In some embodiments, the generating module 1030 is used to:

[0309] Performing a parameter editing operation on the performance characteristic parameter to obtain the edited performance characteristic parameter; wherein the parameter editing operation includes at least one of parameter modification, parameter deletion, and parameter addition;

[0310] The edited performance characteristic parameters are input into the generator, and the three-dimensional model data corresponding to the first object is generated by the generator.

[0311] In some embodiments, the object prompt information refers to the image information, and the image information is used to display the first object under a specified viewing angle; the generating module 1030 is further used to:

[0312] Rendering the three-dimensional model data corresponding to the first object based on the specified viewing angle to obtain a rendered image, wherein the rendered image is used to present a rendering result of the three-dimensional model data at the specified viewing angle;

[0313] The rendering characteristic parameters are adjusted based on a difference between the image information and the rendered image.

[0314] In some embodiments, the generating module 1030 is further used to:

[0315] determining a similarity between the image information and the rendered image;

[0316] When the similarity is less than or equal to a preset similarity, the expression feature parameter is adjusted according to a difference in expression features between the first object included in the image information and the first object included in the rendered image.

[0317] In some embodiments, the object prompt information is used to describe the appearance of the first object and the dynamic effect of the first object, and the denoising result also includes effect feature parameters corresponding to the dynamic effect of the first object; the generating module 1030 is further used to:

[0318] The performance characteristic parameters and the effect characteristic parameters are input into a generator, and the generator generates animation data corresponding to the first object, wherein the animation data includes the three-dimensional model data, and the animation data is used to characterize the dynamic performance of the first object in three-dimensional space.

[0319] In summary, the three-dimensional model generation device provided in the embodiment of the present application denoises the noise data under the guidance of the object prompt information, thereby obtaining the performance feature parameters that match the object appearance performance described in the object prompt information, and inputting the performance feature parameters into the generator can generate the three-dimensional model data of the object. In the present application, the process of obtaining the performance feature parameters is modeled as the denoising process of the noise data, and the performance feature parameters that meet the conditions are quickly generated, which reduces the adjustment process of the performance feature parameters, thereby improving the generation efficiency of the three-dimensional model.

[0320] It should be noted that the specific limitations in the embodiments of the one or more three-dimensional model generation devices provided above can refer to the limitations of the three-dimensional model generation method above, and will not be repeated here. Each module of the above device can be implemented in whole or in part by software, hardware, and a combination thereof. Each module can be embedded in or independent of the processor of the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.

[0321] An embodiment of the present application also provides a computer device, which includes: a processor and a memory, wherein a computer program is stored in the memory; and the processor is used to execute the computer program in the memory to implement the three-dimensional model generation method provided in the above-mentioned method embodiments.

[0322] For example, Fig.11 1 is a block diagram of a computer device 1100 provided by an exemplary embodiment of the present application. Generally, the computer device 1100 includes: a processor 1101 and a memory 1102 .

[0323] The processor 1101 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 1101 may be implemented in at least one hardware form of digital signal processing (DSP), field programmable gate array (FPGA), and programmable logic array (PLA). The processor 1101 may also include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 1101 may be integrated with a graphics processing unit (GPU), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1101 may also include an artificial intelligence (AI) processor, which is used to process computing operations related to machine learning.

[0324] The memory 1102 may include one or more computer-readable storage media, which may be non-transitory. The memory 1102 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1102 is used to store at least one instruction, which is used to be executed by the processor 1101 to implement the generation method of the three-dimensional model provided in each method embodiment of the present application.

[0325] In some embodiments, the computer device 1100 may also optionally include: an input interface 1103 and an output interface 1104. The processor 1101, the memory 1102, and the input interface 1103 and the output interface 1104 may be connected via a bus or a signal line. Each peripheral device may be connected to the input interface 1103 and the output interface 1104 via a bus, a signal line or a circuit board. The input interface 1103 and the output interface 1104 may be used to connect at least one peripheral device related to input / output (I / O) to the processor 1101 and the memory 1102. In some embodiments, the processor 1101, the memory 1102, the input interface 1103, and the output interface 1104 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1101, the memory 1102, the input interface 1103, and the output interface 1104 may be implemented on a separate chip or circuit board, which is not limited in the embodiments of the present application.

[0326] Those skilled in the art will understand that Fig.11 The structure shown in the figure does not constitute a limitation on the computer device 1100, and the computer device 1100 may include more or less components than shown in the figure, or combine some components, or adopt a different component arrangement.

[0327] In an exemplary embodiment, the present application provides a chip, which includes a programmable logic circuit and / or program instructions. When the chip runs on a computer device, it is used to implement the three-dimensional model generation method provided by the above-mentioned various method embodiments.

[0328] In an exemplary embodiment, the present application provides a computer-readable storage medium storing a computer program, which is loaded and executed by a processor to implement the three-dimensional model generation method provided by each of the above method embodiments.

[0329] In an exemplary embodiment, the present application provides a computer program product or a computer program, the computer program product or the computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the processor of the computer device loads and executes the method for generating a three-dimensional model provided in the above method embodiment.

[0330] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0331] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned computer-readable storage medium may be a read-only memory, a disk or an optical disk, etc.

[0332] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented with hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein the communication media include any media that facilitates the transmission of a computer program from one place to another. The storage medium can be any available medium that a general or special-purpose computer can access.

[0333] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for generating a three-dimensional model, characterized in that: The method comprises: Acquire object prompt information of a first object, where the object prompt information is used to describe the first object, and the object prompt information includes at least one of image information and text information; Using the object prompt information as a denoising condition, performing a denoising process on the noise data to obtain a denoising result, wherein the denoising result includes a performance feature parameter corresponding to the appearance performance of the first object; The performance characteristic parameters are input into a generator, and the three-dimensional model data corresponding to the first object is generated by the generator.

2. The method according to claim 1, characterized in that: The denoising process is performed on the noise data using the object prompt information as a denoising condition to obtain a denoising result, including: Inputting the object prompt information and the noise data into a diffusion model, and outputting the denoising result; The diffusion model is a machine learning model that learns the correlation between input prompt information and performance feature parameters during the training phase.

3. The method according to claim 2, characterized in that The diffusion model includes an object encoder and a denoising unit; The step of inputting the object prompt information and the noise data into a diffusion model and outputting the denoising result comprises: Inputting the object prompt information into the object encoder, and outputting an object feature representation corresponding to the object prompt information; wherein, when the object prompt information includes the image information, the object feature representation is used to represent the visual features of the first object, and when the object prompt information includes the text information, the object feature representation is used to represent the semantics of the text information; The object feature representation and the noise data are input into the denoising unit, and the denoising result is output. The denoising unit is used to perform a denoising process on the noise data based on the object feature representation.

4. The method according to claim 3, characterized in that: The object prompt information includes the image information and the text information; The step of inputting the object prompt information into the object encoder and outputting the object feature representation corresponding to the object prompt information comprises: The image information and the text information are input into the object encoder, and the object feature representation is output, where the object feature representation is used to represent a fused feature representation of the image feature representation corresponding to the image information and the text feature representation corresponding to the text information.

5. The method according to claim 3, characterized in that: The denoising unit includes a self-attention layer, a cross-attention layer and a feature transformation layer; The step of inputting the object feature representation and the noise data into the denoising unit and outputting the denoising result comprises: Inputting the noise feature representation corresponding to the noise data into the self-attention layer, and outputting the self-attention feature representation, wherein the self-attention feature representation is used to characterize the association relationship between at least two noise feature elements included in the noise feature representation; Inputting the object feature representation and the self-attention feature representation into the cross-attention layer, and outputting the cross-attention feature representation, wherein the cross-attention feature representation is used to characterize the association relationship between the self-attention feature representation and the object feature representation; The cross-attention feature representation is input into the feature transformation layer, and the denoising result is output.

6. The method according to claim 1, characterized in that The denoising process is performed on the noise data using the object prompt information as a denoising condition to obtain a denoising result, including: Using the object prompt information as a denoising condition, performing a denoising process on the noise data to obtain candidate feature parameters, where the candidate feature parameters refer to feature parameters that characterize the appearance of the first object using data within a preset normalized data range; An inverse normalization operation is performed on the candidate feature parameters to obtain the performance feature parameters as the denoising result.

7. The method according to claim 1, characterized in that The generator includes n variables, the noise data includes n sub-noise data, and n is a positive integer; The denoising process is performed on the noise data using the object prompt information as a denoising condition to obtain a denoising result, including: Taking the object prompt information as a denoising condition, a denoising process is performed on the n sub-noise data to obtain the denoising result, wherein the performance characteristic parameters included in the denoising result include n characteristic parameters, and the n characteristic parameters correspond one-to-one to the n variables.

8. The method according to any one of claims 1 to 7, characterized in that: The step of inputting the performance characteristic parameters into a generator and generating three-dimensional model data corresponding to the first object through the generator includes: Performing a parameter editing operation on the performance characteristic parameter to obtain the edited performance characteristic parameter; wherein the parameter editing operation includes at least one of parameter modification, parameter deletion, and parameter addition; The edited performance characteristic parameters are input into the generator, and the three-dimensional model data corresponding to the first object is generated by the generator.

9. The method according to any one of claims 1 to 7, characterized in that: The object prompt information refers to the image information, and the image information is used to display the first object at a specified viewing angle; After the performance characteristic parameters are input into a generator and the three-dimensional model data corresponding to the first object is generated by the generator, the method further includes: Rendering the three-dimensional model data corresponding to the first object based on the specified viewing angle to obtain a rendered image, wherein the rendered image is used to present a rendering result of the three-dimensional model data at the specified viewing angle; The rendering characteristic parameters are adjusted based on a difference between the image information and the rendered image.

10. The method according to claim 9, characterized in that The adjusting the performance characteristic parameter based on the difference between the image information and the rendered image includes: determining a similarity between the image information and the rendered image; When the similarity is less than or equal to a preset similarity, the expression feature parameter is adjusted according to a difference in expression features between the first object included in the image information and the first object included in the rendered image.

11. The method according to any one of claims 1 to 7, characterized in that: The object prompt information is used to describe the appearance of the first object and the dynamic effect of the first object, and the denoising result also includes effect feature parameters corresponding to the dynamic effect of the first object; The method further comprises: The performance characteristic parameters and the effect characteristic parameters are input into a generator, and the generator generates animation data corresponding to the first object, wherein the animation data includes the three-dimensional model data, and the animation data is used to characterize the dynamic performance of the first object in three-dimensional space.

12. A three-dimensional model generation device, characterized in that: The device comprises: An acquisition module, used to acquire object prompt information of a first object, where the object prompt information is used to describe the first object, and the object prompt information includes at least one of image information and text information; A denoising module, configured to perform a denoising process on the noise data using the object prompt information as a denoising condition to obtain a denoising result, wherein the denoising result includes a performance feature parameter corresponding to the appearance performance of the first object; A generation module is used to input the performance characteristic parameters into a generator, and generate three-dimensional model data corresponding to the first object through the generator.

13. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one program, and the at least one program is loaded and executed by the processor to implement the method for generating a three-dimensional model as described in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that: The storage medium stores at least one program, and the at least one program is loaded and executed by the processor to implement the method for generating a three-dimensional model as described in any one of claims 1 to 11.

15. A computer program product, characterized in that It comprises a computer program, which, when executed by a processor, implements the method for generating a three-dimensional model as described in any one of claims 1 to 11.

Images