Model multi-material information generation method and device and storage medium

Abstract

The invention discloses a model multi-material information generation method and device and a storage medium, and the method comprises the steps: taking random noise as the input data of an initialized convolutional neural network, and outputting initial multi-material information; rendering the initial multi-material information into a preset grid model, and generating a multi-view image set of the initial multi-material information; determining an SDS loss value between the material features of the multi-view image set and the input text features; and updating neural parameters of the initialized convolutional neural network by using the SDS loss value, and optimizing the initial multi-material information according to the updated neural parameters to obtain optimized multi-material information. According to the method and the device, the required multi-material information can be generated through parameter optimization, and each pixel parameter of the multi-material information does not need to be optimized, so that the efficiency of generating the multi-material information can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method, device and storage medium for generating multi-material information of a model. Background Art

[0002] Mesh material information generation technology is an important branch of computer graphics, which involves how to create and apply various material properties on the surface of a 3D model to achieve a more realistic and detailed visual effect. For example, in car design, multi-material information generation technology can be used to create realistic car models, including the metal texture of the car body, the fabric texture of the interior, the rubber texture of the tires, etc., all of which require accurate material information to achieve realistic rendering.

[0003] The existing model multi-material information generation method generates the final model multi-material information by optimizing pixel-level parameters during the model multi-material information generation process. However, the pixel-level parameter optimization process is relatively complicated, and the model multi-material information generation efficiency is low. Summary of the invention

[0004] The present invention provides a model multi-material information generation method, device and storage medium to solve the technical problem that the existing model multi-material information generation method generates the final model multi-material information by optimizing pixel level parameters during the model multi-material information generation process, but the pixel level parameter optimization process is relatively complicated and the model multi-material information generation efficiency is low.

[0005] The present invention provides a method for generating multi-material information of a model, comprising:

[0006] Use random noise as input data to initialize the convolutional neural network and output initial multi-material information;

[0007] Rendering the initial multi-material information into a preset grid model to generate a multi-view atlas of the initial multi-material information;

[0008] Determining an SDS loss value between the material features of the multi-view atlas and the input text features;

[0009] The neural parameters of the initialized convolutional neural network are updated with the SDS loss value, and the initial multi-material information is optimized according to the updated neural parameters to obtain optimized multi-material information.

[0010] Furthermore, the random noise is used as input data for initializing the convolutional neural network to output initial multi-material information, including:

[0011] Random noise is used as input data of the initialized convolutional neural network, and initial multi-material information is generated according to the encoder-decoder structure of the initialized convolutional neural network.

[0012] Further, determining the SDS loss value between the material feature of the multi-view atlas and the input text feature includes:

[0013] The similarity between the material features of the multi-view atlas and the input text features is calculated according to the pre-trained diffusion model, and the SDS loss value between the material features and the input text features is determined according to the similarity.

[0014] Furthermore, the model multi-material information generation method also includes:

[0015] At least one feature image is used to fine-tune the diffusion model, so that when the input text features contain features corresponding to the feature image, the fine-tuned diffusion model generates multi-material information that is the same as the feature image.

[0016] Furthermore, after obtaining the optimized multi-material information, it also includes:

[0017] Determine local features of the model according to the optimized multi-material information;

[0018] The three-dimensional coordinates of the local features of the model are determined, and the camera rendering position is adjusted according to the three-dimensional coordinates.

[0019] The present invention also provides a model multi-material information generation device, comprising:

[0020] An initial multi-material information output module is used to use random noise as input data for initializing the convolutional neural network and output initial multi-material information;

[0021] A multi-view atlas generation module, used for rendering the initial multi-material information into a preset grid model to generate a multi-view atlas of the initial multi-material information;

[0022] An SDS loss value determination module, used to determine an SDS loss value between the material features of the multi-view atlas and the input text features;

[0023] The optimized multi-material information generation module is used to update the neural parameters of the initialized convolutional neural network with the SDS loss value, and optimize the initial multi-material information according to the updated neural parameters to obtain the optimized multi-material information.

[0024] Furthermore, the initial multi-material information output module is also used for:

[0025] Random noise is used as input data of the initialized convolutional neural network, and initial multi-material information is generated according to the encoder-decoder structure of the initialized convolutional neural network.

[0026] Furthermore, the SDS loss value determination module is also used to:

[0027] The similarity between the material features of the multi-view atlas and the input text features is calculated according to the pre-trained diffusion model, and the SDS loss value between the material features and the input text features is determined according to the similarity.

[0028] The present invention also provides a terminal device, comprising: a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, and when the processor executes the computer program, the method for generating multi-material information of a model as described above is implemented.

[0029] The present invention also provides a computer-readable storage medium, which includes a stored computer program; wherein, when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the model multi-material information generation method as described above.

[0030] The embodiment of the present invention determines the SDS loss value between the material features of the multi-view atlas and the input text features, updates the neural parameters of the initialized convolutional neural network with the SDS loss value, and optimizes the initial multi-material information according to the updated neural parameters to obtain the optimized multi-material information. The required multi-material information can be generated through parameter optimization without optimizing each pixel parameter of the multi-material information, thereby effectively improving the efficiency of multi-material information generation.

[0031] Furthermore, the embodiment of the present invention uses at least one feature image to fine-tune the diffusion model, and can generate multi-material information of specific feature information contained in the fine-tuning image when generating multi-material information, thereby effectively improving the effect of generating model multi-material information. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a flowchart of a method for generating multi-material information of a model provided by an embodiment of the present invention;

[0033] Figure 2 is another schematic diagram of a flow chart of a method for generating multi-material information of a model provided by an embodiment of the present invention;

[0034] Figure 3 It is a structural schematic diagram of a device for generating multi-material information of a model provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0036] In the description of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the feature. In the description of this application, unless otherwise specified, "plurality" means two or more.

[0037] In the description of this application, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0038] See also Figure 1 , a method for generating multi-material information of a model, comprising:

[0039] S1, using random noise as input data to initialize the convolutional neural network and outputting initial multi-material information;

[0040] In the embodiment of the present invention, the random noise is a random input data, which is not directly related to the material information and is a random numerical vector.

[0041] The embodiment of the present invention can generate initial multi-material information by initializing the encoder-decoder structure of the convolutional neural network U-Net (i.e., the "hourglass" structure), and the multi-material information includes a diffuse map, a roughness map, and a normal map.

[0042] S2, rendering the initial multi-material information into a preset grid model to generate a multi-view atlas of the initial multi-material information;

[0043] In an embodiment of the present invention, the preset grid model is a preset 3D mesh model. By rendering the initial multi-material information to the preset grid image, different view images of the grid model are obtained to obtain a multi-view atlas, which shows the specific positions of the multiple materials in the model.

[0044] S3, determining the SDS loss value between the material features of the multi-view atlas and the input text features;

[0045] In an embodiment of the present invention, SDS loss value (Score Distillation Sampling Loss) is a loss function used in the text-to-3D generation task, which is used to measure the similarity between the generated 3D object and the text description. Each multi-view atlas has a corresponding material feature, and the material feature in the multi-view atlas can be extracted by feature extraction. The material feature is compared with the input text feature for similarity, thereby determining the SDS loss value. Among them, the input text feature is the expected text feature input by the user according to the demand. The role of the expected text feature is to calculate the similarity difference with the material feature in the multi-view atlas, thereby determining the SDS loss value, and then neural parameter optimization can be performed based on the SDS loss value, so that the multi-material information finally optimized covers the expected text feature.

[0046] S4. Update the neural parameters of the initialized convolutional neural network with the SDS loss value, and optimize the initial multi-material information according to the updated neural parameters to obtain the optimized multi-material information.

[0047] In an embodiment of the present invention, after the SDS loss value is calculated, the direction of neural parameter optimization is determined using the SDS loss value, thereby achieving optimization of the neural parameters, and the multi-material information is initialized based on the convolutional neural network optimization after the neural parameters are optimized to obtain the optimized multi-material information.

[0048] The embodiment of the present invention determines the SDS loss value between the material features of the multi-view atlas and the input text features, updates the neural parameters of the initialized convolutional neural network with the SDS loss value, and optimizes the initial multi-material information according to the updated neural parameters to obtain the optimized multi-material information. The required multi-material information can be generated through parameter optimization without optimizing each pixel parameter of the multi-material information, thereby effectively improving the efficiency of multi-material information generation.

[0049] In one embodiment, step S1, using random noise as input data for initializing a convolutional neural network and outputting initial multi-material information, includes:

[0050] Random noise is used as input data for initializing the convolutional neural network, and initial multi-material information is generated according to the encoder-decoder structure of the initialized convolutional neural network.

[0051] In an embodiment of the present invention, random noise is jump-connected in the encoder-decoder structure, which can restore details in the decoding process, so that the generated multi-material information contains both global semantic information and local details, thereby effectively improving the generation effect of multi-materials of the model.

[0052] In one embodiment, determining the SDS loss value between the material features of the multi-view atlas and the input text features includes:

[0053] The similarity between the material features of the multi-view atlas and the input text features is calculated according to the pre-trained diffusion model, and the SDS loss value between the material features and the input text features is determined according to the similarity.

[0054] In an embodiment of the present invention, a pre-trained diffusion model text-to-image can be used to calculate the SDS loss between the multi-view atlas image and the text feature a[V]car, and then the update direction of the neural parameters in the convolutional neural network is determined based on the optimization of SDS.

[0055] See also Figure 2 , is another flow chart of a method for generating multi-material information of a model provided by an embodiment of the present invention. Figure 2 As shown, the multi-view atlas image and the text feature a[V]car are input into the fine-tuned diffusion model Dreanbooth, and the neural parameters of the convolutional neural network are further updated to generate efficient generative model multi-material information, wherein the multi-view atlas can be a vehicle-related texture atlas.

[0056] The embodiment of the present invention optimizes the neural parameters according to the SDS loss value, so that the generated multi-material information is consistent with the input text features, thereby maintaining the consistency of material properties, such as diffuse reflection, metalness, roughness and normal information, and thus effectively improving the effect of multi-material information generation.

[0057] In one embodiment, the model multi-material information generation method further includes:

[0058] At least one feature image is used to fine-tune the diffusion model, so that when the input text features contain features corresponding to the feature image, the fine-tuned diffusion model generates multi-material information that is the same as the features corresponding to the feature image.

[0059] The feature image of the embodiment of the present invention is an image containing specific features, such as an image of a detail of a vehicle or a specific structure. The embodiment of the present invention uses at least one feature image to fine-tune the diffusion model, and can generate multi-material information of the specific feature information contained in the fine-tuned image when generating multi-material information, thereby effectively improving the effect of generating model multi-material information.

[0060] In one embodiment, after obtaining the optimized multi-material information, the method further includes:

[0061] Determine the local features of the model based on the optimized multi-material information;

[0062] Determine the 3D coordinates of the local features of the model and adjust the camera rendering position according to the 3D coordinates.

[0063] In an embodiment of the present invention, the local features of the model include wheel features, taillight features, and window features. The local features usually have unique geometric shapes and texture features, and can be identified by the vertex and face information of the model. After the local features of the model are identified, the coordinate grid of the local features of the model in the three-dimensional space is obtained, and the camera rendering position where the camera is focused is determined according to these three-dimensional space coordinates, and the posture and focal length in the camera rendering position are adjusted to capture the details of these local features.

[0064] The embodiment of the present invention can refine the local structure of the vehicle by determining the three-dimensional coordinates of the local features of the model and adjusting the camera rendering position according to the three-dimensional coordinates, so that the grid model fully displays the overall structure and local structure of the vehicle.

[0065] The implementation of the embodiments of the present invention has the following beneficial effects:

[0066] The embodiment of the present invention determines the SDS loss value between the material features of the multi-view atlas and the input text features, updates the neural parameters of the initialized convolutional neural network with the SDS loss value, and optimizes the initial multi-material information according to the updated neural parameters to obtain the optimized multi-material information. The required multi-material information can be generated through parameter optimization without optimizing each pixel parameter of the multi-material information, thereby effectively improving the efficiency of multi-material information generation.

[0067] Furthermore, the embodiment of the present invention uses at least one feature image to fine-tune the diffusion model, and can generate multi-material information of specific feature information contained in the fine-tuning image when generating multi-material information, thereby effectively improving the effect of generating multi-material information of the model.

[0068] See also Figure 3 Based on the same inventive concept as the above embodiment, the present invention also provides a model multi-material information generation device, comprising:

[0069] An initial multi-material information output module 10 is used to use random noise as input data for initializing a convolutional neural network and output initial multi-material information;

[0070] A multi-view atlas generation module 20, for rendering the initial multi-material information into a preset grid model to generate a multi-view atlas of the initial multi-material information;

[0071] An SDS loss value determination module 30, for determining an SDS loss value between a material feature of a multi-view atlas and an input text feature;

[0072] The optimized multi-material information generation module 40 is used to update the neural parameters of the initialized convolutional neural network with the SDS loss value, and optimize the initial multi-material information according to the updated neural parameters to obtain the optimized multi-material information.

[0073] In one embodiment, the initial multi-material information output module 10 is further used for:

[0074] Random noise is used as input data for initializing the convolutional neural network, and initial multi-material information is generated according to the encoder-decoder structure of the initialized convolutional neural network.

[0075] In one embodiment, the SDS loss value determination module 30 is further configured to:

[0076] The similarity between the material features of the multi-view atlas and the input text features is calculated according to the pre-trained diffusion model, and the SDS loss value between the material features and the input text features is determined according to the similarity.

[0077] In one embodiment, the model multi-material generation device further includes a camera position adjustment module, which is used to:

[0078] Determine the local features of the model based on the optimized multi-material information;

[0079] Determine the 3D coordinates of the local features of the model and adjust the camera rendering position according to the 3D coordinates.

[0080] In one embodiment, the model local features include wheel features, taillight features, and window features.

[0081] Correspondingly, an embodiment of the present invention also provides a terminal device, including: a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, and when the processor executes the computer program, it implements the model multi-material information generation method of any one of the above embodiments.

[0082] The terminal device of this embodiment includes: a processor, a memory, and a computer program and computer instructions stored in the memory and executable on the processor. When the processor executes the computer program, each step in the above embodiment 1 is implemented, for example Figure 1 Alternatively, when the processor executes the computer program, the functions of each module / unit in the above device embodiment are implemented, such as the optimized multi-material information generating module 40 .

[0083] Exemplarily, the computer program can be divided into one or more modules / units, one or more modules / units are stored in a memory and executed by a processor to complete the present invention. One or more modules / units can be a series of computer program instruction segments that can perform specific functions, and the instruction segments are used to describe the execution process of the computer program in a terminal device. For example, the optimized multi-material information generation module 40 is used to update the neural parameters of the initialized convolutional neural network with the SDS loss value, and optimize the initial multi-material information according to the updated neural parameters to obtain the optimized multi-material information.

[0084] The terminal device may be a computing device such as a desktop computer, a notebook, a PDA, and a cloud server. The terminal device may include, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that the schematic diagram is merely an example of a terminal device and does not constitute a limitation on the terminal device. The terminal device may include more or fewer components than shown in the diagram, or may combine certain components, or different components. For example, the terminal device may also include an input / output device, a network access device, a bus, etc.

[0085] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the terminal device, and uses various interfaces and lines to connect various parts of the entire terminal device.

[0086] The memory can be used to store computer programs and / or modules. The processor realizes various functions of the terminal device by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc.; the data storage area can store data created according to the use of the mobile terminal, etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0087] Among them, if the module / unit integrated in the terminal device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electric carrier signals and telecommunication signals.

[0088] Correspondingly, an embodiment of the present invention also provides a computer-readable storage medium, the computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the model multi-material information generation method as described in any one of the above embodiments.

[0089] The above specific embodiments further illustrate the purpose, technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. It is particularly pointed out that for those skilled in the art, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for generating multi-material information of a model, characterized in that: include: Use random noise as input data to initialize the convolutional neural network and output initial multi-material information; Rendering the initial multi-material information into a preset grid model to generate a multi-view atlas of the initial multi-material information; Determining an SDS loss value between the material features of the multi-view atlas and the input text features; The neural parameters of the initialized convolutional neural network are updated with the SDS loss value, and the initial multi-material information is optimized according to the updated neural parameters to obtain optimized multi-material information.

2. The method for generating model multi-material information according to claim 1, characterized in that: The random noise is used as input data for initializing the convolutional neural network, and the initial multi-material information is output, including: Random noise is used as input data of the initialized convolutional neural network, and initial multi-material information is generated according to the encoder-decoder structure of the initialized convolutional neural network.

3. The method for generating model multi-material information according to claim 1, characterized in that: Determining the SDS loss value between the material feature of the multi-view atlas and the input text feature includes: The similarity between the material features of the multi-view atlas and the input text features is calculated according to the pre-trained diffusion model, and the SDS loss value between the material features and the input text features is determined according to the similarity.

4. The method for generating model multi-material information according to claim 3, characterized in that: Also includes: At least one feature image is used to fine-tune the diffusion model, so that when the input text features contain features corresponding to the feature image, the fine-tuned diffusion model generates multi-material information that is the same as the features corresponding to the feature image.

5. The method for generating model multi-material information according to claim 1, characterized in that: After obtaining the optimized multi-material information, it also includes: Determine local features of the model according to the optimized multi-material information; The three-dimensional coordinates of the local features of the model are determined, and the camera rendering position is adjusted according to the three-dimensional coordinates.

6. The method for generating model multi-material information according to claim 5, characterized in that: The local features of the model include wheel features, taillight features and window features.

7. A device for generating multi-material information of a model, characterized in that: include: An initial multi-material information output module is used to use random noise as input data for initializing the convolutional neural network and output initial multi-material information; A multi-view atlas generation module, used for rendering the initial multi-material information into a preset grid model to generate a multi-view atlas of the initial multi-material information; An SDS loss value determination module, used to determine an SDS loss value between the material features of the multi-view atlas and the input text features; The optimized multi-material information generation module is used to update the neural parameters of the initialized convolutional neural network with the SDS loss value, and optimize the initial multi-material information according to the updated neural parameters to obtain the optimized multi-material information.

8. The model multi-material information generating device according to claim 7, characterized in that: The initial multi-material information output module is also used for: Random noise is used as input data of the initialized convolutional neural network, and initial multi-material information is generated according to the encoder-decoder structure of the initialized convolutional neural network.

9. A terminal device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, the method for generating model multi-material information as described in any one of claims 1 to 6 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program; wherein, when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the model multi-material information generation method according to any one of claims 1 to 6.

Images