Three-dimensional media asset generation method and system based on sketch

Abstract

The invention discloses a sketch-based three-dimensional media asset generation method and system, and relates to the field of three-dimensional media asset generation. The problem that in the existing process from sketch to 3D asset generation, due to the lack of an effective integration and processing mechanism of stylized information, the ever-increasing diversified design concept requirements of users are difficult to meet is solved. The method comprises the following steps: generating a stylized reference graph by adopting a diffusion model; inputting the stylized reference image into a 3D diffusion model, and calculating color loss and transparency loss of the stylized reference image and the 3D rendering image under the corresponding view angle; generating guide images with different visual angles and consistent styles and heights by adopting a horizontal and vertical surrounding camera projection method; performing style optimization on the generated guide images of different visual angles; and geometric similarity optimization is carried out on the generated guide images with highly consistent styles, and generation from the sketch to the three-dimensional media assets is completed based on the above steps. The method and the system are also suitable for the field of three-dimensional media creation.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional media asset generation, and in particular to a sketch-based three-dimensional media asset generation method and system. Background Art

[0002] In today's rapidly evolving technological landscape, image-to-3D generation technology has achieved remarkable breakthroughs. Leveraging a complex and advanced set of algorithms and model systems, natural images are transformed into highly detailed, complex, and expressively detailed 3D assets through a specialized conversion process. This technological advancement undoubtedly demonstrates immense potential and driving innovation in a wide range of fields, including film and television special effects production, virtual reality experience creation, and 3D printing design.

[0003] However, when we focus our attention on real-world application scenarios, it's not difficult to discover many thorny issues. Whether it's non-professional users who lack professional image resource libraries and image processing skills, or experienced professional designers with extensive experience in the design field, they often face the dilemma of finding ideal image samples when carrying out actual creative projects. Such ideal image samples not only require a high degree of fit with the design intent in terms of content and theme, but also must meet high standards in multiple dimensions such as image clarity, color accuracy, and reasonable composition. Due to the scarcity of such ideal image samples, they have to rely more on a more primitive and intuitive method when expressing their unique design inspiration: sketching.

[0004] Sketching, as a means of quickly outlining design concepts and general forms with simple lines and rough outlines, offers the advantages of being convenient, fast, and able to instantly capture moments of inspiration. However, it's important to note that most existing methods for generating 3D assets from sketches have significant shortcomings. During their creation, there's often a lack of effective mechanisms for incorporating and processing stylistic information. Stylistic information encompasses a rich range of content, including artistic styles from diverse cultural backgrounds, unique visual tendencies shaped by popular fashion elements, and the stylistic principles upheld by specific design schools. Without this infusion of stylistic information, the generated 3D assets appear monotonous and lackluster in appearance, failing to showcase their diverse artistic charm and cultural heritage.

[0005] At the same time, the lack of multi-perspective data is also a major drawback. Multi-perspective data can comprehensively record and describe the form, structure, and spatial relationships of an object from different observation angles, providing all-round, three-dimensional information support for the construction of 3D assets. Due to the lack of multi-perspective data, the generated 3D assets are often only constructed based on sketch information from a single perspective. This leads to a significant reduction in the sense of three-dimensionality, spatial layering, and richness of detail. In turn, they are limited to a single and fixed style and seriously lack the delicate and rich sense of reality that objects in the real world have.

[0006] This limitation, caused by the lack of stylized information and multi-view data, makes it difficult to meet users' growing demand for diverse design concepts in many practical application scenarios, such as high-end interior design solution presentations, innovative product concept design promotions, and immersive game scene development. In pursuit of unique, personalized, and visually impactful designs, users often expect 3D assets to incorporate diverse stylistic elements and display a high degree of realism and detail from all angles. However, existing sketch-to-3D asset generation technology clearly needs improvement and advancement. Summary of the Invention

[0007] This invention solves the existing problem of generating 3D assets from sketches, which lacks an effective mechanism for integrating and processing stylized information. This results in the generated 3D assets being monotonous and boring in appearance, making it difficult to meet users' growing demand for diverse design concepts. To solve the above technical problems, this invention is implemented through the following technical solutions:

[0008] Solution 1: The present invention proposes a sketch-based 3D media asset generation method, comprising:

[0009] Step 1: Generate a stylized reference image using a diffusion model;

[0010] Step 2: Input the stylized reference image generated in step 1 into the 3D diffusion model, calculate the color loss and transparency loss between the stylized reference image and the 3D rendering at the corresponding viewing angle, optimize the 3D diffusion model, and obtain the initial 3D model.

[0011] Step 3: Using the horizontal and vertical surround camera projection method based on the initial 3D model obtained in step 2, calculate and generate guidance images with highly consistent styles and different perspectives;

[0012] Step 4: Perform style optimization on the images with consistent styles and different perspectives generated in step 3;

[0013] Step 5: Perform geometric similarity optimization on the images with consistent styles and different perspectives generated in step 3, and complete the generation of three-dimensional media assets for the sketch based on the above steps.

[0014] Furthermore, a preferred embodiment is provided, wherein step 1 of using a diffusion model to generate a stylized reference image includes the steps of forward diffusion and backward denoising.

[0015] Furthermore, a preferred embodiment is provided, wherein the diffusion model is implemented based on a characteristic modulation structure, and the characteristic modulation includes a dual-stream encoder and a single-stream decoder.

[0016] Furthermore, a preferred embodiment is provided, wherein the dual-stream encoder is composed of two parameter-sharing U-Net encoders, and the single-stream decoder is implemented based on the U-Net decoder structure.

[0017] Furthermore, a preferred embodiment is provided, in which the method for calculating the color loss and transparency loss between the stylized reference image and the 3D rendering image at the corresponding viewing angle in step 2 is:

[0018]

[0019] Among them, λ RGB and λ A denote the weights that increase linearly during training, and Respectively represent the color and transparency of the reference image, and Respectively represent the color and transparency of the 3D rendering at the corresponding viewing angle.

[0020] Furthermore, a preferred embodiment is provided, wherein step 4 also includes a step of optimizing the color and shape of the guide image.

[0021] Solution 2: A sketch-based 3D media asset generation system, comprising:

[0022] A stylized reference image generation module, configured to generate a stylized reference image using a diffusion model;

[0023] The 3D initialization module is used to input the stylized reference image generated by the stylized reference image generation module into the 3D diffusion model, calculate the color loss and transparency loss between the stylized reference image and the 3D rendering image at the corresponding viewing angle, and initialize and generate a rough 3D model;

[0024] The multi-view generation module is used to calculate and generate guidance images with highly consistent styles and different perspectives based on the rough 3D model obtained by the 3D initialization module using horizontal and vertical surround camera projection methods;

[0025] The style optimization module is used to optimize the style of images with consistent style but different perspectives generated by the multi-view generation module;

[0026] The geometric similarity optimization module is used to optimize the geometric similarity of images with consistent style and different perspectives generated by the multi-view generation module, and complete the generation from sketches to three-dimensional media assets based on the above modules.

[0027] Solution 3: A computer device includes a memory and a processor, wherein the memory stores a computer program. When the processor runs the computer program stored in the memory, the processor executes any one of the methods described in Solution 1.

[0028] Solution 4: A computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of the method described in any one of Solution 1.

[0029] The present invention is beneficial in that:

[0030] The sketch-based three-dimensional media asset generation method described in the present invention utilizes a generative model to conduct in-depth mining and analysis of multi-perspective data, comprehensively recording and describing the form, structure, and spatial relationship of objects from different observation angles, and providing all-round, three-dimensional information support for the construction of 3D assets, thereby effectively enhancing the three-dimensional sense, spatial layering, and detail richness of the model, so that it is no longer limited to a single and fixed style, but is closer to the delicate and rich sense of reality of objects in the real world, providing creators with more powerful technical support when carrying out actual creative projects, and solving the dilemma of relying on sketching due to the scarcity of ideal image samples but facing poor generation effects.

[0031] The present invention is also applicable to the field of three-dimensional media creation. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is a flowchart of a sketch-based 3D media asset generation method according to the first embodiment.

[0033] Figure 2 Generate a network structure diagram for the reference diagram described in Implementation 11.

[0034] Figure 3 This is a schematic diagram of the IP-Adapter structure described in the eleventh embodiment. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solutions and advantages of the implementation methods of this application clearer, the technical solutions in the implementation methods of this application will be clearly and completely described below in combination with the drawings in the implementation methods of this application. Obviously, the described implementation methods are only part of the implementation methods of this application, not all of the implementation methods.

[0036] Embodiment 1: This embodiment provides a sketch-based 3D media asset generation method, the method comprising:

[0037] Step 1: Generate a stylized reference image using a diffusion model;

[0038] Step 2: Input the stylized reference image generated in step 1 into the 3D diffusion model, calculate the color loss and transparency loss between the stylized reference image and the 3D rendering at the corresponding viewing angle, optimize the 3D diffusion model, and obtain the initial 3D model.

[0039] Step 3: Using the horizontal and vertical surround camera projection method based on the initial 3D model obtained in step 2, calculate and generate guidance images with highly consistent styles and different perspectives;

[0040] Step 4: Perform style optimization on the images with consistent styles and different perspectives generated in step 3;

[0041] Step 5: Perform geometric similarity optimization on the images with consistent styles and different perspectives generated in step 3, and complete the generation of three-dimensional media assets for the sketch based on the above steps.

[0042] Implementation method 2: This implementation method further limits the sketch-based 3D media asset generation method described in implementation method 1. Step 1 uses a diffusion model to generate a stylized reference image, including forward diffusion and reverse denoising steps.

[0043] Implementation method 3: This implementation method further limits the sketch-based 3D media asset generation method described in implementation method 2. The diffusion model is implemented based on a feature modulation structure, and the feature modulation includes a dual-stream encoder and a single-stream decoder.

[0044] Implementation method 4: This implementation method further limits the sketch-based 3D media asset generation method described in implementation method 3. The dual-stream encoder is composed of two parameter-sharing U-Net encoders, and the single-stream decoder is implemented based on the U-Net decoder structure.

[0045] Implementation 5: This implementation further defines the sketch-based 3D media asset generation method described in Implementation 1. In step 2, the method for calculating the color loss and transparency loss between the stylized reference image and the 3D rendering at the corresponding viewing angle is as follows:

[0046]

[0047] Among them, λ RGB and λ A denote the weights that increase linearly during training, and Respectively represent the color and transparency of the reference image, and Respectively represent the color and transparency of the 3D rendering at the corresponding viewing angle.

[0048] Implementation method 6: This implementation method further limits the sketch-based 3D media asset generation method described in implementation method 1. In step 3, a horizontal and vertical surround camera projection method is used to obtain a guide image with highly consistent style every 30 degrees for each rotation circle.

[0049] Implementation 7: This implementation further limits the sketch-based 3D media asset generation method described in Implementation 1, and step 4 further includes a step of optimizing the color and shape of the guide image.

[0050] Embodiment 8: This embodiment proposes a sketch-based 3D media asset generation system, comprising:

[0051] A stylized reference image generation module, configured to generate a stylized reference image using a diffusion model;

[0052] The 3D initialization module is used to input the stylized reference image generated by the stylized reference image generation module into the 3D diffusion model, calculate the color loss and transparency loss between the stylized reference image and the 3D rendering image at the corresponding viewing angle, and initialize and generate a rough 3D model;

[0053] The multi-view generation module is used to calculate and generate guidance images with highly consistent styles and different perspectives based on the rough 3D model obtained by the 3D initialization module using horizontal and vertical surround camera projection methods;

[0054] The style optimization module is used to optimize the style of images with consistent style but different perspectives generated by the multi-view generation module;

[0055] The geometric similarity optimization module is used to optimize the geometric similarity of the images with consistent style and different perspectives generated by the multi-view generation module, and complete the generation from sketches to three-dimensional media assets based on the above steps.

[0056] Implementation method 9. This implementation method proposes a computer device including a memory and a processor, wherein the memory stores a computer program. When the processor runs the computer program stored in the memory, the processor executes the method described in any one of implementation methods 1 to 7.

[0057] Implementation 10: This implementation proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the method described in any one of Implementation 1 to Implementation 7 are implemented.

[0058] Implementation 11: This implementation provides an example, which is used to explain the above implementations 1 to 8. Figures 1 to 3 To illustrate this embodiment, the method described in this embodiment includes the following steps:

[0059] Step 1: Generate stylized reference images:

[0060] This embodiment utilizes a diffusion model to generate a stylized reference image. The diffusion model involves two key stages: a forward diffusion process and a backward denoising process. The goal of this process is to establish a mapping model from the actual data distribution to the Gaussian noise distribution. During the forward diffusion process, Gaussian noise ∈ is gradually introduced into the original clean sample x0. The intensity of the noise increases with the time parameter t∈[1,2,…,T]. Through this diffusion process, the noisy image at step t is obtained, which is mathematically expressed as:

[0061]

[0062] in,

[0063] Conversely, in the denoising process, the model restores the noisy image to a clear image by gradually predicting and eliminating the noise. In the DDIM framework, in order to preserve the key content information, the deterministic sampling process in the transition from time step t to t-1 can be expressed as:

[0064]

[0065] Here, c represents the conditional input (e.g., style feature embedding), ∈ θ represents the noise prediction network. The predicted denoised image is represented as Obtained through:

[0066]

[0067] Figure 2This paper presents the overall architecture used in this project to generate style reference maps, which consists of a dual-stream encoder and a single-stream decoder. The dual-stream encoder consists of two parameter-sharing U-Net encoders, while the single-stream decoder is based on the U-Net decoder structure. A detailed description of the dual-stream downsampling process is as follows:

[0068] The encoder consists of two parallel U-Net encoders, each responsible for extracting different feature levels of the input image. One encoder focuses on extracting content features, while the other focuses on style features.

[0069] By sharing parameters, the two encoders can be updated synchronously, ensuring consistency and efficiency of feature extraction.

[0070]

[0071] where x s Represents the initial input style image, x c Sketch representing the input. f s and f c Represent the image features that carry style and content information respectively. Given a noisy input x t , the denoising process reverses the diffusion process to predict the clean data x t-1 :

[0072]

[0073] Step 2: Use Gaussian initialization of the 3D diffusion model:

[0074] The present invention generates a style reference image based on an image and initializes a 3D Gaussian using a 3D diffusion model (eg, Shap-E).

[0075] a. Input the stylized reference image generated in step 1 into the 3D diffusion model and calculate the color loss and transparency loss between the stylized reference image and the 3D rendering at the corresponding viewing angle. The formula for calculating the total loss of the reference image is as follows:

[0076]

[0077] Among them, λ RGB and λ A denote the weights that increase linearly during training, and Respectively represent the color and transparency of the reference image, and Respectively represent the color and transparency of the 3D rendering at the corresponding viewing angle.

[0078] b. Based on the known perspective, predict the unknown perspective. The optimization method under the unknown perspective is as follows:

[0079]

[0080] Where w(t) is the weight function, φ (g) is the noise predicted by the 2D diffusion prior φ, z t is the result of adding noise to the input at time t, y is the input condition, and ò represents the result of adding noise to the 3D model prediction data.

[0081] Step 3: Camera progressive projection:

[0082] like Figure 1 As shown, after step 2, a horizontal and vertical surround camera projection strategy is adopted. In order to ensure the consistency of the style of the generated guidance image, the present invention adopts a meticulous method: an image is acquired every 30 degrees of each rotation circle.

[0083] Step 4: Style optimization:

[0084] In this paper, we address the challenges of blurring, sticking, and lack of realism encountered when generating 3D Gaussian models from a single viewpoint. In step three, we employ a strategy for style optimization of the camera's progressively projected image. By further performing style optimization on images from different viewpoints, we ensure that the styles of the images generated from different viewpoints remain consistent with the target style map.

[0085] a. The present invention uses Stable-Diffusion-v1-5 as the basic diffusion model. Given a reference image I Ref and the multi-view splash image as the content image I c , guide image I g The estimates are as follows:

[0086] I g =M(I Ref ,I c ,t,λ) (8)

[0087] Where M is the generator of IP-Adapter, t is the sampling time step of inference, and λ∈[0,1] is a hyperparameter that determines the conditional content image I c control intensity.

[0088] b. Further optimize the color and shape of the generated image

[0089]

[0090] in and are the foreground masks of the guidance image and content image, respectively.

[0091] Step 5: Geometric similarity optimization:

[0092] To ensure that the outline of the sketch can accurately guide the optimization process of the 3D Gaussian model, the present invention uses the CLIP image encoder to extract features from the sketch and rendered image, and further calculates the L2 distance between the activated features of the CLIP intermediate layer.

[0093]

[0094] where λ sketch is the coefficient controlling the weight, and CLIP4(·) is the encoder activation of CLIP layer 4. Specifically, the present invention uses the 4th layer of the ResNet101 CLIP model.

[0095] Example 1 was tested on the ShapeNet-Sketch3D dataset and compared with the prior art. The results are shown below. It can be seen that the performance of the method of the present invention is improved compared with both prior art methods.

[0096] method Shap-E One-2-3-45 The present invention CLIP-Similarity 0.642 0.667 0.724

[0097] Those skilled in the art will understand that the above description is only a preferred embodiment of the present invention, and the features described in the various embodiments and / or claims of the present disclosure may be combined or coupled in various ways, even if such a combination or coupling is not explicitly described in the present disclosure. It is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art may still modify the technical solutions described in the aforementioned embodiments or make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included in the scope of protection of the present invention.

[0098] Although preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they are aware of the basic inventive concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention. Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, the present invention is intended to include such changes and modifications as fall within the scope of the claims and their equivalents.

Claims

1. A sketch-based 3D media asset generation method, characterized in that: The method comprises: Step 1: Generate a stylized reference image using a diffusion model; Step 2: Input the stylized reference image generated in step 1 into the 3D diffusion model, calculate the color loss and transparency loss between the stylized reference image and the 3D rendering image at the corresponding viewing angle, optimize the 3D diffusion model, and obtain the initial 3D model; Step 3: Calculate and generate guide images with highly consistent styles and different perspectives by using horizontal and vertical surround camera projection methods based on the initial 3D model obtained in step 2; Step 4: Perform style optimization on the images with consistent styles and different perspectives generated in step 3; Step 5: Optimize the geometric similarity of the images with the same style and different viewing angles generated in step 3, and complete the generation of three-dimensional media assets for the sketch based on the above steps.

2. The sketch-based 3D media asset generation method according to claim 1, characterized in that: Step 1 uses a diffusion model to generate a stylized reference image, which includes forward diffusion and reverse denoising steps.

3. The sketch-based 3D media asset generation method according to claim 2, characterized in that: The diffusion model is implemented based on an eigenmodulation structure, which includes a dual-stream encoder and a single-stream decoder.

4. The sketch-based 3D media asset generation method according to claim 3, characterized in that: The dual-stream encoder consists of two parameter-sharing U-Net encoders, and the single-stream decoder is implemented based on the U-Net decoder structure.

5. The sketch-based 3D media asset generation method according to claim 1, characterized in that: The method for calculating the color loss and transparency loss of the stylized reference image and the 3D rendering image at the corresponding viewing angle in step 2 is: Among them, λ RGB and λ A Represent the weights that increase linearly during training, and Respectively represent the color and transparency of the reference image, and Respectively represent the color and transparency of the 3D rendering at the corresponding viewing angle.

6. The sketch-based 3D media asset generation method according to claim 1, characterized in that: In step 3, a horizontal and vertical surround camera projection method is used to obtain a highly consistent style guidance image every 30 degrees for each rotation circle.

7. The sketch-based 3D media asset generation method according to claim 1, characterized in that: Step 4 also includes the step of optimizing the color and shape of the guide image.

8. A sketch-based 3D media asset generation system, characterized in that: The system comprises: A stylized reference image generation module, used to generate a stylized reference image using a diffusion model; A 3D initialization module is used to input the stylized reference image generated in the stylized reference image generation module into the 3D diffusion model, calculate the color loss and transparency loss of the stylized reference image and the 3D rendering image under the corresponding viewing angle, and initialize and generate a rough 3D model; A multi-view generation module is used to calculate and generate guide images with highly consistent styles and different view angles using the rough 3D model obtained by the 3D initialization module according to the horizontal and vertical surround camera projection method; A style optimization module, which is used to perform style optimization on images with consistent styles and different perspectives generated by the multi-view generation module; The geometric similarity optimization module is used to optimize the geometric similarity of the images with consistent style and different perspectives generated by the multi-perspective generation module, and complete the generation from sketch to three-dimensional media assets based on the above steps.

9. A computer device comprising a memory and a processor, characterized in that A computer program is stored in the memory. When the processor runs the computer program stored in the memory, the processor executes the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images