3D model generation method and device and related equipment

Abstract

The invention provides a 3D model generation method and device and related equipment. The method comprises the steps that an initial text and an initial image input by a user are acquired; generating a posture thermodynamic diagram based on the initial text and the initial image; three views are generated based on fine tuning parameters and the attitude thermodynamic diagram, and the fine tuning parameters are determined based on the initial text and the initial image; preprocessing the three views to obtain a target three view; and performing 3D node processing on the target three views to generate a 3D model. In the embodiment of the invention, the user can automatically generate the 3D model by inputting the initial text and the initial image, and the three-view drawing and 3D modeling links are automatically completed, so that manual participation in the 3D model generation process is not needed, and the labor cost is reduced while the 3D model generation efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus and related equipment for generating 3D models. Background Technology

[0002] In fields such as digital content creation, trendy toy design, game development, and virtual idols, IP modeling (Intellectual Property Modeling) refers to the process of transforming original IPs (such as characters, brand images, story settings, etc.) into producible and interactive digital models through 3D modeling technology.

[0003] Traditional IP modeling processes include independent stages such as concept design, three-view drawing, and 3D modeling. The three-view drawing stage requires designers to manually draw the front view, top view, and side view based on the two-dimensional sketch. The 3D modeling stage requires converting the three views into a polygon mesh, which involves complex operations such as topology optimization and UV unwrapping, and requires a lot of manual adjustments.

[0004] Therefore, existing 3D model generation methods rely on manual processing, which is time-consuming and has high labor costs. Summary of the Invention

[0005] This application provides a 3D model generation method, apparatus, and related equipment, which can solve the technical problems of existing 3D model generation methods relying on manual processing, which is time-consuming and has high labor costs.

[0006] In a first aspect, embodiments of this application provide a 3D model generation method, characterized in that the method includes: Get the initial text and initial image input by the user; Generate a posture heatmap based on the initial text and the initial image; Based on the fine-tuning parameters and the attitude heatmap, a three-view drawing is generated, wherein the fine-tuning parameters are determined based on the initial text and the initial image. The three views are preprocessed to obtain the target three views; The target's three views are processed using 3D nodes to generate a 3D model.

[0007] Optionally, generating a pose heatmap based on the initial text and the initial image includes: Semantic analysis is performed on the initial text to obtain semantic vectors, and noise reduction processing is performed on the initial image; Image encoding is performed on the initial image after noise reduction based on the semantic vector to obtain latent variables; The latent variables are decoded to obtain the target image; The target image is input into a preset painting generation model to generate a pose heatmap; The resolution of the target image is higher than that of the initial image.

[0008] Optionally, generating three views based on the fine-tuning parameters and the attitude heatmap includes: Keyword extraction is performed on the initial text, and image feature extraction is performed on the initial image to obtain fine-tuning parameters; The attitude heatmap is adjusted using the fine-tuning parameters. The adjusted posture heatmap is input into the preset drawing generation model to generate three views.

[0009] Optionally, the three views include a front view, a side view, and a rear view, and the preprocessing of the three views to obtain the target three views includes: Adjust the image sizes of the front view, the side view, and the rear view so that the image sizes of the front view, the side view, and the rear view are the same; Channel separation and feature extraction processes are performed on the front view, the side view, and the rear view to obtain the target three-view diagram.

[0010] Optionally, the step of performing 3D node processing on the target three views to generate a 3D model includes: Perform a texture unwrapping operation on the target three-view drawing to obtain the texture coordinates of the target three-view drawing. The target three-view image is smoothed, and the missing areas of the target three-view image are repaired; The repaired target's three views are transformed into a 3D model.

[0011] Optionally, the fine-tuning parameters include text-image alignment strength, denoising strength, sampling step size, and random seed number.

[0012] Secondly, embodiments of this application provide a 3D model generation apparatus, the 3D model generation apparatus comprising: The acquisition module is used to acquire the initial text and initial image input by the user; The first generation module is used to generate a posture heatmap based on the initial text and the initial image; The second generation module is used to generate three views based on the fine-tuning parameters and the attitude heatmap, wherein the fine-tuning parameters are determined based on the initial text and the initial image. The processing module is used to preprocess the three views to obtain the target three views; The third generation module is used to perform 3D node processing on the target three views to generate a 3D model.

[0013] Thirdly, embodiments of this application provide an electronic device, including: a processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, it implements the steps of the 3D model generation method as described in the first aspect.

[0014] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the 3D model generation method as described in the first aspect.

[0015] Fifthly, embodiments of this application provide a computer program product, including computer instructions that, when executed by a processor, implement the steps of the 3D model generation method as described in the first aspect.

[0016] This application provides a 3D model generation method, apparatus, and related equipment. The method includes: acquiring initial text and an initial image input by a user; generating a posture heatmap based on the initial text and initial image; generating three-view drawings based on fine-tuning parameters and the posture heatmap, wherein the fine-tuning parameters are determined based on the initial text and initial image; preprocessing the three-view drawings to obtain target three-view drawings; and performing 3D node processing on the target three-view drawings to generate a 3D model. In this embodiment, the 3D model can be automatically generated simply by the user inputting initial text and an initial image. The three-view drawing and 3D modeling steps are both completed automatically, eliminating the need for manual intervention in the 3D model generation process. This improves the efficiency of 3D model generation while reducing labor costs. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of a 3D model generation method provided in an embodiment of this application; Figure 2 This is a schematic diagram of the application process of a 3D model generation method provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a 3D model generation device provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] See Figure 1 , Figure 1 This is a flowchart of a 3D model generation method provided in an embodiment of this application, such as... Figure 1 As shown, the method includes the following steps: Step 101: Obtain the initial text and initial image input by the user.

[0021] In an optional application scenario, the 3D model generation method provided in this application embodiment can be applied to an electronic terminal, where users can input initial text and initial images on the interactive interface of the electronic terminal.

[0022] The aforementioned initial text includes, but is not limited to, information such as the virtual character's name, personality, worldview, gender, and required elements.

[0023] Step 102: Generate a posture heatmap based on the initial text and the initial image.

[0024] In this step, after receiving the initial text and initial image input by the user, an attitude heatmap is generated based on the initial text and initial image.

[0025] Optionally, the initial text and initial image can be processed based on the OpenPose pose estimation library to obtain a pose heatmap, also known as an OpenPose heatmap. It should be understood that the pose heatmap includes multiple pose key points.

[0026] Step 103: Generate three views based on the fine-tuning parameters and the attitude heatmap, wherein the fine-tuning parameters are determined based on the initial text and the initial image.

[0027] It should be noted that the fine-tuning parameters can be determined based on the initial text and initial image. For specific implementation methods, please refer to the following examples.

[0028] The fine-tuning parameters mentioned above include text-image alignment strength, denoising strength, sampling step size, and number of random seeds.

[0029] Optionally, the above text-image alignment strength can be represented as CFG Scale, with a value range of 7-12. The text-image alignment strength is used to control the semantic matching degree between the generated image and the prompt words. The higher the value, the more strictly the text description is followed.

[0030] Optionally, the above denoising intensity can be expressed as Denoise, with a value range of 0.2-0.8. The denoising intensity determines the degree of freedom to recover the image from the noise. A high value allows for more creativity, while a low value preserves the features of the original image.

[0031] Optionally, the above sampling step size can be expressed as Sampling, with a value range of Euler a / 50 steps. The sampling step size affects the generation speed and detail quality. A higher number of steps (>50) improves accuracy but increases the time consumption.

[0032] Optionally, the above-mentioned random seed number can be represented as Seed, which is a random number. A fixed seed can reproduce the same generation result and is used for batch generation consistency control.

[0033] In this step, three views can be generated based on fine-tuning parameters and attitude heatmaps. It should be understood that the three views include the front view, side view, and rear view.

[0034] Step 104: Preprocess the three views to obtain the target three views.

[0035] Step 105: Perform 3D node processing on the target three views to generate a 3D model.

[0036] In this step, after obtaining the three-view drawings, they are preprocessed to obtain the target three-view drawings. Then, 3D node processing is performed on the target three-view drawings to generate a 3D model. For details on how to generate the 3D model, please refer to subsequent embodiments.

[0037] In this embodiment, a 3D model can be automatically generated by the user inputting initial text and initial image. The three-view drawing and 3D modeling steps are completed automatically, eliminating the need for manual intervention in the 3D model generation process. This improves the efficiency of 3D model generation while reducing labor costs.

[0038] Furthermore, in this embodiment, since a 3D model can be automatically generated based on the user's input of initial text and initial image, no professional designers are required, thus lowering the technical threshold for generating a 3D model.

[0039] Optionally, generating a pose heatmap based on the initial text and the initial image includes: Semantic analysis is performed on the initial text to obtain semantic vectors, and noise reduction processing is performed on the initial image; Image encoding is performed on the initial image after noise reduction based on the semantic vector to obtain latent variables; The latent variables are decoded to obtain the target image; The target image is input into a preset painting generation model to generate a pose heatmap; The resolution of the target image is higher than that of the initial image.

[0040] Optionally, the above-mentioned painting generation model can be a Stable Diffusion model. Using the controllable mechanism, a three-view diagram is pre-imported into the Stable Diffusion model to ensure that the pose heatmap generated by the Stable Diffusion model has three-dimensional relationships.

[0041] In this embodiment, the initial text can be input into a text encoder, and the text encoder can perform semantic analysis on the initial text to obtain a semantic vector; the initial image can be denoised using a u-net network.

[0042] Furthermore, the initial image after noise reduction is image encoded using the u-net network based on semantic vectors to generate latent variables that conform to the text description.

[0043] After obtaining the latent variables, the target image is decoded into low-dimensional latent variables using a VAE decoder. Optionally, a super-resolution module can be used to improve the quality of details. The resolution of the target image is higher than that of the initial image.

[0044] The target image is then input into the preset painting generation model. Since the painting generation model imports the three-view diagram in advance, it can generate a posture heatmap with three-dimensional relationships.

[0045] In this embodiment, semantic analysis is performed on the initial text to obtain semantic vectors; image encoding is performed on the initial image after noise reduction based on the semantic vectors to obtain latent variables; image decoding is performed on the latent variables to obtain the target image; the target image is input into a preset painting generation model to generate a pose heatmap. The automatic generation of pose heatmaps can be achieved without designers spending a lot of time, which improves the efficiency of 3D model generation.

[0046] Optionally, generating three views based on the fine-tuning parameters and the attitude heatmap includes: Keyword extraction is performed on the initial text, and image feature extraction is performed on the initial image to obtain fine-tuning parameters; The attitude heatmap is adjusted using the fine-tuning parameters. The adjusted posture heatmap is input into the preset drawing generation model to generate three views.

[0047] It should be understood that the above fine-tuning parameters are provided by the initial text and the initial image. Specifically, keyword extraction is performed on the initial text, and image feature extraction is performed on the initial image to obtain the fine-tuning parameters.

[0048] As mentioned above, the fine-tuning parameters include text-image alignment strength, denoising strength, sampling stride, and number of random seeds. The text-image alignment strength controls the semantic matching degree between the generated image and the prompt words. The denoising strength determines the degrees of freedom in recovering the image from noise. The sampling stride affects both generation speed and detail quality.

[0049] In this embodiment, fine-tuning parameters are applied to adjust the posture heatmap image, and the adjusted posture heatmap is input into a preset drawing generation model to generate three views. This achieves automated generation of three views without requiring designers to spend a lot of time, thus improving the efficiency of 3D model generation.

[0050] Optionally, the three views include a front view, a side view, and a rear view, and the preprocessing of the three views to obtain the target three views includes: Adjust the image sizes of the front view, the side view, and the rear view so that the image sizes of the front view, the side view, and the rear view are the same; Channel separation and feature extraction processes are performed on the front view, the side view, and the rear view to obtain the target three-view diagram.

[0051] As mentioned above, the three views include the front view, the side view, and the rear view.

[0052] In this embodiment, after obtaining the three views, the dimensions of the three views are uniformly processed, that is, the image sizes of the front view, side view and rear view are adjusted so that the image sizes of the front view, side view and rear view are the same.

[0053] Further channel separation processing is performed on the front view, side view, and rear view. Optionally, OpenCV tools can be used to split the front view, side view, and rear view into R channel, G channel, and B channel, respectively. Feature extraction processing is then performed on the front view, side view, and rear view to obtain the target three-view image.

[0054] Optionally, the step of performing 3D node processing on the target three views to generate a 3D model includes: Perform a texture unwrapping operation on the target three-view drawing to obtain the texture coordinates of the target three-view drawing. The target three-view image is smoothed, and the missing areas of the target three-view image are repaired; The repaired target's three views are transformed into a 3D model.

[0055] After obtaining the target three-view image, a texture unrolling operation and a smoothing process are performed on the target three-view image. The texture unrolling operation is used to obtain the texture coordinates of the target three-view image, and the smoothing process is used to repair the missing areas of the target three-view image.

[0056] Then, the repaired target three-view drawing is transformed into a 3D model, thus automatically generating a 3D model without the need for designers to use specialized 3D modeling software, thereby improving the efficiency of 3D model generation.

[0057] For a better understanding of the overall technical solution, please refer to [link / reference]. Figure 2 ,like Figure 2 As shown, the system receives initial text and an initial image input from the user; generates fine-tuning parameters based on the initial text and initial image, and obtains a target image based on the initial text and initial image; inputs the target image into the Stable Diffusion model to generate an attitude heatmap; adjusts the attitude heatmap based on the fine-tuning parameters, and inputs the adjusted attitude heatmap into the Stable Diffusion model to generate three views; and calls the Comfy native 3D pipeline processing module in the Stable Diffusion model to process the three views, thereby generating a 3D model.

[0058] The Comfy native 3D pipeline processing module includes a data input and preprocessing layer, a core generation engine layer, and a post-processing and output layer.

[0059] The above data input and preprocessing layer imports batches of 2D multi-view images, i.e., three-view images. Standardized camera parameters (azimuth / tilt / focal length) are generated by points to ensure viewpoint consistency.

[0060] The aforementioned core generation engine layer is used to achieve real-time rendering through a differentiable Gaussian distribution and supports dynamic optimization.

[0061] The post-processing and output layers described above achieve normal smoothing, hole filling, and UV unwrapping. Normal smoothing eliminates jagged edges using the Laplacian smoothing algorithm, and hole filling repairs missing areas using the Poisson reconstruction algorithm. UV unwrapping automatically unwrappes texture coordinates based on the ABF++ algorithm.

[0062] This application also provides a 3D model generation apparatus; for details, please refer to [link to specific details]. Figure 3 , Figure 3 This is a schematic diagram of the structure of a 3D model generation device provided in an embodiment of this application, as shown below. Figure 3 As shown, the 3D model generation device 300 includes: The acquisition module 301 is used to acquire the initial text and initial image input by the user; The first generation module 302 is used to generate a posture heatmap based on the initial text and the initial image; The second generation module 303 is used to generate three views based on the fine-tuning parameters and the attitude heatmap, wherein the fine-tuning parameters are determined based on the initial text and the initial image. Processing module 304 is used to preprocess the three views to obtain the target three views; The third generation module 305 is used to perform 3D node processing on the target three views to generate a 3D model.

[0063] Optionally, the first generation module 302 is specifically used for: Semantic analysis is performed on the initial text to obtain semantic vectors, and noise reduction processing is performed on the initial image; Image encoding is performed on the initial image after noise reduction based on the semantic vector to obtain latent variables; The latent variables are decoded to obtain the target image; The target image is input into a preset painting generation model to generate a pose heatmap; The resolution of the target image is higher than that of the initial image.

[0064] Optionally, the second generation module 303 is specifically used for: Keyword extraction is performed on the initial text, and image feature extraction is performed on the initial image to obtain fine-tuning parameters; The attitude heatmap is adjusted using the fine-tuning parameters. The adjusted posture heatmap is input into the preset drawing generation model to generate three views.

[0065] Optionally, the three views include a front view, a side view, and a rear view, and the processing module 304 is specifically used for: Adjust the image sizes of the front view, the side view, and the rear view so that the image sizes of the front view, the side view, and the rear view are the same; Channel separation and feature extraction processes are performed on the front view, the side view, and the rear view to obtain the target three-view diagram.

[0066] Optionally, the third generation module 305 is specifically used for: Perform a texture unwrapping operation on the target three-view drawing to obtain the texture coordinates of the target three-view drawing. The target three-view image is smoothed, and the missing areas of the target three-view image are repaired; The repaired target's three views are transformed into a 3D model.

[0067] For details, see Figure 4 This application also provides an electronic device, including a bus 401, a transceiver 402, an antenna 403, a bus interface 404, a processor 405, and a memory 406.

[0068] The transceiver 402 is used to acquire the initial text and initial image input by the user; The processor 405 is configured to generate a posture heatmap based on the initial text and the initial image; Based on the fine-tuning parameters and the attitude heatmap, a three-view drawing is generated, wherein the fine-tuning parameters are determined based on the initial text and the initial image. The three views are preprocessed to obtain the target three views; The target's three views are processed using 3D nodes to generate a 3D model.

[0069] exist Figure 4 In this context, a bus architecture (represented by bus 401) is used. Bus 401 can include any number of interconnected buses and bridges, linking various circuits including one or more processors represented by processor 405 and memory represented by memory 406. Bus 401 can also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. Bus interface 404 provides an interface between bus 401 and transceiver 402. Transceiver 402 can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by processor 405 is transmitted over a wireless medium via antenna 430, which further receives data and transmits it to processor 405.

[0070] Processor 405 is responsible for managing bus 401 and general processing, and can also provide various functions, including timing, peripheral interface, voltage regulation, power management, and other control functions. Memory 406 can be used to store data used by processor 405 during operation.

[0071] Optionally, the processor 405 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a complex programmable logic device (CPLD).

[0072] This application also provides a computer-readable storage medium storing a computer program. When executed by a processor, this computer program implements the various processes of the above-described 3D model generation method embodiments and achieves the same technical effects. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0073] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0074] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0075] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A method for generating 3D models, characterized in that, The method includes: Get the initial text and initial image input by the user; Generate a posture heatmap based on the initial text and the initial image; Based on the fine-tuning parameters and the attitude heatmap, a three-view drawing is generated, wherein the fine-tuning parameters are determined based on the initial text and the initial image. The three views are preprocessed to obtain the target three views; The target's three views are processed using 3D nodes to generate a 3D model.

2. The method according to claim 1, characterized in that, The step of generating a pose heatmap based on the initial text and the initial image includes: Semantic analysis is performed on the initial text to obtain semantic vectors, and noise reduction processing is performed on the initial image; Image encoding is performed on the initial image after noise reduction based on the semantic vector to obtain latent variables; The latent variables are decoded to obtain the target image; The target image is input into a preset painting generation model to generate a pose heatmap; The resolution of the target image is higher than that of the initial image.

3. The method according to claim 1, characterized in that, The generation of three views based on the fine-tuning parameters and the attitude heatmap includes: Keyword extraction is performed on the initial text, and image feature extraction is performed on the initial image to obtain fine-tuning parameters; The attitude heatmap is adjusted using the fine-tuning parameters. The adjusted posture heatmap is input into the preset drawing generation model to generate three views.

4. The method according to claim 1, characterized in that, The three views include a front view, a side view, and a rear view. Preprocessing the three views to obtain the target three views includes: Adjust the image sizes of the front view, the side view, and the rear view so that the image sizes of the front view, the side view, and the rear view are the same; Channel separation and feature extraction processes are performed on the front view, the side view, and the rear view to obtain the target three-view diagram.

5. The method according to claim 1, characterized in that, The step of performing 3D node processing on the target three views to generate a 3D model includes: Perform a texture unwrapping operation on the target three-view drawing to obtain the texture coordinates of the target three-view drawing. The target three-view image is smoothed, and the missing areas of the target three-view image are repaired; The repaired target's three views are transformed into a 3D model.

6. The method according to any one of claims 1-5, characterized in that, The fine-tuning parameters include text-image alignment strength, denoising strength, sampling step size, and random seed number.

7. A 3D model generation device, characterized in that, The device includes: The acquisition module is used to acquire the initial text and initial image input by the user; The first generation module is used to generate a posture heatmap based on the initial text and the initial image; The second generation module is used to generate three views based on the fine-tuning parameters and the attitude heatmap, wherein the fine-tuning parameters are determined based on the initial text and the initial image. The processing module is used to preprocess the three views to obtain the target three views; The third generation module is used to perform 3D node processing on the target three views to generate a 3D model.

8. An electronic device, characterized in that, include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the 3D model generation method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the 3D model generation method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, It includes computer instructions that, when executed by a processor, implement the steps of the 3D model generation method as described in any one of claims 1-6.

Images