Three-dimensional model generation method and device, computer equipment and storage medium

Abstract

The invention discloses a three-dimensional model generation method and device, computer equipment and a storage medium. According to the method, a multi-view image set with a consistent style is generated based on an input image through computer vision and machine learning technologies; performing consistency style training on a preset low-rank adaptation module by using the multi-view image set to obtain a trained low-rank adaptation module; generating an initial three-dimensional model with a target style in combination with the trained low-rank adaptation module and a preset first three-dimensional reconstruction model; and generating an intermediate three-dimensional model based on the multi-view image set, and rendering the intermediate three-dimensional model according to a plurality of preset views to obtain a plurality of two-dimensional reference images. The method can be applied to business scenes of financial science and technology, digital medical treatment and the like, target style injection and high-fidelity detail optimization are combined, and it is ensured that the finally generated three-dimensional model accurately keeps the expected style and also presents rich geometric structures and clear textures.

Description

Technical Field

[0001] The present invention relates to the fields of three-dimensional reconstruction technology and financial technology, and in particular to a three-dimensional model generation method, device, computer equipment and storage medium. Background Art

[0002] Currently, generating a three-dimensional virtual human from a single two-dimensional image is an important task in the field of computer vision, and has broad application prospects in video games, the metaverse, virtual reality, and other fields. In the existing technology, the solution based on score distillation sampling (SDS) is one of the mainstream methods. This type of method usually uses a pre-trained two-dimensional diffusion model to generate high-quality two-dimensional images, and combines the projection relationship with the SDS loss function to guide the optimization of an initial three-dimensional model (such as a model based on three-dimensional Gaussian splatting (3DGS) or neural radiance field (NeRF)), thereby gradually generating a detailed three-dimensional virtual image, such as a three-dimensional virtual human.

[0003] However, existing methods still face challenges in generating three-dimensional virtual humans that meet specific business needs, such as generating specific brand styles or marketing campaign themes in the financial industry. Similarly, in the field of digital medicine, for example, customizing a personalized virtual rehabilitator image for an online consultation platform or rehabilitation guidance application, there is also the problem of how to efficiently achieve style customization and ensure the authenticity of the model's three-dimensional details. The standard diffusion model generates results with diverse styles, which is difficult to directly meet precise customization needs. It often requires a lot of trial-and-error adjustments or complex model retraining. In addition, how to efficiently combine the style consistency of two-dimensional images with the high-quality geometry and texture details of three-dimensional models, especially when there is only a single input image, remains a technical problem that needs to be solved. That is, the existing technology lacks a method that can efficiently generate three-dimensional models (such as virtual images) that meet specific style requirements and have high-fidelity three-dimensional details based on a small amount of input information. Summary of the Invention

[0004] Embodiments of the present invention provide a three-dimensional model generation method, apparatus, computer equipment, and storage medium, which aim to achieve the purpose of efficiently generating a three-dimensional model that meets specific style requirements and has high-fidelity three-dimensional details based on a small amount of input information.

[0005] In the first aspect, an embodiment of the present invention provides a three-dimensional model generation method, which is applied to a three-dimensional model generation system, and the method includes the following steps: generating a multi-view image set with a consistent style based on an input image; using the multi-view image set to perform consistent style training on a preset low-rank adaptation module to obtain a trained low-rank adaptation module; combining the trained low-rank adaptation module with a preset first three-dimensional reconstruction model to generate an initial three-dimensional model with a target style; generating an intermediate three-dimensional model based on the multi-view image set, and rendering the intermediate three-dimensional model according to multiple preset perspectives to obtain multiple two-dimensional reference images; refining the multiple two-dimensional reference images respectively to generate multiple two-dimensional supervision images; using the multiple two-dimensional supervision images to optimize the geometry and texture of the initial three-dimensional model to obtain a target three-dimensional model.

[0006] In the second aspect, an embodiment of the present invention also provides a three-dimensional model generation device, including: a multi-view generation unit, for generating a multi-view image set with a consistent style based on an input image; a style adaptation training unit, for using the multi-view image set to perform consistent style training on a preset low-rank adaptation module to obtain a trained low-rank adaptation module; an initial model generation unit, for combining the trained low-rank adaptation module with a preset first three-dimensional reconstruction model to generate an initial three-dimensional model with a target style; an intermediate model processing unit, for generating an intermediate three-dimensional model based on the multi-view image set, and rendering the intermediate three-dimensional model according to a preset plurality of viewpoints to obtain a plurality of two-dimensional reference images; an image refining unit, for respectively refining the plurality of the two-dimensional reference images to generate a plurality of two-dimensional supervision images; a model optimization unit, for using the plurality of the two-dimensional supervision images to optimize the geometry and texture of the initial three-dimensional model to obtain a target three-dimensional model.

[0007] In a third aspect, an embodiment of the present invention further provides a computer device comprising a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0008] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the method described in the first aspect can be implemented.

[0009] Embodiments of the present invention provide a method, apparatus, computer device, and storage medium for generating a three-dimensional model. The method comprises: generating a multi-view image set with a consistent style based on an input image; using the multi-view image set to perform consistent style training on a preset low-rank adaptation module to obtain a trained low-rank adaptation module; combining the trained low-rank adaptation module with a preset first three-dimensional reconstruction model to generate an initial three-dimensional model with a target style; generating an intermediate three-dimensional model based on the multi-view image set, rendering the intermediate three-dimensional model according to multiple preset perspectives to obtain multiple two-dimensional reference images; refining the multiple two-dimensional reference images to generate multiple two-dimensional supervisory images; and optimizing the geometry and texture of the initial three-dimensional model using the multiple two-dimensional supervisory images to obtain a target three-dimensional model.

[0010] The embodiment of the present invention first overcomes the limitation of insufficient single input information by generating a multi-view image set with consistent style based on the input image, and lays the foundation for the subsequent accurate extraction of style and reliable reference of three-dimensional details. Then, the low-rank adaptation module of the multi-view image set is used to perform consistent style training, and the trained low-rank adaptation module is combined with the first three-dimensional reconstruction model to generate an initial three-dimensional model with the target style. This series of operations realizes the efficient capture and injection of specific artistic styles, avoids the complex and time-consuming retraining of complete models for different styles, significantly improves the efficiency and flexibility of style customization, and solves the problems of difficult and inefficient style customization in the background technology.

[0011] To ensure high-detail quality in the final model, this embodiment also generates an intermediate 3D model based on a multi-view image set. This intermediate 3D model is then rendered to produce a 2D reference image, which is then further refined to generate a 2D supervision image. This path focuses on generating a 2D visual reference rich in high-quality geometric and textural detail, providing strong guidance for subsequent 3D model optimization.

[0012] Finally, by utilizing the two-dimensional supervisory image, the geometry and texture of the initial three-dimensional model are optimized to obtain the target three-dimensional model. This method fuses the initial model, which already has the target style, with high-quality detail information. This strategy of separating style injection and detail enhancement to a certain extent and then performing targeted optimization ensures that the target three-dimensional model ultimately generated can not only accurately maintain the specific artistic style expected of the input image, but also present rich geometric structures and clear texture details. Therefore, this embodiment of the invention can efficiently generate a high-quality three-dimensional model that meets specific style requirements and has high-fidelity three-dimensional details with only a small amount of input information, effectively resolving the contradiction between style and details in the background technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0014] Figure 1 A schematic diagram of a flow chart of a three-dimensional model generation method provided in an embodiment of the present invention;

[0015] Figure 2 A schematic diagram of a sub-process of a three-dimensional model generation method provided by an embodiment of the present invention;

[0016] Figure 3 A schematic block diagram of a three-dimensional model generating apparatus provided in an embodiment of the present invention;

[0017] Figure 4 A schematic block diagram of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0018] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0019] It will be understood that when used in this specification and the appended claims, the terms “comprises” and “comprising” indicate the presence of described features, integers, steps, operations, elements and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0020] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the present invention. As used in the specification and appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0021] It should be further understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0022] See also Figure 1 and Figure 2 , Figure 1 A schematic diagram of a flow chart of a three-dimensional model generation method provided in an embodiment of the present invention. Figure 2 This is a schematic diagram of a sub-process for a 3D model generation method according to an embodiment of the present invention. This 3D model generation method is applicable to 3D model generation systems, such as virtual customer service generation in the financial industry or digital healthcare, and avatar customization in metaverse marketing activities.

[0023] In existing technologies, generating 3D virtual humans or 3D objects from single-view images faces a core challenge: how to precisely control the specific artistic style of the final 3D model, such as color, material, and clothing features, while ensuring that the model has rich geometric details and high-definition textures. Traditional approaches, such as training large-scale 3D generative models from scratch for a specific style, consume enormous computing resources and have a long cycle, making them unsuitable for rapid customization. Relying solely on simple style transfer techniques often results in the 3D model losing important geometric and texture details while maintaining the style, making the model appear lifeless and unable to meet the needs of, for example, financial institutions for high-quality, personalized virtual images in digital marketing.

[0024] The three-dimensional model generation method proposed in this embodiment aims to effectively solve the above-mentioned problem through a series of collaborative steps, that is, to generate a high-quality three-dimensional model with a specific style and rich details from a single or a small number of input images at a lower cost and higher efficiency.

[0025] Figure 1 FIG. 1 is a flow chart of a method for generating a three-dimensional model according to an embodiment of the present invention. As shown in the figure, the method includes the following steps S100-600.

[0026] Step S100: generating a multi-view image set with consistent style based on an input image.

[0027] In this example, a bank needs to customize a virtual financial advisor for its wealth management business. Designers provide a pre-designed front-facing image of the avatar as input. This image embodies the bank's desired professional and approachable brand image, including specific clothing style, facial features, and overall color palette.

[0028] After receiving the input image, the 3D model generation system (hereinafter referred to as the system) feeds it into a pre-trained multi-view diffusion model. This multi-view diffusion model understands the 3D structural information of the input image and generates images of the virtual character from different perspectives, such as four different perspectives: 45 degrees to the left, 45 degrees to the right, and from the back. These generated multi-view images maintain the same stylistic features as the input image, including clothing style, color saturation, and lighting conditions, forming a set of multi-view images with a consistent style.

[0029] Step S200: Performing consistent style training on a preset low-rank adaptation module using the multi-view image set to obtain a trained low-rank adaptation module.

[0030] In this embodiment, the system uses the multi-view image set generated in step S100 as training data to train the low-rank adaptation module. The low-rank adaptation module learns and encodes the specific style information of the bank's virtual financial advisor by analyzing the common features in these images, including brand color preferences, clothing texture features, facial expression style, etc. After training, the module can parameterize these style features to form a reusable style encoding. The preset low-rank adaptation module here refers to a low-rank adaptation module whose structure has been defined and whose parameters are in an initial state before performing specific style training, and is ready to be used to learn and encode a specific style.

[0031] Step S300: combining the trained low-rank adaptation module with a preset first three-dimensional reconstruction model to generate an initial three-dimensional model with a target style.

[0032] In this embodiment, the system combines the trained low-rank adaptation module with the pre-trained first 3D reconstruction model. Specifically, the parameters of the low-rank adaptation module are injected into the network structure of the first 3D reconstruction model, enabling the originally general 3D reconstruction model to generate 3D models of a specific style.

[0033] Using this fused model to process the multi-view image set, the system generated an initial 3D virtual financial advisor model. This initial 3D model successfully maintained the bank's desired brand style, but details such as facial texture and clothing folds needed improvement. The pre-trained first 3D reconstruction model, or pre-set first 3D reconstruction model, refers to a pre-selected, pre-trained deep learning model with general 3D reconstruction capabilities.

[0034] Step S400: Generate an intermediate three-dimensional model based on the multi-view image set, and render the intermediate three-dimensional model according to multiple preset view angles to obtain multiple two-dimensional reference images.

[0035] In this embodiment, to obtain reference information for detail enhancement, the system concurrently uses another 3D reconstruction process focused on geometric and textural details. The same multi-view image set is input into the second 3D reconstruction model to generate an intermediate 3D model. While this intermediate 3D model may not be as stylistically consistent as the initial 3D model, it performs better in terms of geometric accuracy and textural detail.

[0036] The system renders the intermediate 3D model from multiple preset standard viewpoints, such as front, left, right, and back, to generate a set of 2D reference images. These reference images contain rich details and provide important references for subsequent optimization.

[0037] Step S500: Refine the plurality of two-dimensional reference images respectively to generate a plurality of two-dimensional supervision images.

[0038] In this embodiment, to further enhance detail quality, the system refines the 2D reference image obtained in step S400. This process enhances visual quality metrics such as texture detail and edge clarity in the reference image through image enhancement. The refined image, serving as the 2D supervisory image, contains high-quality detail information, such as clear facial features and realistic clothing textures.

[0039] Step S600: Utilize the plurality of two-dimensional supervisory images to optimize the geometry and texture of the initial three-dimensional model to obtain a target three-dimensional model.

[0040] In this embodiment, the system uses the high-quality two-dimensional supervision image generated in step S500 to optimize the initial three-dimensional model generated in step S300. The specific process is as follows:

[0041] First, the system renders the initial 3D model from the same perspective as the 2D supervision image to obtain the corresponding rendered image. Then, the difference between the rendered image and the 2D supervision image is calculated, which reflects the shortcomings of the initial 3D model in detail representation.

[0042] Based on the calculated differences, the system uses an optimization algorithm to adjust the geometric and texture parameters of the initial 3D model. This optimization process is iterative, with each iteration reducing the difference between the rendered image and the supervised image, gradually improving the detail of the 3D model.

[0043] After multiple rounds of optimization, the system ultimately generated the target 3D model. This model not only perfectly retained the bank's desired brand style, but also featured rich details such as realistic facial expressions, fine clothing textures, and natural light and shadow effects.

[0044] In summary, this embodiment, through the collaborative work of the above steps, firstly efficiently injects the target style into the initial 3D model through the low-rank adaptation module, then independently generates high-quality 2D detail references, and finally uses these references to supervise and optimize the initial model, thereby effectively solving the problem of single-view in the background technology. Figure 3 When generating 3D models, it is difficult to balance the contradiction between specific style customization and high-detail presentation, which provides an effective way for industries such as finance to quickly generate high-quality, personalized 3D virtual images.

[0045] In another embodiment, in addition to generating virtual customer service or virtual financial advisors in the financial industry, this method can also be applied to digital healthcare scenarios. For example, a professional virtual doctor assistant image can be customized for a large general hospital's online consultation platform. This virtual doctor assistant can be used in scenarios such as the hospital's medical guidance system, initial patient consultations, health education, medication guidance, or postoperative recovery tracking. By providing friendly and professional interactions, it improves the patient service experience and the efficiency of medical information transmission, and alleviates the consultation pressure on medical staff. In this embodiment, a front-facing concept design of the virtual doctor assistant provided by the hospital is used as the input image. The system first generates a set of images with multiple perspectives and consistent styles of the doctor's assistant based on the input image; then, it uses this image set to train a low-rank adaptation module to capture its unique professional style, and combines the trained module with the first 3D reconstruction model to generate an initial 3D model with the initial style of the doctor; at the same time, the system also generates a more detailed intermediate 3D model based on the multi-perspective image set through the second 3D reconstruction model, and renders a 2D reference image; these 2D reference images are refined to enhance details to form multiple 2D supervision images; finally, using these high-quality 2D supervision images, the geometry and texture of the initial 3D model are carefully optimized, thereby obtaining the target virtual doctor assistant model that maintains the professional doctor's style and presents rich 3D details.

[0046] In one embodiment, step S100, generating a multi-view image set having a consistent style based on an input image, includes:

[0047] The input image is input into a multi-view diffusion model, and a plurality of images of the input image at different viewpoints and consistent with the style of the input image are generated by the multi-view diffusion model to obtain the multi-view image set.

[0048] In this embodiment, the 3D model generation system first receives a front-facing image of the virtual financial advisor provided by the designer. This input image serves as the foundation for all subsequent generation, accurately defining the virtual financial advisor's core visual elements and the desired brand perception.

[0049] The system takes this single frontal image as input and passes it to a pre-trained multi-view diffusion model. This model has been trained on a large number of images of diverse characters, occupations, and styles, along with their corresponding multi-view data. This enables it to infer 3D structure from a single image and generate images from new perspectives. In this case, the model understands the professional characteristics, expected professional image, and general geometric form of a virtual financial advisor.

[0050] The multi-view diffusion model begins its generative process after receiving a frontal image of the virtual financial advisor. The model's internal mechanisms capture stylistic features defined in the input image, such as the style of the business suit, facial expression, and overall color tone, and rigorously maintain these features when generating new views.

[0051] The advantage of the multi-view diffusion model lies in its ability to maintain a high degree of stylistic consistency when generating images from different perspectives. Specifically, the model continuously references the stylistic features of the input image during the generation process. For example, when generating an image from a 45-degree left perspective, the model must not only correctly render the person's side profile but also ensure that the color and glossiness of the clothing are consistent with the front view. Similarly, when generating a back view, although the details of the back cannot be directly observed from the input image, the model will reasonably infer and generate a back image that matches the overall style based on the learned prior knowledge.

[0052] In this example, the multi-view diffusion model generates four images of the virtual wealth advisor from different perspectives: front, 45-degree left, 45-degree right, and back. These four images together constitute the multi-view image set. Each generated image strictly maintains the same stylistic features as the input image, reproducing the core visual elements of the virtual wealth advisor and the desired brand perception defined in the original input image.

[0053] The multi-view image set generated in this way provides a rich and consistent source of information for subsequent 3D reconstruction. Compared to traditional single-view reconstruction, multi-view information significantly reduces ambiguity in 3D reconstruction and improves reconstruction quality. Furthermore, because images from all viewpoints maintain a high degree of stylistic consistency, the resulting 3D model retains the desired brand image from any angle, ensuring visual consistency across various virtual wealth advisor application scenarios.

[0054] In this example, the multi-view diffusion model uses a Transformer-based architecture to capture and preserve style features. When the model receives a frontal image of a virtual financial advisor, it first converts the image into a high-dimensional feature representation through an encoder network. The Transformer-based attention mechanism plays a key role in this process: it automatically identifies and focuses on important areas in the image, such as the advisor's face, key clothing features like lapels and buttons, and the overall color scheme.

[0055] Specifically for this application scenario, when the model analyzes a frontal image of a financial advisor, the attention mechanism assigns different weights to different image regions. For example, for a key stylistic element like a dark blue suit, the model forms a strong representation in the feature space, recording information such as the suit's specific hue (RGB values), material feel (inferred through light and shadow), and tailoring style (identified through contour lines). Furthermore, for facial features, the model captures subtle facial expressions that convey professionalism yet friendliness, such as a slightly upturned mouth corner and a confident gaze.

[0056] When generating images from new perspectives, the model uses a conditional generative process to ensure stylistic consistency. When generating side views, the model doesn't simply rotate the image; instead, it understands the suit's structure in three dimensions. It knows how the shoulder line should appear from the side, how the lapel's three-dimensionality should appear, and how to maintain the same dark blue hue across different lighting angles. This understanding ensures that the generated side views adhere to realistic perspective while perfectly preserving the stylistic intent of the original design.

[0057] In one embodiment, step S200, performing consistent style training on a preset low-rank adaptation module using the multi-view image set to obtain a trained low-rank adaptation module, includes:

[0058] The multi-view image set is used as training data, and the low-rank adaptation module is trained by fine-tuning, so that the low-rank adaptation module learns and encodes the style features in the multi-view image set, thereby obtaining the trained low-rank adaptation module.

[0059] In this embodiment, the low-rank adaptation module used is specifically a low-rank adaptation (LoRA) module. Following the multi-view image set generated in the previous step, which contains multiple images of the virtual financial advisor from different perspectives, the system uses it as special training data to fine-tune an initial low-rank adaptation module. The low-rank adaptation module here is attached to a specific layer of a pre-trained deep learning model with image generation capabilities. The deep learning model here can be called a pre-trained model, which is also the first three-dimensional reconstruction model described in the subsequent step S300.

[0060] During fine-tuning, the original weight parameters of the pre-trained model remain frozen, and only the low-rank parameters within the low-rank adaptation module are trained and optimized. The low-rank adaptation module is configured to learn and capture consistent stylistic features shared by all images in the multi-view image collection. These stylistic features are key elements that define the virtual financial advisor's unique visual identity, such as clothing style, color scheme, material expression, facial features and expressions, lighting and atmosphere, and so on.

[0061] By iteratively training on these multi-view images containing rich style information, the low-rank adaptation module can effectively learn and encode (or compress or parameterize) these complex style features into its internal low-rank parameters, that is, the low-rank decomposition of the weight matrix. This process does not require training a large model from scratch, but instead specifically learns new styles by injecting and optimizing the low-rank adaptation matrix into specific layers in the pre-trained model, such as the attention layer or feedforward network layer in the Transformer architecture, or the relevant layers in the diffusion model U-Net structure.

[0062] After training is complete, a trained low-rank adaptation module is obtained. This trained low-rank adaptation module has now internalized the specific style information about the virtual financial advisor extracted from the multi-view image set, forming an efficient and reusable style encoding. This trained low-rank adaptation module can be regarded as a plug-in or adapter for this specific style. In subsequent steps, it can be combined with the first 3D reconstruction model to enable the model to generate an initial 3D model with the target style, that is, the specific style of the bank's virtual financial advisor, without the need to completely retrain the huge first 3D reconstruction model. This greatly reduces the cost and time of stylized customization.

[0063] In one embodiment, S300, combining the trained low-rank adaptation module with a preset first 3D reconstruction model to generate an initial 3D model having a target style, includes:

[0064] S310: Fusing the weight of the trained low-rank adaptation module with the weight of the pre-trained first three-dimensional reconstruction model to obtain a fusion model with style encoding capability.

[0065] In this embodiment, the trained low-rank adaptation module, which has already learned and encoded the specific style information of the virtual financial advisor, is used. Simultaneously, the system pre-sets a first 3D reconstruction model. This first 3D reconstruction model is a pre-trained model, i.e., the pre-trained model described above.

[0066] The weight fusion process specifically refers to applying the parameters of the low-rank adaptation matrix learned in the trained low-rank adaptation module, that is, its weights, to the corresponding layers of the first 3D reconstruction model. For example:

[0067] If the low-rank adaptation module is stored as a separable weight file, then when loading the first 3D reconstruction model, the trained low-rank adaptation module is loaded at the same time, and during the forward propagation of the model, the weight changes introduced by the low-rank adaptation module are added to the original weights of the corresponding layer of the pre-trained model.

[0068] In this way, the specific style information learned by the low-rank adaptation module is effectively injected or grafted into the first 3D reconstruction model. This allows the first 3D reconstruction model, which originally generated a general style, to now specifically generate the target style of the virtual financial advisor. This weighted fusion model is called a fused model. This fused model retains the original 3D reconstruction capabilities and understanding of image features of the first 3D reconstruction model, while also gaining the ability to accurately encode and reproduce the specific target style through the support of the low-rank adaptation module.

[0069] S320: Perform three-dimensional reconstruction on the multi-view image set using the fusion model to generate an initial three-dimensional model with a target style.

[0070] In this embodiment, after obtaining a fusion model with style encoding capability, the system uses the multi-view image set generated in step S100 as input and feeds it to the fusion model.

[0071] When processing these multi-view images, the fusion model integrates the stylistic guidance of the low-rank adaptation module. This means that when parsing the input image and generating the geometry and texture of the 3D model, the model strictly adheres to the stylistic characteristics of the virtual financial advisor encoded by the trained low-rank adaptation module. For example, when generating the model's texture map, it ensures that the clothing color is the specific dark blue previously learned by the low-rank adaptation module; when shaping the facial geometry, it tends to produce expressions that conform to the friendly and professional characteristics learned by the low-rank adaptation module.

[0072] Ultimately, the fusion model outputs an initial 3D model. This initial 3D model, in terms of 3D geometry and surface texture, reflects the target style shared by the original input image and the multi-view image set—that is, the specific brand style of the virtual financial advisor. For example, the generated initial 3D model's clothing, hairstyle, skin color, and demeanor should remain highly consistent with the input image's style when viewed from different angles.

[0073] It's important to emphasize that while this initial 3D model already captures the correct target style, its geometric detail and texture refinement may still require improvement. For example, subtle facial wrinkles and clothing fabric texture details may not be perfect. This is precisely what will require further optimization in subsequent steps. However, at this stage, ensuring style accuracy and consistency is the primary goal, which is achieved through the effective combination of the low-rank adaptation module and the initial 3D reconstruction model.

[0074] In one embodiment, step S400, generating an intermediate 3D model based on the multi-view image set, and rendering the intermediate 3D model according to a plurality of preset view angles to obtain a plurality of 2D reference images, includes:

[0075] S410: Input the multi-view image set into a preset second three-dimensional reconstruction model to generate the intermediate three-dimensional model.

[0076] In this embodiment, in parallel or independently of generating an initial 3D model with the target style in step S300, the system also executes a process for acquiring high-quality geometric and texture details. In this process, the system also uses the multi-view image set generated in step S100, namely, the multi-view images of the virtual financial advisor from the front, 45 degrees to the left, 45 degrees to the right, and back, as input.

[0077] However, this time the input object is a second 3D reconstruction model. This second 3D reconstruction model can be different from the first 3D reconstruction model mentioned in step S300. Its main design goal and optimization direction are more focused on accurately recovering 3D geometric structure and generating high-fidelity texture details from multi-view images, rather than prioritizing the injection of a specific artistic style.

[0078] By inputting the multi-view image set into this second 3D reconstruction model, the model processes these images and reconstructs an intermediate 3D model. The core advantages of this intermediate 3D model lie in its geometric accuracy and rich texture detail. Although this intermediate 3D model may not be strongly guided by the low-rank adaptation module during its generation like the initial 3D model, its overall style may tend towards realism or the model's default style, and may differ from the specific brand style of the virtual financial advisor, it can provide a high-quality geometric and texture blueprint for subsequent optimization.

[0079] S420: Render the intermediate three-dimensional model from multiple preset viewing angles to obtain multiple two-dimensional reference images.

[0080] In this embodiment, after the intermediate 3D model is generated by the second 3D reconstruction model, the system will render the intermediate 3D model from a set of multiple virtual camera perspectives that are preset or selected according to needs.

[0081] These preset perspectives can be consistent with those used when generating the multi-view image set, such as front, 45 degrees to the left, 45 degrees to the right, and back, or they can be other specific perspectives that help to show the details of the model. The rendering process projects the 3D geometric information and texture information of the intermediate 3D model onto a 2D plane, generating a series of 2D images.

[0082] These 2D images rendered from the intermediate 3D model are called 2D reference images. Because the intermediate 3D model itself has high geometric accuracy and rich texture details, these 2D reference images also inherit these advantages. They contain clearer and more detailed visual information than images directly rendered from the initial 3D model, such as clearer contour edges and richer surface details.

[0083] This set of multiple 2D reference images will serve as the basis for the subsequent refinement process in step S500 and will ultimately be used to generate 2D supervisory images that guide the optimization of the initial 3D model. Their core value lies in providing a high-quality, detailed 2D visual reference.

[0084] In one embodiment, step S500, refining the plurality of two-dimensional reference images to generate a plurality of two-dimensional supervisory images, includes:

[0085] S510 , adding preset noise to the plurality of two-dimensional reference images respectively to obtain a plurality of noisy images.

[0086] In this embodiment, the detailed 2D reference images generated in step S420 are multiple 2D images rendered from the intermediate 3D model. To further improve the quality of these images, particularly to enhance their detail and repair minor artifacts that may have occurred during the rendering process, the system first performs a pre-processing step on these 2D reference images: adding noise.

[0087] For each 2D reference image, the system applies a predetermined level of random noise. The type (e.g., Gaussian) and intensity (noise level) of this noise are controllable parameters. The purpose of adding noise is to better stimulate the diffusion model to produce richer, more natural details during the subsequent denoising step. This process of destruction followed by reconstruction improves image quality.

[0088] After this step, each 2D reference image is transformed into a corresponding noisy image. These noisy images are visually slightly blurrier or grainier than the original 2D reference image, but they are ready for the next refinement step.

[0089] S520: Denoise the multiple noisy images to generate multiple two-dimensional supervisory images with enhanced details.

[0090] After obtaining the noisy images, the system feeds them one by one into a pre-trained diffusion model. This diffusion model is used for image denoising and image super-resolution or detail enhancement tasks. After training on a large number of clean images and their corresponding noisy counterparts, it learns how to recover clean, sharp, and detail-rich images from noisy ones.

[0091] When a noisy 2D reference image is fed into the diffusion model, it performs its denoising process. During this process, the diffusion model goes beyond simply removing noise. Based on its learned prior knowledge of the image, it also understands and reconstructs the image's content. This allows it to generate finer details than the original input (i.e., the unnoised 2D reference image) and repair potential flaws, improving the overall visual quality of the image. For example, it can make the texture of clothing more visible, subtle facial features more vivid, and edges sharper.

[0092] The images output after the diffusion model processing are the two-dimensional supervised images. Compared to the previous two-dimensional baseline images, these two-dimensional supervised images have significantly reduced noise levels, richer and clearer details, and enhanced visual elements such as textures and edges. They have higher overall visual quality and may appear more natural and realistic.

[0093] This set of refined 2D supervision images, generated from multiple viewpoints, serves as a powerful guide for optimizing the geometry and texture of the initial 3D model in step S600 due to their high quality and rich detail. They provide critical visual supervision for ultimately generating a high-quality target 3D model.

[0094] In one embodiment, step S600, optimizing the geometry and texture of the initial three-dimensional model using the plurality of two-dimensional supervisory images to obtain a target three-dimensional model, includes:

[0095] S610: Render the initial three-dimensional model from multiple preset viewing angles to obtain multiple rendered images.

[0096] In this embodiment, the system first obtains the initial 3D model generated in step S300. This initial 3D model already has the target style guided by the low-rank adaptation module, such as the specific brand style of the virtual financial advisor, but may still lack geometric details and texture refinement.

[0097] To optimize this initial 3D model, the system renders it from a set of preset virtual camera perspectives. These preset perspectives should be strictly consistent with the perspectives used when generating the 2D supervision image in step S520, that is, the perspectives from which the intermediate 3D model was rendered in step S420. This ensures that when subsequently calculating the loss, the rendered image can be directly compared with the 2D supervision image from the corresponding perspective.

[0098] Through this rendering process, the system generates a set of 2D images generated from the initial 3D model at multiple different perspectives, which are called rendered images. Each rendered image corresponds to a specific perspective and reflects the geometric performance and texture appearance of the current initial 3D model from that perspective.

[0099] S620: Calculate a loss function between the plurality of rendered images and the two-dimensional supervision images of corresponding perspectives.

[0100] In this embodiment, the system compares each rendered image generated in step S610 with the two-dimensional supervision image of its corresponding perspective generated in step S520.

[0101] The comparison is performed by calculating a loss function between them. This loss function quantifies the difference between the rendered image and the 2D supervision image. For multiple images from different viewpoints, the loss values ​​calculated for each viewpoint are weighted averaged or summed to obtain a total loss value. This total loss value comprehensively reflects the overall difference between the current initial 3D model and the high-quality 2D supervision image across all supervision viewpoints. The greater the difference, the higher the loss value.

[0102] S630: Based on the loss function, optimize the geometric and texture parameters of the initial three-dimensional model by gradient descent to obtain the target three-dimensional model.

[0103] In this embodiment, after calculating the value of the loss function, the system uses a gradient-based optimization algorithm to adjust the parameters of the initial three-dimensional model to minimize the loss function.

[0104] The parameters of the initial 3D model include:

[0105] Geometric parameters: For example, if the model is represented by a point cloud, mesh, Neural Radiance Field (NeRF), or 3D Gaussian Splatting, the geometric parameters may include the coordinates of the points, the positions of the mesh vertices, the network weights of the NeRF, the center position / rotation / scale of the Gaussian sphere, etc.

[0106] Texture parameters: for example, pixel values ​​of a texture map, vertex colors, network weights controlling color in a neural field, color / transparency of a Gaussian sphere, etc.

[0107] The optimization process is iterative. In each iteration, based on the parameters of the current initial 3D model, S610 is executed to render a rendered image; S620 is executed to calculate the loss between the rendered image and the 2D supervision image; the gradient of the loss function with respect to the geometric and texture parameters of the initial 3D model is calculated; and based on the calculated gradient, the parameters of the initial 3D model are updated according to the rules of the optimization algorithm.

[0108] By continuously iterating this process, the geometric shape of the model will gradually approach the precise form reflected by the two-dimensional supervision image, and the texture will become clearer and richer, thereby gradually reducing the difference between the rendered image and the two-dimensional supervision image.

[0109] When the loss function converges to a certain degree or reaches a preset number of iterations, the optimization process stops. The three-dimensional model obtained at this time after geometry and texture optimization is the target three-dimensional model to be generated by the embodiment of the present invention. This target three-dimensional model not only maintains the correct target style originally injected by the low-rank adaptation module, but also through the guidance of the two-dimensional supervision image, its geometric details, such as subtle changes in facial expressions, the naturalness of clothing folds and texture quality, such as the realism of the skin and the clarity of the material, have been significantly improved, thereby achieving high-quality three-dimensional model generation.

[0110] Figure 3 This is a schematic block diagram of a three-dimensional model generation device provided by an embodiment of the present invention. Figure 3 As shown, corresponding to the above three-dimensional model generation method, the present invention also provides a three-dimensional model generation device 100. The device can be configured in a terminal such as a desktop computer, a tablet computer, a laptop computer, etc. Figure 3 The apparatus includes a multi-view generation unit 110, a style adaptation training unit 120, an initial model generation unit 130, an intermediate model processing unit 140, an image refining unit 150, and a model optimization unit 160.

[0111] A bank needed to customize a virtual financial advisor for its wealth management business. Designers provided a pre-designed frontal image of the avatar as input. This image embodied the bank's desired professional and approachable brand image, including specific clothing style, facial features, and overall color palette.

[0112] The multi-perspective generation unit 110 is used to generate a set of multi-perspective images with a consistent style based on an input image. Specifically, the multi-perspective generation unit 110 is configured to input the input image into a pre-trained multi-perspective diffusion unit after receiving it. The multi-perspective diffusion unit is capable of understanding the three-dimensional structural information of the input image and generating images of the virtual character from different perspectives, such as generating images from four different perspectives, such as 45 degrees to the left, 45 degrees to the right, and the back. These generated multi-perspective images maintain style features consistent with the input image, including clothing style, color saturation, lighting conditions, etc., forming a set of multi-perspective images with a consistent style.

[0113] The style adaptation training unit 120 is used to perform consistent style training on the preset low-rank adaptation module using the multi-view image set to obtain a trained low-rank adaptation module. Specifically, the style adaptation training unit 120 is configured to use the multi-view image set generated by the multi-view generation unit 110 as training data to train the low-rank adaptation module. The low-rank adaptation module learns and encodes the specific style information of the bank's virtual financial advisor by analyzing the common features in these images, including brand color preferences, clothing texture features, facial expression style, etc. After training is completed, the module can parameterize these style features to form a reusable style encoding.

[0114] The initial model generation unit 130 is configured to combine the trained low-rank adaptation module with the pre-trained first 3D reconstruction model to generate an initial 3D model with a target style. The initial model generation unit 130 is configured to combine the trained low-rank adaptation module with the pre-trained first 3D reconstruction unit. Specifically, the parameters of the low-rank adaptation module are injected into the network structure of the first 3D reconstruction unit, enabling the previously general 3D reconstruction model to generate 3D models with a specific style.

[0115] Using this fused unit to process the multi-view image set, the initial model generation unit 130 generates an initial 3D virtual financial advisor model. This initial 3D virtual financial advisor model successfully maintains the bank's desired brand style, but details such as facial texture and clothing wrinkles need improvement.

[0116] The intermediate model processing unit 140 is configured to generate an intermediate 3D model based on the multi-view image set and render the intermediate 3D model according to multiple preset viewpoints to obtain multiple 2D reference images. The intermediate model processing unit 140 is configured to input the same multi-view image set into the second 3D reconstruction unit to generate an intermediate 3D model. While this intermediate 3D model may not be as stylistically consistent as the initial 3D model, it may exhibit superior geometric accuracy and texture detail.

[0117] The intermediate model processing unit 140 renders the intermediate 3D model from multiple preset standard viewpoints, such as the front, left, right, and back, to generate a set of 2D reference images. These reference images contain rich details and provide important references for subsequent optimization.

[0118] Image refining unit 150 is configured to refine each of the multiple 2D reference images to generate multiple 2D supervisory images. Image refining unit 150 is configured to refine the 2D reference images obtained by intermediate model processing unit 140. This process enhances visual quality metrics such as texture detail and edge clarity in the reference images through image enhancement. The refined images, serving as 2D supervisory images, contain high-quality detail information, such as clear facial features and realistic clothing textures.

[0119] The model optimization unit 160 is configured to optimize the geometry and texture of the initial 3D model using the plurality of 2D supervisory images to obtain a target 3D model. The model optimization unit 160 optimizes the initial 3D model generated by the initial model generation unit 130 using the high-quality 2D supervisory images generated by the image refinement unit 150. The specific process is as follows:

[0120] First, the model optimization unit 160 renders the initial 3D model from the same perspective as the 2D supervisory image to obtain a corresponding rendered image. Then, the difference between the rendered image and the 2D supervisory image is calculated, which reflects the shortcomings of the initial 3D model in detail representation.

[0121] Based on the calculated differences, the model optimization unit 160 adjusts the geometric and texture parameters of the initial 3D model using an optimization algorithm. This optimization process is iterative, with each iteration reducing the difference between the rendered image and the supervised image, gradually improving the detail of the 3D model.

[0122] After multiple rounds of optimization, the target 3D model was finally obtained. This model not only perfectly maintains the bank's desired brand style characteristics, but also has rich details such as realistic facial expressions, fine clothing textures, and natural light and shadow effects.

[0123] In summary, this embodiment, through the collaborative work of the above units, firstly efficiently injects the target style into the initial 3D model through the low-rank adaptation module, then independently generates high-quality 2D detail references, and finally uses these references to supervise and optimize the initial model, thereby effectively solving the problem of single-view in the background technology. Figure 3 When generating 3D models, it is difficult to balance the contradiction between specific style customization and high-detail presentation, which provides an effective way for industries such as finance to quickly generate high-quality, personalized 3D virtual images.

[0124] It should be noted that those skilled in the art can clearly understand that the specific implementation process of the above-mentioned device and each unit can refer to the corresponding description in the aforementioned method embodiment. For the convenience and brevity of description, it will not be repeated here.

[0125] The above device can be implemented in the form of a computer program. The computer program can be used in Figure 4 Runs on the computer device shown.

[0126] See also Figure 4 , Figure 4 This is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device 500 can be a terminal or a server. The terminal can be a smart phone, tablet computer, laptop computer, desktop computer, personal digital assistant, wearable device, or other electronic device with communication capabilities. The server can be a standalone server or a server cluster consisting of multiple servers.

[0127] See Figure 4 The computer device 500 includes a processor 502 , a memory, and a network interface 505 connected via a system bus 501 , wherein the memory may include a non-volatile storage medium 503 and an internal memory 504 .

[0128] The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions, which, when executed, may cause the processor 502 to perform a method.

[0129] The processor 502 is used to provide computing and control capabilities to support the operation of the entire computer device 500.

[0130] The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503 . When the computer program 5032 is executed by the processor 502 , the processor 502 may execute a method.

[0131] The network interface 505 is used to communicate with other devices through the network. Figure 4 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present application, and does not constitute a limitation on the computer device 500 to which the solution of the present application is applied. The specific computer device 500 may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0132] The processor 502 is configured to run a computer program 5032 stored in the memory to implement the three-dimensional model generation method according to an embodiment of the present invention.

[0133] It should be understood that in the embodiment of the present application, the processor 502 may be a central processing unit (CPU), and the processor 502 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0134] Those skilled in the art will appreciate that all or part of the steps in the method of the above-described embodiment can be implemented by instructing the relevant hardware through a computer program. The computer program includes program instructions, which can be stored in a storage medium that is computer-readable. The program instructions are executed by at least one processor in the computer system to implement the steps in the method of the above-described embodiment.

[0135] Therefore, the present invention further provides a storage medium. The storage medium may be a computer-readable storage medium. The storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed by a processor, the three-dimensional model generation method according to an embodiment of the present invention is implemented.

[0136] The storage medium may be any computer-readable storage medium that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disk.

[0137] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the composition and steps of each example according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0138] In the several embodiments provided herein, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the various units is merely a logical functional division, and actual implementation may employ other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be omitted or not implemented.

[0139] The steps in the methods of the embodiments of the present invention may be adjusted in order, combined, or deleted as needed. The units in the devices of the embodiments of the present invention may be combined, divided, or deleted as needed. Furthermore, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit.

[0140] If this integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the existing technology, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, terminal, or network device, etc.) to execute all or part of the steps of the method described in various embodiments of the present invention.

[0141] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and such modifications or substitutions are intended to be within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A three-dimensional model generation method, the method being applied to a three-dimensional model generation system, characterized in that: The method comprises the following steps: Generate a set of multi-view images with consistent style based on the input image; Performing consistent style training on a preset low-rank adaptation module using the multi-view image set to obtain a trained low-rank adaptation module; Combining the trained low-rank adaptation module with a preset first three-dimensional reconstruction model to generate an initial three-dimensional model with a target style; generating an intermediate three-dimensional model based on the multi-view image set, and rendering the intermediate three-dimensional model according to a plurality of preset view angles to obtain a plurality of two-dimensional reference images; Refining the plurality of two-dimensional reference images respectively to generate a plurality of two-dimensional supervision images; The geometry and texture of the initial three-dimensional model are optimized using the plurality of two-dimensional supervisory images to obtain a target three-dimensional model.

2. The method according to claim 1, characterized in that The step of generating a multi-view image set having a consistent style based on an input image includes: The input image is input into a multi-view diffusion model, and a plurality of images of the input image at different viewpoints and consistent with the style of the input image are generated by the multi-view diffusion model to obtain the multi-view image set.

3. The method according to claim 2, characterized in that The method of performing consistent style training on a preset low-rank adaptation module using the multi-view image set to obtain a trained low-rank adaptation module includes: The multi-view image set is used as training data, and the low-rank adaptation module is trained by fine-tuning, so that the low-rank adaptation module learns and encodes the style features in the multi-view image set, thereby obtaining the trained low-rank adaptation module.

4. The method according to claim 3, characterized in that Combining the trained low-rank adaptation module with the preset first three-dimensional reconstruction model to generate an initial three-dimensional model with a target style includes: fusing the weights of the trained low-rank adaptation module with the weights of the pre-trained first 3D reconstruction model to obtain a fusion model with style encoding capability; The fusion model is used to perform three-dimensional reconstruction processing on the multi-view image set to generate an initial three-dimensional model with a target style.

5. The method according to claim 1, wherein Generating an intermediate three-dimensional model based on the multi-view image set, and rendering the intermediate three-dimensional model according to a plurality of preset view angles to obtain a plurality of two-dimensional reference images, includes: Inputting the multi-view image set into a preset second three-dimensional reconstruction model to generate the intermediate three-dimensional model; The intermediate three-dimensional model is rendered from a plurality of preset viewing angles to obtain a plurality of the two-dimensional reference images.

6. The method according to claim 5, characterized in that The refining process is performed on the plurality of two-dimensional reference images respectively to generate a plurality of two-dimensional supervision images, including: adding preset noise to the plurality of two-dimensional reference images respectively to obtain a plurality of noisy images; Denoising is performed on the multiple noisy images to generate multiple two-dimensional supervisory images with enhanced details.

7. The method according to any one of claims 1 to 6, characterized in that The method of optimizing the geometry and texture of the initial three-dimensional model using the plurality of two-dimensional supervisory images to obtain a target three-dimensional model includes: Rendering the initial three-dimensional model from multiple preset perspectives to obtain multiple rendered images; Calculating a loss function between the plurality of rendered images and the two-dimensional supervision images of corresponding perspectives; Based on the loss function, the geometric and texture parameters of the initial three-dimensional model are optimized by gradient descent to obtain the target three-dimensional model.

8. A three-dimensional model generating device, characterized in that: include: A multi-view generation unit, configured to generate a multi-view image set with consistent style based on an input image; a style adaptation training unit, configured to perform consistent style training on a preset low-rank adaptation module using the multi-view image set to obtain a trained low-rank adaptation module; an initial model generating unit, configured to combine the trained low-rank adaptation module with a preset first 3D reconstruction model to generate an initial 3D model having a target style; an intermediate model processing unit, configured to generate an intermediate three-dimensional model based on the multi-view image set, and render the intermediate three-dimensional model according to a plurality of preset view angles to obtain a plurality of two-dimensional reference images; An image refining unit, configured to perform refinement processing on the plurality of two-dimensional reference images respectively to generate a plurality of two-dimensional supervision images; The model optimization unit is used to optimize the geometry and texture of the initial three-dimensional model using the multiple two-dimensional supervision images to obtain a target three-dimensional model.

9. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images