A method for generating a three-dimensional model and related apparatus

Abstract

A method and related apparatus for generating a 3D model are disclosed. The method includes: acquiring one or more first images of a target object; acquiring the first images; inputting the first images into a first diffusion model to obtain a second image; a noise removal module of the first diffusion model for denoising based on second noise to be removed to obtain the second image; the second noise to be removed is obtained by smoothing multiple first noises belonging to a second viewpoint; and generating a target 3D model based on the first and second images. In this application, a 3D model can be generated based on images from multiple viewpoints, improving the generation quality of the 3D model of the target object and solving the problem of low generation quality of the 3D model of the target object caused by a limited number of observation viewpoints. Furthermore, the smoothing process ensures the consistency of the 3D structure of the target object in the second image.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method and apparatus for generating three-dimensional models. Background Technology

[0002] In the field of 3D model generation technology, it is necessary to provide images of the target object from multiple viewing angles to observe the target object from different perspectives, thereby improving the quality of the generated 3D model of the target object. If the number of viewing angles is insufficient, the quality of the generated 3D model of the target object will be greatly reduced. Summary of the Invention

[0003] To address the aforementioned problems, this application provides a method and apparatus for generating three-dimensional models, which solves the problem of low quality in the generation of three-dimensional models of target objects.

[0004] Therefore, the following technical solution is adopted in this application:

[0005] In a first aspect, this application provides a method for generating a three-dimensional model, which mainly includes the following steps: inputting a first image into a first diffusion model to obtain a second image; the first diffusion model includes a noise prediction module and a noise removal module, wherein the noise prediction module is used to predict a first noise to be removed, representing a second viewpoint, using the first image as a condition; the noise removal module is used to denoise based on the second noise to be removed, to obtain the second image; and generating a target three-dimensional model based on the first image and the second image. The second noise to be removed is obtained by smoothing multiple first noises belonging to a second viewpoint; the first image represents the initial image of the target object in the first viewpoint; and the second image represents the target image of the target object in the second viewpoint.

[0006] In this application, a first diffusion model is used to generate a second image from a second perspective based on a first image from a first viewpoint. Then, a target 3D model of the target object is generated based on the first and second images from different viewpoints. This allows for the generation of a 3D model from multiple viewpoints, improving the quality of the generated 3D model and solving the problem of low quality caused by a limited number of observation viewpoints. It also reduces the possibility of issues such as reconstruction holes and rendering artifacts. Furthermore, because multiple noise samples from a second viewpoint are smoothed, the 3D structure of the target object in the second image remains consistent, further improving the quality of the generated 3D model and reducing the possibility of reconstruction holes and rendering artifacts.

[0007] In one possible implementation, the method further includes: processing a third image to obtain reference noise; the third image represents an initial image of the target object in a second viewpoint; the reference noise includes noise from image inpainting in the second viewpoint and geometric noise of the target object in the second viewpoint. A noise removal module is specifically used to: denoise based on the reference noise and the second noise to be removed to obtain a second image.

[0008] In this implementation, based on the third image, at least one of the noise used for image inpainting from the second viewpoint (also referred to as the first reference noise) and the geometric noise of the target object in the second viewpoint (also referred to as the second reference noise) is obtained. The first reference noise is used to characterize the image quality of the target object in the second viewpoint, thereby improving the image quality of the second image. The second reference noise is used to provide geometric information from the second image, thereby improving the geometric structure information of the second image and further ensuring the three-dimensional consistency of different second images.

[0009] In one possible implementation, an input image is acquired; an initial 3D model is generated based on the input image; and a third image and a first image are obtained based on the initial 3D model.

[0010] In this implementation, both the third image and the first image are obtained from the user's input image, which ensures the consistency of the three-dimensional structure of the third image and the second image. The third image is obtained by performing a viewpoint transformation on the input image. The third image is a low-quality image relative to the second image. Thus, the third image can serve as a reference for the second image, thereby providing the second image with first reference noise to improve its image quality and second reference noise to prompt and guide its geometric structure, further ensuring the three-dimensional consistency of the second image from different second viewpoints.

[0011] In one possible implementation, the above-mentioned denoising based on the first reference noise and / or the second reference noise, and the second noise to be removed to obtain the second image includes: weighting the second noise to be removed and at least one of the first reference noise and the second reference noise to obtain the target noise; and denoising based on the target noise to obtain the second image.

[0012] In this implementation, the target noise is obtained by weighting at least one of the second noise to be removed and the first and second reference noises. This allows for the determination of the degree of influence of the first and second reference noises on the fusion noise through different weights, thereby enabling the second image to have different image quality and geometric information, improving the applicability and generalization of the second image to different application scenarios.

[0013] In one possible implementation, during the inverse denoising iteration of generating the first image, the weight of the first reference noise from the same second viewpoint is gradually increased; and / or, the weight of the second reference noise from the same second viewpoint is gradually decreased.

[0014] In this implementation, the weight of the first reference noise from the same second viewpoint gradually increases. Thus, in the later stages of the denoising process, the weight of the first reference noise is smaller, the geometric information of the target object has less geometric guidance for the denoising process, and the geometric information of the generated noisy image from the second viewpoint is already relatively complete and clear. The geometric guidance of the first reference noise is even less, which better aligns with the characteristics of the image denoising process and prevents the geometric information of the first reference noise from interfering with or adversely affecting the geometric information of the generated second image. Conversely, the weight of the second reference noise from the same second viewpoint gradually decreases. Thus, in the later stages of the denoising process, the weight of the second reference noise is larger, providing greater guidance for image restoration of the second image, resulting in a higher quality generated second image.

[0015] In one possible implementation, the above-mentioned generation of a target 3D model based on the first image and the second image includes: inputting the first image into a second diffusion model to obtain a fourth image, the fourth image representing a repaired image of the target object in a first viewpoint; and generating a target 3D model of the target object based on the fourth image and the second image.

[0016] In this implementation, a first image is input into a second diffusion model to generate one or more fourth images after image restoration. Based on the fourth and second images, a target 3D model of the target object is generated, further improving the generation quality of the target 3D model.

[0017] In one possible implementation, generating the target 3D model based on the first image and the second image includes: generating the 3D model through K iterations based on the first image and the second image; where K is a positive integer; and the k-th iteration process, where k is a positive integer less than or equal to K, includes: obtaining the first image of the current iteration step based on the 3D model of the previous iteration step; inputting the first image of the current iteration step into a first diffusion model to obtain the second image of the current iteration step; and generating the 3D model of the current iteration step based on the first image and the second image of the current iteration step.

[0018] In this implementation, the target 3D model of the target object is generated through K iterations. Thus, the image quality of the first and second images continuously improves during the K iterations, thereby ensuring the quality of the generated target 3D model of the target object.

[0019] In one possible implementation, the first perspective described above represents a perspective within the observable field of view, and the second perspective represents a perspective outside the observable field of view.

[0020] In this implementation, the first viewpoint is within the observable viewpoint range, ensuring that the first image contains partial 3D structural information of the target object, thereby generating a second image outside the observable viewpoint range. When generating a target 3D model based on the first and second images, the first image can provide 3D structural information of the target object within the observable viewpoint range, while the second image can provide 3D structural information of the target object outside the observable viewpoint range, thus ensuring the quality of the generated target 3D model. This makes the application more widely applicable, such as scenarios where the target object is only observable within a partial viewpoint range. For example, an image or video of the target object is captured within a local viewpoint range; the image or video of this local viewpoint (one or more viewpoints within the observable viewpoint range) is processed to obtain the first image.

[0021] In one possible implementation, the method further includes: acquiring an input video, the input video representing a video of the target object within an observable field of view; and processing the input video to obtain a first image.

[0022] In this implementation, compared to methods that generate 3D models based on observation videos of a panoramic target object, this application can generate a 3D model of the target object based on observation videos within a certain observable field of view of the user. This makes the application more widely applicable, for example, in scenarios where only a portion of the target object's field of view is observable. For instance, the observation video can be processed, such as through frame segmentation, with each first image representing a single frame in the observation video.

[0023] Secondly, this application provides a three-dimensional model generation apparatus, mainly comprising: a processing module for inputting a first image into a first diffusion model to obtain a second image; the first image representing an initial image of a target object in a first viewpoint; the second image representing a target image of the target object in a second viewpoint; the first diffusion model including a noise prediction submodule and a noise removal submodule, the noise prediction submodule being used to predict a first noise to be removed for representing the second viewpoint using the first image as a condition; the noise removal submodule being used to denoise according to the second noise to be removed to obtain the second image; the second noise to be removed being obtained by smoothing multiple first noises belonging to a second viewpoint; and generating a target three-dimensional model based on the first image and the second image.

[0024] In one possible implementation, the processing module is further configured to: process the third image to obtain reference noise; the third image represents the initial image of the target object in the second viewpoint; the reference noise includes at least one of noise used for image inpainting in the second viewpoint and geometric noise of the target object in the second viewpoint. The noise removal submodule is specifically configured to: denoise based on the reference noise and the second noise to be removed to obtain the second image.

[0025] In one possible implementation, the acquisition module is used to: acquire an input image; generate an initial 3D model based on the input image; and obtain a first image and a third image based on the initial 3D model.

[0026] In one possible implementation, when generating a target 3D model based on the first image and the second image, the processing module is specifically used to: input the first image into the second diffusion model to obtain a fourth image, the third image representing the repaired image of the target object in the first viewpoint; and generate a target 3D model of the target object based on the fourth image and the second image.

[0027] In one possible implementation, when generating the target 3D model based on the first image and the second image, the processing module is specifically used to: generate the 3D model through K iterations based on the first image and the second image; where K is a positive integer; and the k-th iteration process, where k is a positive integer less than or equal to K, includes: obtaining the first image of the current iteration step based on the 3D model of the previous iteration step; inputting the first image of the current iteration step into the first diffusion model to obtain the second image of the current iteration step; and generating the 3D model of the current iteration step based on the first image and the second image of the current iteration step.

[0028] In one possible implementation, the first perspective represents the perspective within the observable field of view, and the second perspective represents the perspective outside the observable field of view.

[0029] Thirdly, this application provides a chip system including a processor and a power supply circuit, the power supply circuit being used to supply power to the processor, and the processor being used to execute the method described in the first aspect and any possible implementation thereof.

[0030] Fourthly, this application provides a computing device including a processor and a memory. The processor is used to execute instructions stored in the memory to cause the computing device to perform the methods described in the first aspect and any possible implementation thereof.

[0031] Fifthly, this application provides a computing device cluster, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs the method described in the first aspect and any possible implementation thereof.

[0032] In a sixth aspect, this application provides a computer-readable storage medium including computer program instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method described in the first aspect and any possible implementation thereof, wherein the cluster of computing devices includes at least one computing device.

[0033] In a seventh aspect, this application provides a computer program product containing instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method described in the first aspect and any possible implementation thereof, wherein the cluster of computing devices includes at least one computing device.

[0034] It is understood that the beneficial effects of the second to seventh aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0035] The accompanying drawings used in the embodiments or technical description are briefly introduced below.

[0036] Figure 1 A schematic diagram illustrating a method for generating a 3D model provided in related technologies;

[0037] Figure 2a This is a schematic diagram of the composition of a three-dimensional model generation system provided in the embodiments of this application;

[0038] Figure 2b This is a schematic diagram illustrating the composition of an example of a three-dimensional model generation system provided in this application embodiment;

[0039] Figure 3a This is a schematic diagram illustrating the composition of a three-dimensional model generation framework provided in an embodiment of this application;

[0040] Figure 3b This is a schematic diagram illustrating the working process of the multi-view generation module provided in the embodiments of this application;

[0041] Figure 3c This is a schematic diagram of the denoising process of a second view at a time step of the multi-view generation module provided in the embodiments of this application;

[0042] Figure 3d This is a schematic diagram of the noise removal process provided in the embodiments of this application;

[0043] Figure 3e This is a schematic diagram illustrating image inpainting of an interpolated image using the IR model provided in the embodiments of this application;

[0044] Figure 4a This is a schematic diagram illustrating an example of applying a multi-view generation module during the 3D model generation process provided in this application embodiment;

[0045] Figure 4b These are schematic diagrams illustrating examples of various perspectives provided in the embodiments of this application;

[0046] Figure 4c This is a schematic diagram illustrating an example of the working process of the multi-view generation module provided in the embodiments of this application;

[0047] Figure 5 A flowchart illustrating a three-dimensional model generation method provided in an embodiment of this application;

[0048] Figure 6 This is a schematic diagram of the composition of a three-dimensional model generation device provided in the embodiments of this application;

[0049] Figure 7 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0050] Figure 8 This is a schematic diagram of the structure of a computing device cluster provided in an embodiment of this application;

[0051] Figure 9 This is a schematic diagram of another computing device cluster provided in an embodiment of this application. Detailed Implementation

[0052] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0053] In this article, the term "and / or" describes the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three cases: A exists alone, A and B exist simultaneously, and B exists alone. The symbol " / " in this article indicates that the related objects have an "or" relationship; for example, A / B means A or B.

[0054] The terms "first" and "second," etc., used in the specification and claims herein are used to distinguish different objects, not to describe a specific order of objects. For example, "first response message" and "second response message," etc., are used to distinguish different response messages, not to describe a specific order of response messages.

[0055] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0056] In the description of the embodiments in this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, etc.; multiple elements means two or more elements, etc. "Several" means one or more.

[0057] To facilitate understanding of the solutions provided in the embodiments of this application, a brief introduction to some of the terms involved in this solution will be given first.

[0058] Generative models are neural network models that can learn and simulate data distributions to generate new samples similar to real data. Common applications of generative models include text generation and image generation.

[0059] Diffusion models are a type of generative model that generates high-quality samples by simulating the gradual "diffusion" process of data. Originating from non-equilibrium thermodynamics, diffusion models are a type of generative model based on probability likelihood. In other words, diffusion models are a type of generative model based on deep learning. Their basic architecture includes an image encoder and an image decoder, and they have a forward noise addition process and a reverse denoising process. The forward noise addition process, or diffusion process, gradually adds noise to a real image to obtain a noisy image that follows a standard normal distribution. The reverse denoising process can be used for image generation tasks. Using the noisy image as the denoising target, the diffusion model predicts the noise in the noisy image based on the input conditional information, and then performs denoising processing on the noisy image to generate an image that removes noise and meets the conditional information.

[0060] 3D Gaussian Splatting (3DGS) is a 3D model generation method that renders an image from a specific viewpoint by the mutual occlusion of Gaussian spheres in space. 3DGS is a computer graphics method used for real-time rendering and scene reconstruction. It represents a scene as a set of one or more 3D Gaussian distributions, each with attributes such as position, covariance, and color. This technique utilizes differentiable rendering methods to accurately reconstruct complex scenes by optimizing Gaussian parameters, supporting real-time rendering and dynamic scene processing. Its advantages lie in its efficiency and flexibility, making it suitable for fields such as virtual reality and augmented reality. The 3DGS reconstruction pipeline includes the following steps: (1) Initialization: Initialize a 3D Gaussian ellipsoid using the input image and camera parameters to represent the point distribution in the scene; (2) Optimization: Adjust the parameters of the Gaussian ellipsoid (such as position, scale, and orientation) through iterative optimization algorithms (such as gradient descent) to minimize the difference between the rendered image and the input image; (3) Rendering: Project the 3D Gaussian ellipsoid onto the 2D image plane using a highly optimized rasterizer to generate a realistic rendered image.

[0061] Multiview Diffusion (MVD): By using an image from one viewpoint as a guide, the noise "diffusion" process is gradually reversed from Gaussian noise to generate one or more new viewpoint images.

[0062] Image Restoration (IR) Diffusion Model: The IR diffusion model can be understood as a conventional diffusion model used for image restoration (IR) tasks. Using a low-quality image as conditional information, the IR diffusion model recovers a high-quality original image from a noisy image by simulating a gradual process of adding and removing noise. The IR diffusion model is also known as the image restoration model.

[0063] Interpolated images: Within a known field of view, new viewpoint images are generated by interpolating multiple existing images using interpolation methods. Interpolated images are typically used to smooth transitions or supplement missing viewpoint information. The generated image's viewpoint lies between the viewpoints of multiple existing images, resulting in more realistic and natural content. It is commonly used in applications such as video frame interpolation and viewpoint completion in 3D reconstruction. Implemented through linear interpolation or other interpolation algorithms, it can preserve image details and continuity.

[0064] Extrapolation: Generating new viewpoint images outside the known viewpoint range using extrapolation methods. Extrapolation is typically used to expand the viewpoint range or generate images beyond the original viewpoint. The generated image's viewpoint exceeds the known range, and its content may have some uncertainty and distortion. It is commonly used in scenarios such as image deblurring and viewpoint expansion. Implemented through extrapolation algorithms, it can generate images with the opposite effect to interpolation, such as deblurring or enhancing contrast.

[0065] Global view / non-global view: A global view refers to one or more views that can observe the whole picture of a target object; a non-global view refers to one or more views that can observe part of the appearance of a target object from a limited observation range. A non-global view is also called a local view. For example, the first view in the embodiment of this application can refer to a non-global view or a local view. Then, a three-dimensional model of the target object is generated based on the image of the target object in the non-global view or the local view.

[0066] Target object: refers to the object generated by the three-dimensional model generation method of this application embodiment. For example, when generating a three-dimensional model of a chair according to the three-dimensional model generation method of this application embodiment, the target object is the "chair". As another example, when generating a three-dimensional model of a terminal device according to the three-dimensional model generation method of this application embodiment, the target object is the terminal device.

[0067] In the field of 3D model generation, it is necessary to provide images of a target object from one or more viewing perspectives to observe the target object from different angles, thereby improving the quality of the generated 3D structure of the target object. If the number of viewing perspectives is small, the quality of the generated 3D model of the target object will be greatly reduced, leading to low-quality problems such as holes and artifacts. Therefore, a 3D model generation method is needed to generate more second images based on a small number of first images of the target object, thereby generating a higher-quality 3D model. Here, the first image represents an image of the target object from a first-view perspective; the second image represents an image of the target object from a second-view perspective. The first and second perspectives represent different viewpoints.

[0068] Figure 1 This is a schematic diagram illustrating a method for generating a 3D model provided in related technologies. For example... Figure 1 As shown, in related technologies, a first image of the target object is input into a multiview diffusion model (MVD) to generate one or more second images of the target object. The first and second images are then stitched together to obtain images of the target object from all perspectives. Using a 3D model generation method, such as 3D Gaussian Splatting (3DGS), a 3D model of the target object is obtained based on the images from all perspectives.

[0069] In related technologies, when the input to MVD is multiple first images, MVD needs to be called multiple times. However, each call to MVD is performed independently, which results in second images generated from different first images having only a stitching relationship and no three-dimensional physical constraints. Consequently, there will be a strong three-dimensional inconsistency problem between the generated images from different perspectives, leading to poor quality of the generated three-dimensional model of the target object, such as reconstruction holes and rendering artifacts.

[0070] In view of this, embodiments of this application propose another technical solution for generating a three-dimensional model. A method for generating a three-dimensional model includes: acquiring a first image, the first image representing an image of a target object from a first viewpoint; inputting the first image into a first diffusion model to output a second image; the second image representing an image of the target object from a second viewpoint, the second viewpoint and the first viewpoint being different viewpoints; wherein, during the denoising process of the first diffusion model, first noise to be removed belonging to the same second viewpoint is averaged to obtain second noise to be removed; and, based on the second noise to be removed, a target noise is obtained, so that the first diffusion model obtains the second image based on the target noise; and a target three-dimensional model of the target object is generated based on the first image and the second image.

[0071] In this embodiment, a second image is generated based on a first image using a first diffusion model; then, a target 3D model of the target object is generated based on one or more first images and one or more second images. Because one or more noise samples from the same second perspective are processed, images generated from different first images at the same perspective maintain consistency in 3D structure, improving the generation quality of the 3D model of the target object and reducing the possibility of problems such as reconstruction holes and rendering artifacts.

[0072] To better understand the technical solutions of the embodiments of this application, the application scenarios of the technical solutions of the embodiments of this application are described below, but the description of the application scenarios does not constitute a limitation on the embodiments of this application.

[0073] Figure 2a This is a schematic diagram illustrating the composition of a three-dimensional model generation system provided in an embodiment of this application. For example... Figure 2aAs shown in this embodiment, a 3D model generation system 100 includes a terminal 110 and a server 120. For example, the terminal 110 acquires a first image input by the user, also known as the input image, and sends it to the server 120. The server 120 deploys a multi-view diffusion model and, based on the 3D model generation method of this embodiment, generates one or more second images by calling the multi-view diffusion model according to the first image. Then, based on a 3D model generation method, such as 3D Gaussian sputtering (3DGS), it generates a 3D model of the target object based on the first image and the generated one or more second images.

[0074] Optionally, server 120 may send a 3D model of the target object to terminal 110 so that the user can obtain the 3D model of the target object, and terminal 110 may display the rendering result of the 3D model.

[0075] Optionally, server 120 may include one or more servers. Figure 2a (The example includes a server), and the server 120 can provide the methods or apparatus provided in the embodiments of this application to one or more terminals.

[0076] Optionally, a relevant application may be installed on the terminal 110, and the application or webpage may provide an interface; the terminal 110 may receive the first image input by the user on the interface and send the first image to the server 120.

[0077] It should be understood that in some optional implementations, terminal 110 may also complete its work independently without the cooperation of server 120, and this application embodiment is not limited in this regard. In some optional implementations, server 120 may also rely on a first image received from other devices besides the terminal or locally stored, without the cooperation of terminal 110, and this application embodiment is not limited in this regard.

[0078] The following description Figure 2a The product form of the terminal 110. In this application embodiment, the terminal 110 can be a mobile phone, tablet computer, wearable device, in-vehicle device, augmented reality (AR) / virtual reality (VR) device, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), etc., and this application embodiment does not impose any restrictions on it.

[0079] It can be further understood that server 120 can be various types of servers, such as x89 architecture servers, specifically rack servers, blade servers, high-density servers, rack servers, or high-performance servers, etc. In other words, this application embodiment does not specifically limit the specific type of server. Furthermore, it can be understood that... Figure 2a The server structure shown does not constitute a limitation on the server structure. A server may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0080] Furthermore, server 120 can be configured as an independent physical server, or as a server cluster or distributed system consisting of multiple physical servers. It can also be configured as a cloud server or cloud server cluster that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The cloud server cluster is deployed in several cloud data centers. The software can be an application that implements object control methods, etc., but is not limited to the above forms.

[0081] In one possible scenario, server 120 can function as a cloud platform (a software platform employing application virtualization technology, integrating multiple functions such as software search, download, use, management, and backup). In practical use, server 120 can deploy a cloud management platform and a data center, with terminal 110 interacting with the cloud through the cloud management platform. Additionally, the data center can deploy nodes, which can be virtual machine instances, container instances, physical servers, etc.

[0082] In another possible scenario, the method provided in this application embodiment can be implemented by software. The software has a user interface portion and a data processing portion. Terminal 110 runs the user interface portion of the software, and server 120 runs the data processing portion of the software. During the process of terminal 110 running the user interface portion of the software, it can acquire user input data and call the data processing portion running on server 120 to process the user input data, thereby implementing the method provided in this application embodiment.

[0083] In other words, the method provided in this application embodiment can be applied to the terminal 110 or the server 120. In specific implementation, it can run as software on the terminal 110 or the server 120; for example, the software can be a service or an application. This application embodiment can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application embodiment can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can reside in local and remote computer storage media, including storage devices.

[0084] Furthermore, the systems that can implement the technical solutions of this application embodiment include cloud servers, web pages, application program interfaces (APIs), mobile application software, etc. The large language model of this application embodiment can be deployed on the server side, website, or mobile application software to meet the text generation needs of enterprise users and individual users, and can also be provided to developers for secondary development and use through open API interfaces. Furthermore, the implementation of this application embodiment mainly involves deploying the neural network model involved in this application embodiment on related platforms and applications, such as cloud platforms, websites, and applications (APPs).

[0085] Figure 2b This is a schematic diagram illustrating the composition of an example of a three-dimensional model generation system provided in an embodiment of this application. For example... Figure 2b As shown, in one example, in a 3D model generation system 100, the terminal 110 is an example of a user interface (UI) in a terminal device; the server 120 is an example of a cloud server.

[0086] In this example, the user inputs one or more images from different perspectives through a UI interface and uploads them to a cloud server. The cloud server deploys a multi-view diffusion model. Based on the multi-view image generation method of this application embodiment, one or more second images are generated from one or more first images. Based on the 3D model generation method of this application embodiment, a 3D model of the target object is generated from one or more first images and one or more second images. The cloud server can send the 3D model of the target object to the UI interface so that the user can obtain the 3D model of the target object. The 3D model can be displayed, for example, as an image from a certain rendering perspective presented to the UI interface.

[0087] In this example, the embodiments of this application can be applied to terminal systems and image library applications, enabling them to enhance the ability to upscale images to "three-dimensional" (3D). For instance, a user only needs to capture a video of a side view of a target object, which can be understood as one or more first images. The video is then uploaded to a cloud server via the network. The cloud server uses the technical solution of the embodiments of this application to generate a second image and optimize it to obtain a high-quality 3D model. The generated 3D model is then sent to the user, and the rendering result of the 3D model is displayed through a UI interface.

[0088] In this example, the embodiments of this application can be applied to various 3D model generation software, mobile phone photo albums, personal computers (PCs) and other terminal devices, and the algorithms corresponding to the embodiments of this application can be deployed on cloud servers.

[0089] Figure 3a This is a schematic diagram illustrating the composition of a three-dimensional model generation framework provided in an embodiment of this application. For example... Figure 3a As shown, a 3D model generation framework includes a multi-view generation module and a 3D generation module. The multi-view generation module uses N first images as conditional information and M noisy images as denoising objects. After denoising the M noisy images, the multi-view generation module generates M corresponding second images. The 3D generation module generates a 3D model of the target object based on the N first images and the M second images. Here, M and N are positive integers. The M noisy images are used to generate M second images; that is, one noisy image corresponds to one second viewpoint; one noisy image corresponds to one second image; one second viewpoint corresponds to one second image. Thus, one noisy image is used as the denoising object for the MVD model, and after denoising, a corresponding second image is generated. The second image represents the target image of the target object in the second viewpoint. The number of both first and second views can be one or more.

[0090] For example, the m-th noisy image is used to generate the m-th second image from a second perspective. The MVD model uses at least one of the N first images and the parameter bit condition information of the m-th second perspective to denoise the m-th noisy image. After multiple iterations of the inverse denoising process, the m-th second image from a second perspective is obtained. The parameters of the m-th second perspective are, for example, camera parameter conditions. That is, the perspective transformation process of the MVD model is assumed to be the camera's shooting process at different perspectives, such as rotation parameters and deformation parameters, to control the second perspective of the generated second image.

[0091] Optionally, the N input images from the first viewpoint are preprocessed to obtain N1 interpolated images and / or N2 extrapolated images. The extrapolated images, also known as third images, represent the initial image of the target object in the second viewpoint. Using the N1 interpolated images as the first images, the MVD model of the multi-view generation module uses these N1 interpolated images as conditional information and M noisy images as denoising objects to obtain the noise to be removed. The IR model of the multi-view generation module uses the N2 extrapolated images as conditional information and M noisy images as denoising objects to obtain the first reference noise. Based on the noise to be removed and the first reference noise, fused noise is obtained. The MVD model of the multi-view generation module generates corresponding M second images after denoising the M noisy images by fusing the noise. Here, M, N1, N2, and N are positive integers. Optionally, N1 is greater than N, thus increasing the number of input images for the MVD model, enriching the input information, and resulting in higher quality second images. Optionally, N2 is greater than or equal to M; the extrapolated image can be used as a reference for the second image, and N2 is greater than or equal to M, which can ensure that each second image corresponds to at least one extrapolated image as a reference for its image generation process.

[0092] It is understandable that a 3D model must include at least the target object. Optionally, the 3D model may also include environmental information about the target object. For example, when generating a 3D model of a "car," the target object is the "car," and the 3D model may include not only the "car" but also the ground, parking space markings, the driver, etc.

[0093] Optionally, one or more interpolated images are used as first images and input into the MVD model to generate one or more second images. An extrapolated image serves as a reference for the second images; an extrapolated image from the same viewpoint can also be used as a reference for the second images from the same viewpoint. For example, an extrapolated image from the second viewpoint m is input into the IR model to predict a first reference noise. This first reference noise provides noise for image inpainting of the second image from the second viewpoint m, thereby improving the quality of the second image from the second viewpoint m.

[0094] Optionally, the extrapolated image is also used to obtain a second reference noise after adding noise. This provides geometric information to the second image, improving its geometric structure. Furthermore, since different extrapolated images originate from the same user input image, this ensures the three-dimensional consistency of the second images from different second perspectives.

[0095] Optionally, one or more interpolated images can be used as the first image, thus increasing the number of input images for the MVD model, enriching the input information, and resulting in a higher quality second image.

[0096] Optionally, one or more first images are input into the IR model to output an image-restored first image, thereby improving the quality of the first image. The image-restored first image is also referred to as the fourth image, which represents the restored image of the target object in the first viewpoint.

[0097] Furthermore, since the number of first images is relatively small, directly generating multi-view images from the first images yields poor results. Therefore, in this embodiment, the N first-view input images are preprocessed to obtain N1 interpolated images and N2 extrapolated images. For example, an initial 3D model of the target object is generated based on the N first-view input images; then, N1 interpolated images and N2 extrapolated images are obtained based on the initial 3D model. Alternatively, a deep learning model or geometric transformation model is used to predict the unknown N1 interpolated images and N2 extrapolated images based on the known N first-view input images. Yet another example is obtaining the unknown N1 interpolated images and N2 extrapolated images by interpolating the pixels of the known N first-view input images, such as through linear interpolation, polynomial interpolation, or attention interpolation.

[0098] In contrast to extrapolated images, interpolated images represent images within the same field of view as the user's input image. This results in higher quality interpolated images, ensuring the accuracy of the noise to be removed, and thus guaranteeing the accuracy of the second image.

[0099] In this embodiment, the first viewpoint is also referred to as the observable viewpoint; since the number of first images is relatively small, it can reflect part of the three-dimensional structural information of the target object, and the first viewpoint is also referred to as the local viewpoint. The second viewpoint is also referred to as the unobserved viewpoint, and the second image refers to the viewpoint image that needs to be generated but has not been observed; since the number of second images is relatively large, it can reflect most or all of the three-dimensional structural information of the target object, and the total viewpoints corresponding to the second viewpoint and the first viewpoint are also referred to as the global viewpoint or all viewpoints.

[0100] For example, the implementation of the 3D generation module, that is, the implementation of generating a 3D model from a 2D image, includes: 3D Gaussian Splatting (3DGS) technology, stereo vision generation technology, structured light technology, point cloud data generation technology, etc., and the embodiments of this application do not limit this.

[0101] Furthermore, the multi-view generation module includes MVD and IR models. The MVD model is also known as the first diffusion model, and the IR model is also known as the second diffusion model. The multi-view generation module will be further explained below.

[0102] Figure 3b This is a schematic diagram illustrating the working process of the multi-view generation module provided in the embodiments of this application. For example... Figure 3b As shown, in the process of generating M second images, the MVD model in the multi-view generation module uses N first images to denoise M noisy images; each noisy image corresponds to a second viewpoint, which is used to generate a second image, resulting in M ​​second images with second viewpoints.

[0103] For example, the denoising process of the multi-view generation module is divided into T time steps, where T is a positive integer. In the first time step, the multi-view generation module uses N first images as conditional information, performs viewpoint transformation using the MVD model, and denoises M noisy images (T) to obtain M noisy images (T-1) corresponding to the second viewpoint; ...; In the T-t+1 time step, the multi-view generation module uses N first images as conditional information, performs viewpoint transformation using the MVD model, and denoises M noisy images (t) to obtain M noisy images (t-1) corresponding to the second viewpoint; ...; In the T time step, the multi-view generation module uses N first images as conditional information, performs viewpoint transformation using the MVD model, and denoises M noisy images (2) to obtain the final M second images (1). Wherein, t is a positive integer less than or equal to T.

[0104] Figure 3c This is a schematic diagram of the denoising process at a time step of a second perspective of the multi-view generation module provided in this application embodiment. Figure 3c As shown, taking the m-th second perspective (i.e., second perspective m) out of M second perspectives as an example, the denoising process of the second perspective m at the T-t+1 time step is illustrated in an exemplary manner.

[0105] In this example, the denoising object at the (T-t+1)th time step of the second viewpoint m is the noisy image (t). The conditional information of the MVD model consists of N first images (e.g., N1 interpolated images), generating the noisy image (t-1) at the Ttth time step as the noisy image of the second viewpoint m. Specifically, the noise prediction module of the MVD model uses the N first images as conditional information to perform an inverse denoising process, predicting the noise in the noisy image (t). The noise processing module in this embodiment can handle one or more noises belonging to the same second viewpoint. Smoothing is performed to obtain the final noise to be removed. Smoothing is also known as averaging.

[0106] Figure 3d This is a schematic diagram illustrating the noise removal process provided in an embodiment of this application. Figure 3dAs shown, the noise removal process includes: using the first image as conditional information, the noise prediction module predicts the noise in M1 noisy images (t) from the second viewpoints, thus obtaining M1 noise values ​​corresponding to the M1 second viewpoints. M1 noise For example, noise (1-1) Noise (1-2)..., noise (1-M1) M1 noises; using the second first image as conditional information, the noise prediction module predicts the noise in the M2 noisy images (t) from the second viewpoints, thus obtaining the M2 noises corresponding to the M2 second viewpoints. M2 noises For example, noise (2-1) Noise (2-2)..., noise (2-M1) and M2 noises; ...; using the Nth first image as conditional information, predict M N The noise in a noisy image (t) from a second-view perspective is used to obtain M. N M corresponding to each second perspective N noise M N noise For example, noise (N-1), noise (N-2)..., noise (N-M1) etc. M N Noise; where M1, M2, ..., M N The integers are positive integers; M1, M2, ..., M N Less than or equal to M. Among them, for noise... (ab) provides further explanation: 'a' represents the viewpoint transformation based on the 'a'th first image, and 'b' represents the noise in the 'b'th second viewpoint among the predicted multiple second viewpoints.

[0107] Each interpolated image is used to generate one or more second images, which means that the one or more second perspectives generated from different interpolated images may have overlapping second perspectives. For example, a second perspective m may be one of M1 second perspectives and also one of M2 second perspectives. Thus, by analyzing the noise in the noisy image (t) of the second perspective m... It may be a prediction result from one or more interpolated images; for example, using the first first image as conditional information, M1 second images with second perspectives are generated; the second perspective among the M1 second perspectives is second perspective m; using the second first image as conditional information, M2 second images with second perspectives are generated; the fifth perspective among the M2 second perspectives is second perspective m; the multiple second images generated from the remaining first images do not include second perspective m; thus, the multiple noises corresponding to second perspective m Including noise (1-2) Noise (2-5).

[0108] Optionally, the noise processing module is used to process one or more noises corresponding to the prediction results from one or more interpolated images for the second viewpoint m. (Also known as the first noise to be removed) is processed to obtain the noise in the final noisy image (t) from the second viewpoint m. Final noise It is also known as the second noise to be removed.

[0109] For example, the noise processing module processes one or more noises. Smoothing methods include, for example: pixel-wise averaging, which calculates the average pixel value at each pixel location in the registered image; median filtering averaging, which takes the median of pixel values ​​instead of the mean; and weighted averaging, which calculates the average value based on the median of one or more noise values. Conduct an evaluation; assign weights based on the evaluation results, such as proximity to one or more noise levels. noise of the average value Greater weights; achieving the processing of one or more noises through a neural network model. The processing, etc.

[0110] Optionally, the multi-view generation module includes an IR model. The following describes... Figure 3c The working process of the IR model is illustrated by example.

[0111] In this embodiment, the IR model uses the extrapolated image of the second viewpoint m among N2 extrapolated images as conditional information, and the noisy image (t) of the second viewpoint m as the denoising object to predict the noise in the noisy image (t). noise This characterizes the noise during image inpainting of an extrapolated image from a second-viewpoint m. (Noise) It is also known as the first reference noise.

[0112] In this embodiment, both the interpolated and extrapolated images are obtained from the user's input image, which ensures the three-dimensional consistency of the interpolated and extrapolated images, and thus ensures the three-dimensional consistency of the second image. However, since the interpolated and extrapolated images are generated based on the user's input image, their quality is relatively low, which will result in a lower quality second image.

[0113] Therefore, for each second viewpoint of the second image, one or more extrapolated images corresponding to each second viewpoint are obtained based on the input image. During the viewpoint transformation process in the MVD model, the noise prediction module obtains the first reference noise of the extrapolated image of the second viewpoint m during the image inpainting process. Then generate the IR model Noise to be removed generated by MVD model The noise is then fused (e.g., weighted processing) to obtain fused noise. The noise removal module of the MVD model then removes noise based on this fused noise. This ensures both the 3D consistency of the second image and improves its quality. Consequently, the second image exhibits higher 3D consistency, accuracy, and overall image quality. This improves the quality of the generated 3D model of the target object and reduces the likelihood of issues such as reconstruction holes and rendering artifacts. The input image represents a known image of the target object. Depending on the usage scenario, the input image may have different sources, such as user-inputted images or images transmitted from electronic devices. This application does not limit the source of the input image in its embodiments.

[0114] It should be noted that the IR diffusion model can be understood as a conventional diffusion model used for image restoration (IR) tasks. The IR diffusion model recovers a high-quality original image from a noisy image by simulating the gradual addition and removal of noise. The working principle of the IR diffusion model will not be elaborated here. Optionally, the multi-view generation module includes a noise weighting module for weighting the noise in the noisy image (t) from the same second viewpoint m. and noise After weighted processing, the predicted noise values ​​∈ in the final noisy image (t) are obtained. t The weighting formula is given in formula (1). Where w MVD (t) characterizes noise The weight, w IR (t) characterizes noise The weights. For example, w MVD (t) = 1, noise to be removed The weight is 1, indicating that it is fully adopted.

[0115]

[0116] Due to errors in the image interpolation process, the quality of the N1 interpolated images and N2 extrapolated images is lower than that of the N input images provided by the user. Therefore, for the same second viewpoint m, fused noise is used. and noise noise ∈ t Denoising the noisy image (t) can both ensure the 3D consistency of the second image from the second viewpoint m and improve its quality. Optionally, the multi-view generation module includes a noise-adding module, which adds noise to the N2 interpolated images, such as by adding Gaussian noise, to obtain one or more noisy images (render). The noisy image (render) corresponding to the second viewpoint m is used as noise. This serves as input to the noise-weighting module. The noisy image (render) contains geometric information of the target object from the extrapolated image of the same second-viewpoint m. This allows the noise ∈ t The denoising process considers the geometric information of the target object, improving the accuracy and 3D consistency of the generated M second images. Noise addition is an optional step; adding noise to the interpolated image can prevent overfitting of the geometric structure based on the interpolated image. Alternatively, the interpolated image can be input into a neural network model such as a diffusion model, outputting noise.

[0117] For example, the noise weighting module is also used to weight the noise in the noisy image (t) of the second viewpoint m. noise and noise After weighted processing, the predicted noise values ​​∈ in the final noisy image (t) are obtained. t The weighted formula is given in formula (2). Where w R (t) characterizes noise The weight.

[0118]

[0119] Among them, noise weight w R One implementation of (t) is shown in Equation (3). In Equation (3), τ represents the preset noise. The offset value, τ is the translation parameter, which controls the position of the function on the time axis; σ represents the preset noise. The standard deviation, σ, is the scale parameter that controls the steepness of the function. tanh characterizes the hyperbolic tangent function. tanh is the hyperbolic tangent function; formula (3) is a variant of tanh, thus, w R The range of w is (0,1). Where, the smaller t is, the better w becomes. RThe smaller (t) is; when t=1, w R (t)≈0. Thus, in the early stages of the denoising process, the noise... w R When (t) is large, the geometric information of the target object provides stronger geometric guidance for the denoising process, which can accelerate the denoising process; in the later stages of the denoising process, the noise... w R (t) is smaller, the geometric information of the target object has weaker geometric guidance for the denoising process, and the geometric information of the generated noisy second-view image is already relatively complete and clear, with less noise. The geometric guidance is less stringent, which better suits the characteristics of the image denoising process and prevents noise from being properly addressed. The geometric information of the second image is interfered with and adversely affected by the geometric information of the second image.

[0120]

[0121] Among them, noise weight w IR One implementation of (t) is shown in Equation (4). In Equation (4), η represents noise. The scaling factor is used to control the weight w. IR The magnitude of (t). Weight w IR (t) and weight w R The trend of (t) is opposite. Thus, in the later stages of the denoising process, the noise... w IR A larger (t) value provides greater guidance for image restoration of the second image, resulting in a higher quality second image.

[0122]

[0123] It is understandable that the generation of multiple noises can occur simultaneously. For example, a second reference noise. The noise prediction module in the MVD model predicts the noise in the noisy image (t). Optionally, this embodiment of the application also uses an IR model to predict noise based on the interpolated image. To improve the image quality of the second image; and / or, simultaneously, to add noise to the interpolated image or to generate noise based on the interpolated image itself. Furthermore, based on noise and noise At least one of them, and noise The fusion is performed to obtain the fused noise ∈ t According to the fusion noise ∈ tDenoising the noisy image (t) to obtain the noisy image (t-1) can make the final second image have the characteristics of three-dimensional consistency, high quality and high three-dimensional structural accuracy.

[0124] Optionally, the noise removal module of the MVD model is further explained. The noise removal module of the MVD model is used to remove noise ∈ […] from the noisy image (t) at the (T-t+1)th time step of the second viewpoint m. t This yields the noisy image (t-1) at the Tt-th time step of the second viewpoint m. The noise removal module can be implemented using, but is not limited to, neural networks based on encoders and decoders; neural networks based on self-attention mechanisms; and deep neural networks based on residual connections.

[0125] It should be noted that, alternatively, multiple noises can be input into the noise removal module of the MVD model for processing without going through the noise weighting module, in order to generate a second image.

[0126] For example, the input conditional information of the multi-view diffusion model also includes camera parameter conditions, such as rotation parameters and deformation parameters, to control the viewpoint of the generated image; multi-view consistency conditions, in order to ensure the semantic and geometric consistency of the generated multi-view images, such as by using an epipolar segment attention mechanism to conditionally apply the denoising step of one view to its neighboring views, thereby enhancing 3D consistency, etc.

[0127] Figure 3e This is a schematic diagram illustrating image inpainting of an interpolated image using the IR model provided in this embodiment. Figure 3e As shown, on the one hand, during the noise removal process of the MVD model, the IR model is used to predict the noise of image inpainting in the second image. On the other hand, the IR model is also used to predict the noise of image inpainting in the first image, thereby improving the quality of the first image. For example, the first image can be N1 interpolated images; the N1 interpolated images are input into the IR model, and the IR model outputs N1 interpolated images after image inpainting.

[0128] Figure 4a This is a schematic diagram illustrating an example of applying a multi-view generation module during the 3D model generation process, as provided in an embodiment of this application. Figure 4a As shown, in one example, the 3D model is obtained through K iterations of the 3D model reconstruction pipeline, resulting in a higher quality 3D model. The 3D model reconstruction pipeline represents the method of obtaining a 3D model from images from one or more viewpoints using a 3D reconstruction method. For example, the 3D reconstruction method could be 3D Gaussian Splatting (3DGS), and the 3D model reconstruction pipeline is also called the 3DGS reconstruction pipeline.

[0129] In this example, the reconstruction pipeline for a single iteration of the 3D model mainly includes: acquiring the user's input image; obtaining an initial 3D model of the target object based on the input image using a 3D reconstruction method; determining one or more interpolated and extrapolated images based on the initial 3D model; wherein, the one or more interpolated images serve as first images, used as input to the MVD model to generate one or more second images; the one or more extrapolated images, relative to the low-quality second images, can serve as references for the one or more second images, used as input to the IR model to predict one or more first reference noises, providing noise for image inpainting of second images from the same second perspective, thereby improving the quality of the second images; optionally, the one or more extrapolated images on the input side are also used to obtain second reference noise after noise addition, prompting geometric information of the second images from the same second perspective, thereby improving the geometric structure information of the second images, and since different extrapolated images come from the same user input image, this ensures the 3D consistency of second images from different second perspectives. Optionally, the one or more interpolated images, as first images, are also used as input to the IR model to improve the quality of the first image.

[0130] Then, based on the interpolated and extrapolated images generated in the previous iteration step and the input image, the initial 3D model of the target object is obtained through a 3D reconstruction method. The 3D model reconstruction pipeline is then re-entered, and after K iterations, the final target 3D model is obtained. The 3D model reconstruction pipeline is executed for K iterations. That is, based on the generated second image (high-quality extrapolated image), first image (high-quality interpolated image), and input image, a 3D model of the target object is generated using a 3D reconstruction method. This 3D model is used as the initial 3D model. Based on the initial 3D model, images from the interpolated and extrapolated perspectives are obtained again; the multi-view generation module is called again to generate the second image (high-quality extrapolated image) and first image (high-quality interpolated image); based on the generated second image (high-quality extrapolated image), first image (high-quality interpolated image), and input image, a 3D model of the target object is generated again using a 3D reconstruction method. This continues until the iteration stopping condition is met, and the finally generated 3D model of the target object is used as the target 3D model. The iteration stopping conditions include reaching the set number of iterations (K iterations); evaluating the quality of the generated 3D model of the target object, and stopping the iteration of generating the 3D model of the target object when it meets the quality requirements.

[0131] Figure 4b These are schematic diagrams illustrating examples of various perspectives provided in the embodiments of this application. For example... Figure 4bAs shown, in this example, the target object is a "laptop". The image input by the user is an image of the target object from at least one perspective, such as an image taken by the user from the front (denoted as 0°), the left (denoted as 90°), or the right front (denoted as -45°).

[0132] Images (views). Here, the frontal view is denoted as 0°, representing the camera's initial position; the left-side view is denoted as 90°, representing the shooting angle from a top-down position, with the camera rotating 90° clockwise from its initial position; and the right-front view is denoted as -45°, representing the shooting angle from a top-down position, with the camera rotating 45° counter-clockwise from its initial position.

[0133] Furthermore, based on the user's input image, the observable viewing angle range is determined to be -45° to 90°. One or more viewing angles within this range are also called interpolated viewing angles; when the interpolated image is used as a known image viewing angle, the interpolated viewing angle is the first viewing angle. One or more viewing angles outside this range are also called extrapolated viewing angles; when the extrapolated image is used as an unknown image viewing angle, the extrapolated viewing angle is the second viewing angle. "Outside the viewing angle range" refers to the unobservable viewing angle range of 90° to -45° in the clockwise rotation direction.

[0134] Furthermore, the interpolated image represents the image obtained from at least one viewpoint within the known viewpoint range. The extrapolated viewpoint image represents the image from at least one viewpoint outside the known viewpoint range.

[0135] In this example, consider images taken by a user of a target object, a "laptop," from a frontal view (denoted as 0°), a left-side view (denoted as 90°), and a right-front view (denoted as -45°). The interpolated perspective represents a viewpoint ranging from -45° to 90° clockwise. Specifically, it represents at least one perspective taken during a top-down view as the camera rotates clockwise from a -45° right-front view to a 90° left-side view. For example, interpolated perspective images might include images with a 3° perspective, images with an 80° perspective, etc.

[0136] In this example, the extrapolated viewpoint represents a viewpoint ranging from 90° to -45° clockwise. That is, in the top-down direction, it is at least one viewpoint during the process of the camera rotating clockwise from a 90° left-side viewpoint to a -45° right-front viewpoint. For example, images with extrapolated viewpoints include images with a 110° viewpoint, images with a 180° viewpoint (the rear viewpoint of the target object), etc.

[0137] Figure 4cThis is a schematic diagram illustrating an example of the operation of the multi-view generation module provided in this application embodiment. The operation of this example multi-view generation module will be further explained. Figure 4c As shown, the denoising process of the multi-view generation module is divided into T time steps, where T is a positive integer. The noise images (T) of the M second-view perspectives are denoised multiple times to obtain the final M second-view images (1). The denoising process of the second-view m at the T-t+1th time step is used as an example for illustration.

[0138] In this example, the multi-view generation module includes an MVD model, an IR model, a noise addition module, a noise weighting module, and a noise removal module.

[0139] Furthermore, the input to the MVD model includes: a noisy image (t) from the second viewpoint m and an image from a known interpolated viewpoint. The noisy image (t) serves as the object to be denoised, and the image from the known interpolated viewpoint serves as conditional information. The interpolated viewpoint refers to the viewpoint within the user-input image's viewpoint range; the extrapolated viewpoint refers to the viewpoint outside the user-input image's viewpoint range. Therefore, the image from the interpolated viewpoint is of higher quality. Thus, when the conditional information of the MVD model's input consists of the image from the known interpolated viewpoint among N1 interpolated images and N2 extrapolated images, the conditional information is more accurate, and the generated M second images are of higher quality.

[0140] The MVD model outputs one or more second images corresponding to each interpolated viewpoint when the image is conditional information. Since there may be overlap between the one or more second images corresponding to different interpolated viewpoints, the noise in the noisy image (t) of the second viewpoint m predicted by the MVD model is significant. There may be one or more, requiring the handling of one or more noises. Summation yields Then take the average value; this yields the final noise level. The predicted value.

[0141] Furthermore, the input to the IR model includes: a noisy image (t) from the second viewpoint m, and images from known interpolated and extrapolated viewpoints. The noisy image (t) serves as the denoising target, while the images from known interpolated and extrapolated viewpoints serve as conditional information. This information is then used to inpaint the interpolated and extrapolated images, resulting in high-quality interpolated and extrapolated images. Specifically, for the extrapolated image, the IR model predicts the noise in the noisy image (t), obtaining the noise level. For the same second image, add noise and noise The fusion process involves denoising based on the fusion noise, thereby achieving the fusion of the second image generation process and the image restoration process.

[0142] Furthermore, the input to the noise-adding module includes: an image with a known interpolated viewpoint and an image with a known extrapolated viewpoint; the output includes a noisy image (render). The noisy image (render) is used as noise. This serves as geometric information guidance for the denoising process of the multi-view module. Furthermore, it provides geometric guidance for the generation of the second image, the image inpainting process, and the image inpainting process of the first image.

[0143] Furthermore, the noise weighting module is used to weight the noise in the noisy image (t) from the second viewpoint m. noise and noise After weighted processing, the predicted noise values ​​∈ in the final noisy image (t) are obtained. t .

[0144] Furthermore, the noise removal module is used to remove noise ∈ ∈ in the noisy image (t) at the (T-t+1)th time step of the second viewpoint m. t We obtain the noisy image (t-1) at the Tt-th time step of the second perspective m.

[0145] In this embodiment, the problem of 3D inconsistency when the MVD model generates the second image is solved. The mode of "single image input, multiple images output" is extended to "multiple images input, multiple images output". At the same time, the artifacts and 3D inconsistencies generated when optimizing 3DGS are directly called in the 3DGS reconstruction pipeline are solved. The ultimate goal is to reconstruct a complete high-quality 3DGS from the local observation image input.

[0146] This application proposes a method for 3D object reconstruction based on a non-global perspective. On one hand, this application provides a multi-view generation module with multi-view fusion and denoising. This module includes an MVD model and an IR model, and proposes a denoising strategy for fusing and balancing the noise predicted by the two diffusion models. This extends MVD from "single image input, multiple image output" to "multiple image input, multiple image output," thereby generating high-quality and 3D consistent unknown multi-view views.

[0147] In this embodiment of the application, taking 3DGS as an example, a method for generating 3D models using an iterative generative 3DGS optimization strategy is proposed. Through K repeated "rendering-generation-optimization" processes, newly generated high-quality interpolated and extrapolated images are introduced into the 3DGS optimization process, gradually iteratively optimizing low-quality 3DGS into high-quality 3DGS.

[0148] In this embodiment, a multi-view fusion and denoising multi-view generation module with "multiple images in, multiple images out" is first invented. Its core is the inverse process of the diffusion model, that is, gradually removing noise from Gaussian noise to finally obtain a clear image. The specific denoising process is as follows: A common MVD model receives a single image (a single first image) as input and generates noise predictions for new viewpoints (one or more second images). When there are one or more input first images, the predicted new viewpoints will overlap. In this embodiment, the overlapping new viewpoint noise is directly averaged to finally obtain the smoothed new viewpoint noise prediction at time t. Since the input images are derived from low-quality 3DGS renderings, the image quality is generally poor. Therefore, this embodiment also introduces an IR model in the multi-view generation module to predict the amount of noise in the restored image. At the same time, it adds noise to the interpolated perspective of the 3DGS rendering. Used as geometric guidance. The fusion of the above three types of noise yields the noise predicted by the "multi-view fusion noise removal module" (multi-view input, multi-view output). Since the three types of noise have different magnitudes, this application embodiment uses different weighting functions for each. and Used to balance the three different types of noise, the final noise prediction at time t after fusion is: By iterating through the above denoising process, a clear new perspective image can be obtained.

[0149] In this embodiment, after obtaining a clear new perspective image, taking 3DGS as an example, this embodiment proposes an iterative generative 3DGS strategy. Unlike previous methods that first generate an image and then call the 3DGS reconstruction pipeline again, this embodiment uses an iterative optimization approach for the original low-quality 3DGS (example of the initial 3D model); repeatedly performing the "3DGS rendering-generation-optimization 3DGS" process to finally obtain a high-quality 3DGS (example of the target 3D model). Optimizing the 3DGS refers to obtaining low-quality interpolated and low-quality extrapolated images based on the low-quality 3DGS, calling the multi-view generation module of this embodiment to obtain high-quality interpolated and low-quality extrapolated images; and then performing 3DGS generation and rendering based on the low-quality interpolated and low-quality extrapolated images to obtain a high-quality 3DGS.

[0150] In this embodiment, a low-quality 3DGS is first reconstructed using a non-global input image. Then, interpolated and extrapolated images are rendered. Using the "multi-image input, multi-image output" multi-view generation module of this embodiment, high-quality interpolated / extrapolated images are generated from the low-quality images. The 3DGS reconstruction pipeline is then invoked to update and optimize the original low-quality 3DGS, resulting in a high-quality 3DGS. K iterations of the above "rendering-generation-optimization" process yield the final high-quality 3DGS.

[0151] In this application embodiment, a multi-view generation module for multi-view fusion denoising with "multiple images in, multiple images out" is proposed. It dynamically fuses the noise to be removed from the perspective transformation of the MVD model and the noise corresponding to the prior image restoration process generated by the IR model to generate one or more second images with three-dimensional consistency.

[0152] This application provides an iterative generative method for generating 3D models. Taking 3DGS as an example, unlike previous methods that only call the 3DGS reconstruction pipeline once, this application repeatedly performs the "rendering-generation-optimization" process, gradually optimizing low-quality 3DGS to high-quality 3DGS; thus, the unknown 3D reconstruction quality of unobservable targets from a second-view perspective is superior to the 3DGS reconstruction methods of related technologies.

[0153] Figure 5 This is a flowchart illustrating a three-dimensional model generation method provided in an embodiment of this application. Figure 5 As shown in the figure, this application provides a method for generating a three-dimensional model, which mainly includes the following steps:

[0154] Step S510: Acquire a first image. The first image represents an image of the target object from a first viewpoint. For example, one or more first images represent known images of the target object observed from one or more first viewpoints; one or more first images represent images of the target object from one or more first viewpoints; and one or more first viewpoints represent one or more viewpoints within the observable viewpoint range.

[0155] Step S520: Input the first image into the first diffusion model to obtain the second image. The first diffusion model includes a noise prediction module and a noise removal module. The noise prediction module uses the first image as a condition to predict a first noise to be removed, representing the second viewpoint. The noise removal module performs denoising based on the second noise to be removed to obtain the second image. The second noise to be removed is obtained by smoothing multiple first noises belonging to a single second viewpoint.

[0156] For example, using a first diffusion model, a viewpoint transformation is performed on each first image to predict the noise to be removed from multiple sets of noisy images. Each set of noise to be removed corresponds to a first image, and multiple sets of noisy images are used to generate one or more second images. When one or more noises to be removed belong to the same second viewpoint, the one or more noises to be removed are processed. The second viewpoint represents a viewpoint outside the observable viewpoint range. The one or more second images represent images of the target object in one or more second viewpoints.

[0157] Step S530: Generate a target 3D model of the target object based on the first image and the second image.

[0158] In this embodiment, a second image is generated based on a first image using a first diffusion model. Then, a target 3D model of the target object is generated based on one or more first images and one or more second images. Because one or more noise samples from the same second viewpoint are processed, images generated from different first images at the same viewpoint maintain consistency in 3D structure, improving the generation quality of the 3D model of the target object and reducing the possibility of problems such as reconstruction holes and rendering artifacts.

[0159] In one possible implementation, the method further includes: processing a third image to obtain reference noise; the third image represents an initial image of the target object in a second viewpoint; the reference noise includes at least one of noise used for image inpainting in the second viewpoint and geometric noise of the target object in the second viewpoint. A noise removal module is specifically configured to: denoise based on the reference noise and the second noise to be removed to obtain a second image.

[0160] The noise used for image inpainting from a second-view perspective is also referred to as the first reference noise, such as the noise mentioned above. The first reference noise is used to characterize the image quality of the target object from the second viewpoint. The geometric noise of the target object in the second viewpoint is also called the second reference noise, such as the noise mentioned above. The second reference noise is used to characterize the geometric information of the target object from the second perspective.

[0161] In this implementation, at least one of a first reference noise and a second reference noise is obtained based on the third image; the first reference noise is used to characterize the image quality of the target object from the second perspective, and the second reference noise is used to provide geometric information of the second image to improve the geometric structure information of the second image and further ensure the three-dimensional consistency of different second images.

[0162] It should be noted that the first and second reference noises can be obtained by any method, such as through a neural network model or through image processing methods. For example, one or more third images are input into an IR model to predict one or more first reference noises, providing noise for image inpainting of second images from the same second viewpoint, thereby improving the quality of the second images. Noise is added to one or more third images to obtain corresponding one or more second reference noises; these second reference noises are used to indicate the geometric information of the second images from the same second viewpoint, thereby improving the geometric structure information of the second images. Furthermore, since different third images come from the same user input image, this ensures the three-dimensional consistency of the second images from different second viewpoints.

[0163] In one possible implementation, an input image is acquired; an initial 3D model is generated based on the input image; and at least one of a third image and a first image is obtained based on the initial 3D model.

[0164] In this implementation, different third images come from the same initial 3D model, ensuring the 3D consistency of different third images, and further ensuring the 3D consistency of second images from different second perspectives.

[0165] In this implementation, both the third image and the first image are obtained from the user's input image, which ensures the consistency of the three-dimensional structure of the third image and the second image. Thus, the third image can serve as a reference for the second image, thereby providing the second image with first reference noise to improve the image quality of the second image; and providing the second reference noise to prompt and guide the geometric structure of the second image, further ensuring the three-dimensional consistency of the second image from different second perspectives.

[0166] In one possible implementation, the above-mentioned denoising based on the first reference noise and / or the second reference noise, and the second noise to be removed to obtain the second image includes: weighting the second noise to be removed and at least one of the first reference noise and the second reference noise to obtain the target noise; and denoising based on the target noise to obtain the second image.

[0167] In this implementation, the influence of the first reference noise and the second reference noise on the fusion noise can be determined by different weights, thereby enabling the second image to have different image quality and geometric information, and improving the applicability and generalization of the second image to different application scenarios.

[0168] In one possible implementation, during the inverse denoising iteration of generating the first image, the weight of the first reference noise from the same second viewpoint is gradually increased; and / or, the weight of the second reference noise from the same second viewpoint is gradually decreased.

[0169] In this implementation, the weight of the first reference noise from the same second viewpoint gradually increases. Thus, in the later stages of the denoising process, the weight of the first reference noise is smaller, the geometric information of the target object has less geometric guidance for the denoising process, and the geometric information of the generated noisy image from the second viewpoint is already relatively complete and clear. The geometric guidance of the first reference noise is even less, which better aligns with the characteristics of the image denoising process and prevents the geometric information of the first reference noise from interfering with or adversely affecting the geometric information of the generated second image. Conversely, the weight of the second reference noise from the same second viewpoint gradually decreases. Thus, in the later stages of the denoising process, the weight of the second reference noise is larger, providing greater guidance for image restoration of the second image, resulting in a higher quality generated second image.

[0170] In one possible implementation, the above-mentioned acquisition of one or more first images of the target object includes: acquiring an input image of the target object; the input image representing an image of the target object within an observable field of view; obtaining one or more interpolated images based on the input image, and using the one or more interpolated images as one or more first images; the interpolated images representing images within the field of view.

[0171] In this implementation, different interpolated images are derived from the same user input image, and the interpolated images are compared to the low-quality first image. This solves the problem that the number of user input images is small, resulting in poor performance when directly generating multi-view images from the user input images. Furthermore, compared to the third image, the interpolated image represents the image within the same viewpoint range as the user input image, which results in higher quality interpolated images, ensuring the accuracy of noise removal and thus guaranteeing the accuracy of the second image.

[0172] In one possible implementation, the above-mentioned viewpoint transformation of the first image to obtain the third image includes: acquiring the user's input image; generating an initial three-dimensional model of the target object based on the input image; obtaining the third image based on the initial three-dimensional model; and obtaining the first image based on the initial three-dimensional model.

[0173] In this implementation, different interpolated images and third images come from the same initial 3D model, further ensuring the 3D consistency of second images from different second perspectives.

[0174] In one possible implementation, the above method further includes: generating a target 3D model based on a first image and a second image, comprising: inputting the first image into a second diffusion model to obtain a fourth image, the fourth image representing a repaired image of the target object in a first viewpoint; and generating a target 3D model of the target object based on the fourth image and the second image.

[0175] In this implementation, the first image may have low quality issues. One or more first images are input into a second diffusion model to generate a fourth image after image restoration. Based on the fourth image and the second image, a target 3D model of the target object is generated, ensuring the quality of the generated target 3D model.

[0176] In one possible implementation, generating the target 3D model based on the first image and the second image includes: generating the 3D model through K iterations based on the first image and the second image; where K is a positive integer; and the k-th iteration process, where k is a positive integer less than or equal to K, includes: obtaining the first image of the current iteration step based on the 3D model of the previous iteration step; inputting the first image of the current iteration step into a first diffusion model to obtain the second image of the current iteration step; and generating the 3D model of the current iteration step based on the first image and the second image of the current iteration step.

[0177] In this implementation, the target 3D model of the target object is generated through K iterations. Thus, the image quality of the first and second images continuously improves during the K iterations, thereby ensuring the quality of the generated target 3D model of the target object.

[0178] In one possible implementation, the above-mentioned generation of a target 3D model of a target object based on one or more first images and one or more second images includes: acquiring an input image of the target object; the input image representing an image of the target object within an observable field of view; obtaining one or more interpolated images and one or more third images based on the input image, using the one or more interpolated images as one or more first images; the interpolated images representing images within the field of view; the third images representing images outside the field of view, with one or more third images corresponding one-to-one with one or more second images based on a second field of view; and generating a target 3D model of the target object through K iterations based on the input image, one or more interpolated images, and one or more third images; where K is a positive integer. The k-th iteration process, where k is a positive integer less than or equal to K, includes: generating one or more interpolated images and one or more third images for the current iteration step based on the 3D model of the previous iteration step; wherein the one or more interpolated images of the current iteration step are used as one or more first images of the current iteration step, and the one or more third images correspond one-to-one with one or more second images based on a second perspective; inputting the one or more interpolated images of the current iteration step into a first diffusion model, and inputting the one or more third images of the current iteration step into a second diffusion model to obtain one or more second images; inputting the one or more interpolated images of the current iteration step into the second diffusion model to obtain another one or more interpolated images of the current iteration step; generating the 3D model of the current iteration step based on the input images, the other one or more interpolated images of the current iteration step, and the one or more third images of the current iteration step.

[0179] In this implementation, the target 3D model of the target object is generated through K iterations. In this way, the image quality of the generated interpolated and third images continuously improves during the K iterations. Then, based on the generated high-quality interpolated and third images and the input image, a 3D reconstruction method is used to obtain the initial 3D model of the target object. This ensures the quality of the generated target 3D model of the target object.

[0180] In one possible implementation, the first perspective described above represents a perspective within the observable field of view, and the second perspective represents a perspective outside the observable field of view.

[0181] In this implementation, the first viewpoint is within the observable viewpoint range, ensuring that the first image contains partial 3D structural information of the target object, thereby generating a second image outside the observable viewpoint range. When generating a target 3D model based on the first and second images, the first image can provide 3D structural information of the target object within the observable viewpoint range, while the second image can provide 3D structural information of the target object outside the observable viewpoint range, thus ensuring the quality of the generated target 3D model. This makes the application more widely applicable, such as scenarios where the target object is only observable within a partial viewpoint range. For example, an image or video of the target object is captured within a local viewpoint range; the image or video of this local viewpoint (one or more viewpoints within the observable viewpoint range) is processed to obtain the first image.

[0182] In one possible implementation, the aforementioned one or more first images represent images that conform to a preset rule, and the preset rule represents one or more first viewpoints within an observable viewpoint range.

[0183] In this implementation, one or more first viewpoints are within the observable field of view, ensuring the structural integrity of the target object in one or more first images and guaranteeing the accuracy of the prediction results for the second image. This application's embodiments have a wider range of applicability, such as in scenarios where only a portion of the target object's field of view is observable.

[0184] In one possible implementation, the method further includes: acquiring an input video, which represents a video of the target object within an observable field of view; and processing the input video to obtain a first image. For example, acquiring an observation video input by a user, which represents a video of the observed target object within an observable field of view; and processing the observation video to obtain one or more first images conforming to a preset rule.

[0185] In this implementation, compared to methods that generate a 3D model based on a complete observation video of the target object, this embodiment can generate a 3D model of the target object based on an observation video within a certain observable field of view of the user. This broadens the applicability of this application, for example, in scenarios where only a portion of the target object is observable from a specific field of view. For instance, the observation video can be processed, such as through frame segmentation, with each first image representing a single frame in the observation video; or, keyframe images from the observation video can be used as the first images.

[0186] Based on the same concept as the foregoing embodiments, this application also provides a three-dimensional model generation apparatus.

[0187] Figure 6This is a schematic diagram illustrating the composition of a three-dimensional model generation device provided in an embodiment of this application. Figure 6 As shown in the figure, this application embodiment provides a three-dimensional model generation device 600, which mainly includes:

[0188] The acquisition module 610 is used to acquire an input image and generate an initial 3D model based on the input image; and obtain a first image and a third image based on the initial 3D model; the first image represents the initial image of the target object in a first viewpoint; and the third image represents the initial image of the target object in a second viewpoint.

[0189] The processing module 620 is used to input a first image into a first diffusion model to obtain a second image; the second image represents the target image of the target object in a second viewpoint; the first diffusion model includes a noise prediction submodule and a noise removal submodule, the noise prediction submodule is used to use the first image as a condition to predict a first noise to be removed for representing the second viewpoint; the noise removal submodule is used to denoise according to the second noise to be removed to obtain the second image; the second noise to be removed is obtained by smoothing multiple first noises belonging to a second viewpoint; and a target 3D model is generated based on the first image and the second image.

[0190] In one possible implementation, the processing module is further configured to: represent the third image as an initial image of the target object in a second viewpoint; process the third image to obtain reference noise; the reference noise includes at least one of noise used for image inpainting in the second viewpoint and geometric noise of the target object in the second viewpoint. The noise removal submodule is specifically configured to: denoise based on the reference noise and the second noise to be removed to obtain the second image.

[0191] In one possible implementation, when generating a target 3D model based on the first image and the second image, the processing module 620 is specifically used to: input the first image into the second diffusion model to obtain a fourth image, the third image representing the repaired image of the target object in the first viewpoint; and generate a target 3D model of the target object based on the fourth image and the second image.

[0192] In one possible implementation, when generating the target 3D model based on the first image and the second image, the processing module 620 is specifically used to: generate the target 3D model through K iterations based on the first image and the second image; where K is a positive integer; and the k-th iteration process, where k is a positive integer less than or equal to K, includes: obtaining the first image of the current iteration step based on the 3D model of the previous iteration step; inputting the first image of the current iteration step into the first diffusion model to obtain the second image of the current iteration step; and generating the 3D model of the current iteration step based on the first image and the second image of the current iteration step.

[0193] In one possible implementation, the first perspective represents the perspective within the observable field of view, and the second perspective represents the perspective outside the observable field of view.

[0194] The following is about... Figure 6 The software and hardware implementation of the 3D model generation device 600 shown (hereinafter referred to as: the device of this application embodiment) needs further explanation.

[0195] As an example of a software functional unit, a module can include code running on a computing instance. A computing instance can include at least one of a physical host (computing device), a virtual machine, or a container. Furthermore, the aforementioned computing instance can be one or more. For example, a module can include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code can be distributed within the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code can be distributed within the same availability zone (AZ) or in different AZs, each AZ comprising one or more geographically proximate data centers. Typically, a region can include multiple AZs.

[0196] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0197] As an example of a hardware functional unit, a module may include at least one computing device, such as a server. Alternatively, a module may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The aforementioned PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0198] The multiple computing devices included in the module can be distributed within the same region or in different regions. Similarly, the multiple computing devices included in the module can be distributed within the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in the module can be distributed within the same Virtual Private Cloud (VPC) or multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0199] It should be noted that, in other embodiments, the apparatus of this application embodiment additionally provides one or more modules for performing any of the steps included in the above implementation. The steps implemented by one or more modules in the apparatus of this application embodiment can be specified as needed, and more or fewer modules can be obtained than in the embodiments of this application to implement different steps in the above method, thereby realizing all the functions of the apparatus of this application embodiment.

[0200] This application also provides a computing device 700. For example... Figure 7 As shown, the computing device 700 includes a bus 702, a processor 704, a memory 706, and a communication interface 708. The processor 704, memory 706, and communication interface 708 communicate with each other via the bus 702. The computing device 700 can be a server, such as a central server, an edge server, or a local server in a local data center, or it can be an electronic device such as a desktop computer, laptop computer, or smartphone. It should be understood that this application does not limit the number of processors and memories in the computing device 700.

[0201] The 702 bus can be a Peripheral Component Interconnect (PCI) bus, a Peripheral Component Interconnect Express (PCIe) bus, an Extended Industry Standard Architecture (EISA) bus, a Unified Bus (Ubus or UB), a Compute Express Link (CXL), a Cache Coherent Interconnect for Accelerators (CCIX), etc. The Unified Bus is also known as the Lingqu Bus. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 7The bus 704 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 704 may include a path for transmitting information between various components of the computing device 700 (e.g., memory 706, processor 704, communication interface 708).

[0202] The processor 704 may include any one or more computing devices such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP) or a digital signal processor (DSP), an ASIC, an FPGA, a CPLD, an NPU, a SoC, an offload card, or an accelerator card.

[0203] Memory 706 may include volatile memory, such as random access memory (RAM). Processor 704 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD). Furthermore, memory 706 may also be implemented using storage class memory (SCM), phase change memory (PCM), or other types of storage media.

[0204] It is worth noting that the same type of storage medium can be configured in the same computing device to realize the function of memory 706, or two or more types of storage media can be configured to realize the function of memory 706. This application does not limit this.

[0205] The memory 706 stores executable program code, and the processor 704 executes the executable program code to implement the functions of the apparatus in the embodiments of this application, thereby implementing the methods described in the above embodiments. That is, the memory 706 stores instructions for executing the methods described in the above embodiments.

[0206] Alternatively, the memory 706 may store executable code, which the processor 704 executes to implement the functions of the apparatus in the embodiments of this application, thereby implementing the methods described in the above embodiments. That is, the memory 706 stores instructions for executing the methods described in the above embodiments.

[0207] The communication interface 708 uses transceiver modules, such as, but not limited to, network interface cards and transceivers, to enable communication between the computing device 700 and other devices or communication networks.

[0208] As one possible implementation, the computing device 700 may also include a chip system, which includes a processor and a power supply circuit. The power supply circuit supplies power to the processor, and the processor executes the operation steps corresponding to the methods in the embodiments of this application. For simplicity, further details are omitted here. The processor can be implemented using a GPU, or it can be implemented using computing devices or AI chips such as a DPU, NPU, XPU, SoC, offload card, or accelerator card.

[0209] As one possible implementation, the computing device 700 may include various types of processors 704, meaning the computing device 700 is a heterogeneous device. For example, the computing device 700 may include a CPU and a GPU, and at least one of the processors 704 may execute the operation steps corresponding to the methods of the embodiments of this application. For the sake of brevity, further details are omitted here.

[0210] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be an electronic device such as a desktop computer, a laptop computer, or a smartphone.

[0211] like Figure 8 As shown, the computing device cluster includes at least one computing device 700. The memory 706 of one or more computing devices 700 in the computing device cluster may store the same instructions for performing the methods described in the above embodiments.

[0212] In some possible implementations, the memory 706 of one or more computing devices 700 in the computing device cluster may also store partial instructions for executing the methods described in the above embodiments. In other words, a combination of one or more computing devices 700 can jointly execute instructions for executing the methods described in the above embodiments.

[0213] It should be noted that the memories 706 in different computing devices 700 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the apparatus in this application embodiment. That is, the instructions stored in the memories 706 of different computing devices 700 can implement the functions of one or more modules of the apparatus in this application embodiment.

[0214] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 9 One possible implementation is shown. For example... Figure 9 As shown, two computing devices 700A and 700B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this possible implementation, the memory 706 in computing device 700A stores instructions for the functions of one or more modules of the apparatus of this application embodiment. Meanwhile, the memory 706 in computing device 700B stores instructions for the functions of another one or more modules of the apparatus of this application embodiment.

[0215] It should be understood that Figure 9 The functions of the computing device 700A shown can also be performed by multiple computing devices 700. Similarly, the functions of the computing device 700B can also be performed by multiple computing devices 700.

[0216] This application also provides another computing device cluster. The connection relationships between the computing devices in this computing device cluster can be similarly referred to... Figure 8 and Figure 9 The connection method of the computing device cluster is different in that the memory 706 of one or more computing devices 700 in the computing device cluster can store the same instructions for executing the methods in the above embodiments.

[0217] In some possible implementations, the memory 706 of one or more computing devices 700 in the computing device cluster may also store partial instructions for executing the methods of the embodiments of this application. In other words, a combination of one or more computing devices 700 can jointly execute the instructions for performing the aforementioned methods.

[0218] Based on the methods in the above embodiments, this application provides a computer-readable storage medium including computer program instructions. When executed by a cluster of computing devices including at least one computing device, the computer program instructions cause the cluster of computing devices to perform the methods in the above embodiments. Exemplarily, the computer-readable storage medium can be any available medium capable of being stored in the cluster of computing devices or a data storage device such as a data center containing one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives).

[0219] Based on the methods in the above embodiments, this application provides a computer program product containing instructions. The computer program product may be software or program products containing instructions, capable of running on a computing device or stored on any available medium. When the instructions are executed by a cluster of computing devices containing at least one computing device, at least one computing device in the cluster of computing devices performs the methods in the above embodiments.

[0220] It is understood that the processor in the embodiments of this application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor.

[0221] The method steps in the embodiments of this application can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an ASIC.

[0222] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0223] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application.

[0224] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.

Claims

1. A method for generating a three-dimensional model, characterized in that, include: The first image is input into the first diffusion model to obtain the second image; The first image represents the initial image of the target object from a first perspective; The second image represents the target image of the target object in a second viewpoint; The first diffusion model includes a noise prediction module and a noise removal module. The noise prediction module is used to predict a first noise to be removed, which characterizes the second viewpoint, using the first image as a condition. The noise removal module is used to remove noise based on a second noise to be removed in order to obtain the second image; the second noise to be removed is obtained by smoothing multiple first noises belonging to a second viewpoint. A target 3D model is generated based on the first image and the second image.

2. The method according to claim 1, characterized in that, Also includes: The third image is processed to obtain reference noise; The third image represents the initial image of the target object from the second perspective; The reference noise includes at least one of the noise used for image inpainting in the second viewpoint and the geometric noise of the target object in the second viewpoint; The noise removal module is specifically used for: The second image is obtained by denoising based on the reference noise and the second noise to be removed.

3. The method according to claim 2, characterized in that, Also includes: Obtain the input image; Based on the input image, an initial 3D model is generated; Based on the initial 3D model, the first image and the third image are obtained.

4. The method according to any one of claims 1-3, characterized in that, The step of generating a target 3D model based on the first image and the second image includes: The first image is input into the second diffusion model to obtain a fourth image, which represents the repaired image of the target object in the first viewpoint; Based on the fourth image and the second image, a target 3D model of the target object is generated.

5. The method according to any one of claims 1-4, characterized in that, The step of generating a target 3D model based on the first image and the second image includes: Based on the first image and the second image, the target 3D model is generated through K iterations; where K is a positive integer. In the k-th iteration, k is a positive integer less than or equal to K, including: Based on the three-dimensional model of the previous iteration step, the first image of the current iteration step is obtained; Input the first image of the current iteration step into the first diffusion model to obtain the second image of the current iteration step; Generate a 3D model for the current iteration step based on the first image and the second image of the current iteration step.

6. The method according to any one of claims 1-5, characterized in that, The first viewpoint represents the viewpoint within the observable viewpoint range, and the second viewpoint represents the viewpoint outside the observable viewpoint range.

7. A three-dimensional model generation device, characterized in that, include: The processing module is used to input the first image into the first diffusion model to obtain the second image; The first image represents the initial image of the target object from a first perspective; The second image represents the target image of the target object in a second viewpoint; The first diffusion model includes a noise prediction submodule and a noise removal submodule. The noise prediction submodule is used to predict a first noise to be removed, which is used to characterize the second viewpoint, by taking the first image as a condition. The noise removal submodule is used to denoise according to the second noise to be removed in order to obtain the second image; the second noise to be removed is obtained by smoothing multiple first noises belonging to a second viewpoint. A target 3D model is generated based on the first image and the second image.

8. The apparatus according to claim 7, characterized in that, The processing module is also used for: The third image is processed to obtain reference noise; the third image represents the initial image of the target object in the second viewpoint; the reference noise includes at least one of the noise used for image inpainting in the second viewpoint and the geometric noise of the target object in the second viewpoint. The noise removal submodule is specifically used for: The second image is obtained by denoising based on the reference noise and the second noise to be removed.

9. The apparatus according to claim 8, characterized in that, It also includes an acquisition module; the acquisition module is used for: Obtain the input image; Based on the input image, an initial 3D model is generated; Based on the initial 3D model, the first image and the third image are obtained.

10. The apparatus according to any one of claims 7-9, characterized in that, The step of generating a target 3D model based on the first image and the second image includes: The first image is input into the second diffusion model to obtain the fourth image, and the third image represents the repaired image of the target object in the first viewpoint; Based on the fourth image and the second image, a target 3D model of the target object is generated.

11. The apparatus according to any one of claims 7-10, characterized in that, The step of generating a target 3D model based on the first image and the second image includes: Based on the first image and the second image, the 3D model is generated through K iterations; K is a positive integer. In the k-th iteration, k is a positive integer less than or equal to K, including: Based on the three-dimensional model of the previous iteration step, the first image of the current iteration step is obtained; Input the first image of the current iteration step into the first diffusion model to obtain the second image of the current iteration step; Generate a 3D model for the current iteration step based on the first image and the second image of the current iteration step.

12. The apparatus according to any one of claims 7-11, characterized in that, The first viewpoint represents the viewpoint within the observable viewpoint range, and the second viewpoint represents the viewpoint outside the observable viewpoint range.

13. A chip system, characterized in that, The chip system includes a processor and a power supply circuit, the power supply circuit being used to supply power to the processor, the processor being used to execute the method as described in any one of claims 1-6.

14. A computing device, characterized in that, The computing device includes a processor and memory; The processor is configured to execute instructions stored in the memory to cause the computing device to perform the method as described in any one of claims 1-6.

15. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1-6.

16. A computer-readable storage medium, characterized in that, The method includes computer program instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method as described in any one of claims 1-6, wherein the cluster of computing devices includes at least one computing device.

17. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device cluster, the computing device cluster causes the computing device cluster to perform the method as described in any one of claims 1-6, wherein the computing device cluster includes at least one computing device.

Images