Three-dimensional reconstruction method and device based on structured three-dimensional Gaussian representation

Abstract

The embodiment of the invention discloses a three-dimensional reconstruction method based on structured three-dimensional Gaussian representation. The method comprises the following steps: acquiring a target image containing a target object at multiple visual angles; generating an unstructured Gaussian primitive representation corresponding to the target object according to the target image; establishing a transmission matrix describing the unstructured Gaussian primitive distribution and the structured Gaussian primitive distribution, solving the transmission matrix by using an optimal transmission algorithm, and generating a structured Gaussian primitive representation corresponding to the target object according to the optimal transmission matrix; obtaining a text description for the target object, and complementing the structured Gaussian primitives according to the text description by using a diffusion model obtained by pre-training; and performing three-dimensional reconstruction by using the complete structured Gaussian primitive. According to the method, the optimal transmission algorithm is used for carrying out structured representation on the three-dimensional Gaussian primitives, and text description is used for complementing the Gaussian primitives through the diffusion model, so that the quality of three-dimensional reconstruction is improved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a three-dimensional reconstruction method and device based on structured three-dimensional Gaussian representation. Background Art

[0002] With the development of computer vision technology, 3D reconstruction technology has been widely used in many fields. As an effective probability distribution representation, Gaussian primitives have been widely used in the description of 3D object surfaces and feature points, but the unstructured representation of traditional Gaussian primitives cannot meet the needs of fine 3D reconstruction in structured scenes. Summary of the invention

[0003] The present application provides a three-dimensional reconstruction method and device based on structured three-dimensional Gaussian representation, so as to improve the accuracy of the three-dimensional reconstruction model through structured Gaussian basis element representation.

[0004] This application provides the following solutions:

[0005] According to a first aspect, a three-dimensional reconstruction method based on a structured three-dimensional Gaussian representation is provided, the method comprising: obtaining a target image including multiple perspectives of a target object, wherein the target image of the multiple perspectives has a perspective missing relative to the target image of the full perspective; generating an unstructured Gaussian primitive representation corresponding to the target object according to the target image; establishing a transmission matrix describing the distribution of unstructured Gaussian primitives and structured Gaussian primitives, solving the transmission matrix using an optimal transmission algorithm to obtain an optimal transmission matrix, and generating a structured Gaussian primitive representation corresponding to the target object from the unstructured Gaussian primitives according to the optimal transmission matrix; obtaining a text description of the target object, completing the structured Gaussian primitives according to the text description using a pre-trained diffusion model, and obtaining a complete structured Gaussian primitive; performing three-dimensional reconstruction using the complete structured Gaussian primitives to obtain a three-dimensional reconstruction model corresponding to the target object.

[0006] According to an achievable method in an embodiment of the present application, generating an unstructured Gaussian primitive representation corresponding to the target object based on the target image includes: performing image segmentation on the target image to obtain a target object area; performing feature point extraction on the target object area to obtain a plurality of feature points; clustering the feature points using a clustering algorithm, and performing Gaussian distribution fitting using the clustering results to obtain an unstructured Gaussian primitive.

[0007] According to an achievable method in an embodiment of the present application, the method also includes: the structured Gaussian primitives are represented by a structured grid, and the resolution of the structured Gaussian primitives is adjusted according to a scaling factor, and the scaling factor includes the size of the structured grid; or, the structured Gaussian primitives are represented by a point cloud, and the resolution of the structured Gaussian primitives is adjusted according to a scaling factor, and the scaling factor includes the number of samplings or the sampling density.

[0008] According to an achievable method in an embodiment of the present application, the diffusion model obtained by pre-training is used to complete the structured Gaussian primitives according to the text description to obtain a complete structured Gaussian primitives, including: encoding the text description to obtain a high-dimensional vector representation corresponding to the text description; fusing the high-dimensional vector and the structured Gaussian primitives to obtain a fused vector; and generating the complete structured Gaussian primitives by the diffusion model according to the fused vector.

[0009] According to an implementable method in an embodiment of the present application, encoding the text description to obtain a high-dimensional vector representation corresponding to the text description includes: loading trained low-rank matrix parameters, combining the low-rank matrix parameters with the parameter matrix of the language model to obtain a combined matrix; and using the combined matrix to generate a high-dimensional vector representation corresponding to the text description.

[0010] According to an achievable manner in an embodiment of the present application, generating the complete structured Gaussian primitives from the diffusion model according to the fusion vector includes: acquiring control information, and generating guidance information according to the control information using a control network;

[0011] The complete structured Gaussian primitives are generated by the diffusion model according to the guidance information and the fusion vectors.

[0012] According to an achievable method in an embodiment of the present application, the use of the complete structured Gaussian primitives to perform three-dimensional reconstruction to obtain a three-dimensional reconstruction model corresponding to the target object includes: using the complete structured Gaussian primitives to perform three-dimensional reconstruction to obtain a pre-reconstructed model of the target object; rasterizing the pre-reconstructed model to obtain two-dimensional images at multiple angles, the two-dimensional images including feature information of their corresponding viewing angles; determining the correspondence between different viewing angles based on the feature information; and generating the three-dimensional reconstruction model corresponding to the target object based on the correspondence.

[0013] According to a second aspect, a three-dimensional reconstruction device based on a structured three-dimensional Gaussian representation is provided, the device comprising: a target image acquisition unit, configured to acquire a target image of multiple perspectives including a target object, wherein the target image of the multiple perspectives has a perspective missing relative to the target image of the full perspective; an unstructured Gaussian primitive generation unit, configured to generate an unstructured Gaussian primitive representation corresponding to the target object according to the target image; a structured Gaussian primitive generation unit, configured to establish a transmission matrix describing the distribution of unstructured Gaussian primitives and the distribution of structured Gaussian primitives, solve the transmission matrix using an optimal transmission algorithm to obtain an optimal transmission matrix, and generate a structured Gaussian primitive representation corresponding to the target object from the unstructured Gaussian primitives according to the optimal transmission matrix; a Gaussian primitive completion unit, configured to acquire a text description of the target object, complete the structured Gaussian primitives according to the text description using a pre-trained diffusion model, and obtain a complete structured Gaussian primitive; a three-dimensional reconstruction unit, configured to perform three-dimensional reconstruction using the complete structured Gaussian primitives to obtain a three-dimensional reconstruction model corresponding to the target object.

[0014] According to a third aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the steps of any one of the methods described in the first aspect are implemented.

[0015] According to a fourth aspect, there is provided an electronic device, comprising:

[0016] One or more processors; and a memory associated with the one or more processors, the memory being used to store program instructions, and the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of the first aspects above.

[0017] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0018] 1) This application uses an optimal transmission algorithm to convert the unstructured Gaussian representation of the target object into a structured Gaussian representation, and uses the text description of the target object to complete the structured Gaussian using a diffusion model, and then performs three-dimensional reconstruction based on the structured Gaussian. This application uses the optimal transmission algorithm, the text description of the target object, and the diffusion model to gradually optimize the Gaussian representation of the target object, which can effectively solve the defects of traditional unstructured Gaussian in three-dimensional reconstruction, and accurately complete the Gaussian, so as to achieve fine three-dimensional reconstruction of the object.

[0019] 2) This application uses image segmentation, feature point extraction and clustering algorithms to obtain unstructured Gaussian primitives of the target object, which enhances the accuracy of feature extraction, reduces computational complexity and simplifies the generation process of unstructured Gaussian primitives.

[0020] 3) This application adjusts the density and resolution of the structured Gaussian primitives by setting a scaling factor, and can flexibly control the resolution of the three-dimensional model under different computing resources and accuracy requirements to meet the needs of different application scenarios.

[0021] 4) This application encodes the text description into a high-dimensional vector and fuses it with a structured Gaussian primitive, which helps the model understand the high-level semantic information of the target object and realizes the joint modeling of visual data and text data.

[0022] 5) This application uses a combination of low-rank matrix parameters and language models to more effectively extract and express key information in text descriptions, so that the text information can be better integrated with the Gaussian primitives generated during the three-dimensional reconstruction process, thereby improving the accuracy of the reconstructed model and the detail restoration effect.

[0023] 6) This application introduces control information when the diffusion model generates structured Gaussian primitives, uses a control network to generate guidance information, and combines the guidance information with the fusion vector to make the generated Gaussian primitives more accurate and in line with expectations. This guidance mechanism ensures that the model follows a specific direction when completing, thereby avoiding interference from irrelevant information and improving the stability and reliability of 3D reconstruction.

[0024] 7) This application first generates a pre-reconstructed model, and then generates two-dimensional images of multiple perspectives through rasterization, so as to analyze and determine the feature information and correspondence of different perspectives. This method can enhance the detail performance and visual consistency of the three-dimensional model, and ensure that the reconstructed model can accurately display the true shape and characteristics of the target object at different angles.

[0025] Of course, any product implementing the present application does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0027] Figure 1 A flowchart of a three-dimensional reconstruction method based on structured three-dimensional Gaussian representation provided in an embodiment of the present application;

[0028] Figure 2A schematic diagram of a process of a three-dimensional reconstruction method based on structured three-dimensional Gaussian representation provided in an embodiment of the present application;

[0029] Figure 3 A structural block diagram of a three-dimensional reconstruction device based on structured three-dimensional Gaussian representation provided in an embodiment of the present application;

[0030] Figure 4 A schematic block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0032] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0033] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0034] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0035] At present, there are some 3D reconstruction technologies, which usually directly perform 3D reconstruction from unstructured Gaussian primitives. However, due to the lack of organization and strict topological constraints of unstructured data, the generated 3D model is usually difficult to achieve high-precision detail representation, especially in the details of complex shapes, where the unstructured representation may not be accurate enough. At the same time, when performing 3D reconstruction on a target object, multiple 2D images containing different perspectives of the target object are usually used as target images for 3D reconstruction. However, these target images usually have a certain degree of perspective loss, which results in the generated 3D reconstruction model being incomplete, morphologically distorted, and having surface discontinuities or holes, which seriously affects the quality of 3D reconstruction.

[0036] In view of this, the present application provides a new idea. Figure 1 A flow chart of a 3D reconstruction method based on structured 3D Gaussian representation provided in an embodiment of the present application, such as Figure 1 As shown in , the method may include the following steps:

[0037] Step 101: Acquire a target image of multiple perspectives including a target object, wherein the target image of the multiple perspectives has a perspective missing relative to the target image of the full perspective.

[0038] Step 102: Generate an unstructured Gaussian primitive representation corresponding to the target object according to the target image.

[0039] Step 103: Establish a transfer matrix describing the distribution of unstructured Gaussian primitives and the distribution of structured Gaussian primitives, solve the transfer matrix using an optimal transfer algorithm to obtain an optimal transfer matrix, and generate a structured Gaussian primitive representation corresponding to the target object from the unstructured Gaussian primitives according to the optimal transfer matrix.

[0040] Step 104: Obtain a text description of the target object, and use the pre-trained diffusion model to complete the structured Gaussian primitives according to the text description to obtain a complete structured Gaussian primitives.

[0041] Step 105: Perform three-dimensional reconstruction using the complete structured Gaussian primitives to obtain a three-dimensional reconstructed model corresponding to the target object.

[0042] It can be seen from the above process that the present application uses the optimal transmission algorithm to convert the unstructured Gaussian representation of the target object into a structured Gaussian representation, and uses the text description of the target object to complete the structured Gaussian using the diffusion model, and then performs three-dimensional reconstruction based on the structured Gaussian. The present application uses the optimal transmission algorithm, the text description of the target object and the diffusion model to gradually optimize the Gaussian representation of the target object, which can effectively solve the defects of the traditional unstructured Gaussian in three-dimensional reconstruction and realize the fine reconstruction and completion of the object.

[0043] The following describes in detail each step in the above process and the effects that can be further produced in combination with the embodiments. First, the above step 101, i.e., "obtaining a target image of multiple perspectives including a target object, where the target image of the multiple perspectives has a perspective loss relative to the target image of the full perspective", is described in detail in combination with the embodiments.

[0044] The present application aims to perform three-dimensional reconstruction on a target image, which is an image containing a target object. The present application obtains multiple target images containing multiple perspectives and provides multi-perspective information through different imaging angles. Among them, the present application can realize a three-dimensional reconstruction operation when the target image has a perspective loss. Therefore, the target images of multiple perspectives obtained by the present application have a perspective loss relative to the target image of the full perspective.

[0045] The target image can be obtained in a variety of ways, for example, through a public data platform, or by shooting with different imaging angles using a camera. After the image containing the target object is obtained by the data platform or the camera, the image can be pre-processed by cutting, geometric correction, etc., to obtain a higher quality target image.

[0046] The above step 102, namely "generating an unstructured Gaussian primitive representation corresponding to the target object according to the target image" is described in detail below in conjunction with an embodiment.

[0047] Unstructured Gaussian primitives are a probabilistic representation of the distribution of feature points of an object. They are usually composed of parameters such as the mean, variance, and weight of the distribution and are used to describe the characteristics of a local area on the surface of an object. This representation constructs a point cloud or continuous distribution model in three-dimensional space, which can reflect the shape and structure of the object to a certain extent.

[0048] According to the target image, obtaining the unstructured Gaussian primitive representation corresponding to the target object can be achieved in a variety of ways. As a feasible way, point clouds can be extracted from multiple target images, and a three-dimensional point cloud representation of the target object can be obtained through multi-view image matching technology, such as Structure from Motion (SFM) or Multi-View Stereo (MVS), and then Gaussian fitting is performed on the point cloud to generate unstructured Gaussian primitives.

[0049] Another possible implementation method includes the following steps:

[0050] First, the target image can be segmented to obtain the target object area. In addition to the target object, the target image may also contain some other objects or background patterns. The target object area in the target image needs to be extracted through image segmentation. The area obtained by image segmentation is the target object area. Among them, image segmentation can be implemented using algorithms such as edge detection and semantic segmentation.

[0051] After that, feature points are extracted from the target object area to obtain multiple feature points. Feature point extraction aims to identify key points that can uniquely describe the characteristics of the object from the two-dimensional image of the target object. These feature points usually have the following properties: stability: they can remain consistent under different viewing angles, lighting and noise conditions; distinguishability: they have good distinguishability and can be used to match the same feature in different images; repeatability: they can appear repeatedly in multiple images, providing necessary data support for three-dimensional reconstruction.

[0052] Feature point extraction can be achieved using a variety of algorithms, for example, using the Harris corner detection method to find corners with strong responses by calculating the autocorrelation matrix of the image. Or using the scale-invariant feature transformation algorithm to detect extreme points in multi-scale space to extract feature points that are invariant to scale and rotation.

[0053] A clustering algorithm is used to cluster the feature points, and the clustering results are used to fit the Gaussian distribution to obtain unstructured Gaussian basis elements. Specifically, a set of feature points extracted from the image is selected as the input of the clustering algorithm; the feature points are clustered using the selected clustering algorithm. The clustering results will form multiple clusters, each of which contains feature points with similar characteristics; to evaluate the effectiveness of clustering, indicators such as the silhouette coefficient and the Davies-Bouldin index can be used to evaluate the quality of clustering.

[0054] After clustering is completed, a Gaussian distribution is fitted based on the clustering results. Specifically, for each cluster, a set of feature points is extracted; for each cluster, its mean, covariance matrix and other statistical parameters are calculated to form the basis of an unstructured Gaussian distribution. These parameters can describe the distribution of feature points in space. Using the obtained parameters, an unstructured Gaussian distribution model is constructed. Each unstructured Gaussian basis element is composed of multiple Gaussian distributions, which are defined by their mean and variance, and describe the distribution of feature points in the area. At the same time, for multiple clustering results, it may be necessary to merge multiple unstructured Gaussian basis elements to form a complete unstructured Gaussian basis element representation. For example, similar Gaussian basis elements can be merged into a larger Gaussian basis element according to needs.

[0055] The present application can also perform standardization processing on unstructured Gaussian basis elements. The standardization processing can balance the differences between different unstructured Gaussian basis elements and avoid the influence of numerical deviation on the transfer matrix solution process, thereby improving the accuracy of the optimal transfer matrix solution.

[0056] The following is a detailed description of step 103, i.e., "establishing a transfer matrix describing the distribution of unstructured Gaussian primitives and the distribution of structured Gaussian primitives, solving the transfer matrix using an optimal transfer algorithm to obtain an optimal transfer matrix, and generating a structured Gaussian primitive representation corresponding to the target object from the unstructured Gaussian primitives based on the optimal transfer matrix" in conjunction with an embodiment.

[0057] The transfer matrix is ​​a mathematical tool used to describe the relationship between two distributions. In this application, the transfer matrix is ​​used to represent the correspondence between the unstructured Gaussian primitive distribution and the structured Gaussian primitive distribution. Specifically, each element of the transfer matrix can be regarded as the "transmission amount" from one primitive distribution (unstructured) to another primitive distribution (structured).

[0058] The steps of constructing the transfer matrix include: first, the feature points in the unstructured Gaussian primitives and the structured Gaussian primitives need to be matched. By calculating the distance or similarity measure between the two, a corresponding relationship can be established; according to the matching of the feature points, the elements of the transfer matrix are assigned values. Generally speaking, each element T of the matrix ij The "transmission amount" from the i-th unstructured Gaussian primitive to the j-th structured Gaussian primitive can be calculated by a distance metric (such as Euclidean distance) or a similarity function. At the same time, in order to ensure the validity of the transmission matrix, it may be necessary to normalize it so that the sum of each row or column is equal to 1. This helps to treat the transmission matrix as a probability distribution in subsequent processing.

[0059] The optimal transmission algorithm is a mathematical tool used to find the best solution to transform one distribution into another. Its goal is to minimize the transmission cost. The optimal transmission algorithm is used to solve the transmission matrix. The optimal transmission matrix can be obtained by the following steps:

[0060] First, the transfer cost function is defined. The transfer cost function is a function based on distance or similarity measure, which represents the "transportation cost" from unstructured primitives to structured primitives.

[0061] After that, the transfer matrix is ​​solved by an optimization algorithm (such as Sinkhorn distance, linear programming, etc.) to find the optimal transfer matrix. Each element in the optimal transfer matrix represents the optimal amount of transmission from each unstructured Gaussian basis element to the structured Gaussian basis element. An iterative algorithm is used to solve the optimal transfer matrix until the convergence condition is reached. The iteration process adjusts the elements of the transfer matrix to reduce the total transmission cost.

[0062] Preferably, a regularization term can be added to the transmission cost function, and the regularization term can introduce prior knowledge or other information (such as marginal distribution, smoothness, sparsity, etc.), so that the optimization problem has a clearer direction.

[0063] After obtaining the optimal transmission matrix, the structured Gaussian representation corresponding to the target object is obtained according to the optimal transmission matrix. This process can be implemented in a variety of ways. For example, the unstructured source Gaussian elements can be directly combined in a weighted manner to obtain the structured Gaussian representation of the target object; or the unstructured Gaussian elements can be weighted and combined into structured Gaussian elements using the weights in the optimal transmission matrix; the expectation maximization algorithm can also be used to iteratively update the parameters of the structured Gaussian elements so that the distribution of the structured Gaussian elements matches the transmission rules guided by the optimal transmission matrix as much as possible.

[0064] Structured Gaussian primitives can be represented in many ways, such as point cloud, where the Gaussian primitives generated by the optimal transfer matrix are regarded as sparse point clouds distributed in space, and the three-dimensional coordinates, density and other attributes of each point are recorded. The resolution of the structured Gaussian primitive representation is adjusted according to the scaling factor, which includes the number of samples or the sampling density.

[0065] The structured Gaussian primitives can also be represented by a structured grid, which refers to dividing the three-dimensional space into regular grid cells, each of which can contain one or more Gaussian primitives for describing the spatial characteristics of an object in the cell, wherein the denser the grid (the higher the resolution), the more detailed information each cell contains. Therefore, the resolution of the structured Gaussian primitive representation can be adjusted according to a scaling factor, which includes the size of the structured grid.

[0066] Resolution here usually refers to the fineness or sampling density of the Gaussian primitives in 3D space. The resolution of the Gaussian primitives can be adjusted by different representation methods to maintain the aspect ratio of the object and achieve different levels of detail through specific scaling factors. The resolution is adjusted by increasing or decreasing the value of the scaling factor. For example, a low resolution can use a smaller voxel grid to represent the overall structure, while a higher resolution uses a denser voxel grid to capture more details.

[0067] The above step 104, i.e., "obtaining a text description of the target object, and completing the structured Gaussian primitives according to the text description using a pre-trained diffusion model to obtain a complete structured Gaussian primitives" is described in detail below in conjunction with an embodiment.

[0068] This application uses the pre-trained diffusion model to complete the structured Gaussian primitives. The diffusion model is a generative model that gradually converts data into noise and then reversely restores the data from the noise. In the scenario of Gaussian primitive structured completion, the diffusion model can complete the missing or incomplete parts from the existing structured Gaussian primitives by simulating the gradual generation process of data.

[0069] When the diffusion model is used for completion, the text description of the target object is used to guide the reasoning of the diffusion model. Figure 2 A process diagram of a three-dimensional reconstruction method based on a structured three-dimensional Gaussian representation provided for an embodiment of the present application. Wherein, the text description of the target object is a text that can describe the characteristics of the target object, including but not limited to: the text description can be the category information of the target object, such as "cat", "building", "plant", etc., and the category information can help the diffusion model generate Gaussian primitives that meet the characteristics of a specific category; it can be a description of the shape, size, position or spatial structure of the target object, such as "circular", "rectangular", "distributed in the left area", etc., which can affect the distribution parameters of the Gaussian primitives, so that the completion result is more in line with the geometric characteristics of the object; it can be a description of the visual data, color or texture of the target object, such as "blue and striped", which can help the diffusion model generate more recognizable features when generating the completed Gaussian primitives; for a target object with motion characteristics, the text description can be a description of its behavior or posture, such as "moving forward", "leaning", etc., which can help complete the distribution of Gaussian primitives that reflect dynamics.

[0070] Text descriptions can be obtained in a variety of ways, including manual annotation and automated extraction. Manual annotation is a precise description of an object provided directly by a person, which is suitable for more sophisticated scene requirements; automated extraction can use image label generation tools, feature extraction models, or natural language processing models to automatically generate descriptions from data. In addition, you can also use knowledge bases or metadata tags to find detailed descriptions of target objects, or provide descriptions of specific requirements through user input to meet personalized completion requirements. These description methods can provide rich semantic information for the generation model and enhance the accuracy and consistency of the completion results.

[0071] After obtaining the text description, the text description and the structured Gaussian primitives can be input into the pre-trained diffusion model together. The diffusion model can complete the structured Gaussian primitives according to the text description to obtain a complete structured Gaussian primitive.

[0072] As an implementable method, the present application may further perform encoding processing on the text description, converting the text description into a high-dimensional vector representation; fusing the high-dimensional vector and the structured Gaussian primitive to obtain a fused vector, and generating the complete structured Gaussian primitive from the diffusion model based on the fused vector.

[0073] Among them, the text description can be further encoded by traditional encoding methods, such as sentence encoding, multimodal encoding, etc.; it can also be implemented by using a language model model; the high-dimensional vector representation of the text description is fused with the structured Gaussian basis unit. This step can be achieved in several ways, such as weighted summation, convolution, attention mechanism, etc. The purpose of fusion is to combine the semantic information of the text with the information of the Gaussian distribution to form a fusion vector that contains the characteristics of both.

[0074] As another feasible method, a large language model can be used to generate a high-dimensional vector representation corresponding to the text description. In particular, the Low-Rank Adaptation of Large Language Models (LoRA) technology of large language models can be used to enable large language models to adapt to specific tasks by loading trained low-rank matrix parameters. Specifically, the trained low-rank matrix parameters are loaded, and the low-rank matrix parameters are combined with the parameter matrix of the language model to obtain a combined matrix; the high-dimensional vector representation corresponding to the text description is generated using the combined matrix. Among them, the combination of the low-rank matrix parameters and the parameter matrix of the language model can add, weighted sum, convolve, and other processes to the two matrices, so that the low-rank matrix is ​​superimposed on the parameters of the original model, so that the accuracy of the results generated by the model is higher.

[0075] The language model can be based on a large language model LLM (Large Language Model) or an ordinary pre-trained language model. Among them, the large language model, or LLM, refers to a deep learning model trained with a large amount of text data, which can generate natural language text or understand the meaning of language text. It is characterized by large scale and huge number of parameters (usually reaching more than 10 billion levels), and is usually based on a deep learning architecture such as a Transformer architecture. The difference between LLM and ordinary pre-trained language models lies in the parameter scale. When the parameter scale exceeds a certain level, the model achieves significant performance improvement and exhibits capabilities that do not exist in small models, such as in-context learning capabilities, which can learn complex patterns in language and perform a wide range of tasks, including text summarization, translation, sentiment analysis, multi-round dialogue, etc. Therefore, in order to distinguish it from traditional pre-trained language models, such models with a parameter scale exceeding a certain level are called LLM. In general, it can be considered that a language model with a parameter scale of more than 10 billion based on a deep learning architecture is called a large language model. Common LLMs include: GTP-3 (Generative Pre-trained Transformer 3), T5 (Text-to-Text Transfer Transformer), GTP-4, PaLM (a large language model proposed by Google), LLaMA (Large Language Model Meta AI, a large language model released by Meta AI), and so on.

[0076] As another practicable manner, when the present application generates a complete structured Gaussian primitive based on the fusion vector using the diffusion model, the completion process of the Gaussian primitive can also be guided by a control network (ControlNet), and ControlNet can be used as an effective control mechanism to make the model follow specific structural information during the generation process. Specifically, control information is obtained, and guidance information is generated based on the control information using the control network; the complete structured Gaussian primitive is generated by the diffusion model based on the guidance information and the fusion vector.

[0077] Among them, the control information is used as the input of the control network to constrain the generation process of the diffusion model. The control information can be a structural diagram of the target object, such as an edge map, a depth map, a posture map, etc., or a Gaussian basis distribution map describing the density or position distribution of the target object in space, etc. The control network outputs guidance information based on the control information, and the guidance information constrains the generation process of the diffusion model to ensure that the generated content meets the input conditional information.

[0078] The above step 105, namely "using the complete structured Gaussian primitives to perform 3D reconstruction to obtain a 3D reconstructed model corresponding to the target object" is described in detail below in conjunction with the embodiments.

[0079] First, the complete structured Gaussian primitives are preprocessed to extract features containing three-dimensional spatial position and distribution information. Then, based on the structured Gaussian primitives, they are converted into three-dimensional point cloud data of the target object through a three-dimensional reconstruction algorithm. In this process, a density estimation algorithm based on Gaussian distribution can be used to achieve point cloud completion and density adjustment of the Gaussian primitives, so that the generated point cloud is more consistent with the morphological characteristics of the target object. Subsequently, a surface reconstruction algorithm is used to convert the point cloud data into a continuous three-dimensional mesh model, for example, through Poisson surface reconstruction, Marching Cubes or Delaunay triangulation and other methods, a three-dimensional model with a complete surface structure is generated.

[0080] Preferably, the generated three-dimensional model can be optimized, for example, by removing redundant noise and smoothing, to improve the accuracy and visual effect of the model, thereby obtaining a three-dimensional reconstructed model of the target object that conforms to the structured Gaussian primitives.

[0081] As an implementable method, the present application can optimize the three-dimensional reconstruction model using rasterization technology. Specifically: use the complete structured Gaussian primitives to perform three-dimensional reconstruction to obtain a pre-reconstructed model of the target object; rasterize the pre-reconstructed model to obtain two-dimensional images at multiple angles, the two-dimensional images including feature information of the corresponding viewing angles; determine the correspondence between different viewing angles based on the feature information; and obtain the three-dimensional reconstruction model of the target object based on the correspondence.

[0082] Among them, rasterization technology is widely used in view synthesis and multi-view geometric reconstruction. This technology generates two-dimensional view images at different angles by projecting a three-dimensional object from different perspectives. These view images can represent the depth information and geometric structure of the scene. By adopting a rasterization processing method, the target three-dimensional model is converted into multiple two-dimensional projection images according to the set camera perspective and internal and external parameter parameters, which can effectively capture the features, textures and details of the object surface. The present application first renders the pre-reconstructed model from multiple set angles into a two-dimensional image through rasterization, so that the feature information of different perspectives can be fully displayed in the two-dimensional image. Then, by analyzing and extracting features from the multi-view images one by one, the adjacent views are matched by features using methods such as image registration or stereo vision to determine the correspondence between different perspectives, and the accuracy and stability of reconstruction can be improved by increasing the number of perspectives. Finally, according to the above-mentioned optimized correspondence, the three-dimensional reconstruction model of the target object is obtained. Among them, the generation of the pre-reconstructed model and the generation of the three-dimensional reconstruction model can be obtained by the three-dimensional reconstruction method mentioned above, and no specific restrictions are made here.

[0083] In this application, the diffusion model obtained by pre-training can be trained using the following method:

[0084] Acquire training data including a plurality of training samples, wherein the training samples are image samples including a target object, and the image samples include a full-view image representation of the target object sample; use the image samples to obtain an unstructured Gaussian primitive corresponding to the target object, structure the unstructured Gaussian primitive into a Gaussian primitive, and obtain a structured Gaussian primitive sample corresponding to the target object, wherein the structured Gaussian primitive sample is represented in a three-dimensional grid space; sample the three-dimensional grid space of the structured Gaussian primitive sample to obtain a three-dimensional local block, remove the area corresponding to the three-dimensional local block in the three-dimensional grid space of the target object sample, and obtain an input sample; input the input sample into the diffusion model, and obtain an output Gaussian primitive for the input sample generated by the diffusion model based on the input sample, wherein the training objectives include: minimizing the difference between the output Gaussian primitive for the input sample and the structured Gaussian primitive sample.

[0085] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0086] According to an embodiment of another aspect, a three-dimensional reconstruction apparatus based on structured three-dimensional Gaussian representation is provided. Figure 3 A schematic block diagram of a 3D reconstruction apparatus based on structured 3D Gaussian representation according to an embodiment is shown, wherein the apparatus 300 comprises:

[0087] The target image acquisition unit 301 is configured to acquire target images of multiple perspectives including the target object, where the target images of the multiple perspectives have perspective missing relative to the target image of the full perspective.

[0088] The unstructured Gaussian primitive generation unit 302 is configured to generate an unstructured Gaussian primitive representation corresponding to the target object according to the target image.

[0089] The structured Gaussian primitive generation unit 303 is configured to establish a transmission matrix describing the distribution of unstructured Gaussian primitives and the distribution of structured Gaussian primitives, solve the transmission matrix using an optimal transmission algorithm to obtain an optimal transmission matrix, and generate a structured Gaussian primitive representation corresponding to the target object from the unstructured Gaussian primitives based on the optimal transmission matrix.

[0090] The Gaussian primitive completion unit 304 is configured to obtain a text description of the target object, and use the pre-trained diffusion model to complete the structured Gaussian primitive according to the text description to obtain a complete structured Gaussian primitive.

[0091] The 3D reconstruction unit 305 is configured to perform 3D reconstruction using the complete structured Gaussian primitives to obtain a 3D reconstructed model corresponding to the target object.

[0092] As one of the feasible ways, the unstructured Gaussian primitive generation unit 302 can be configured as follows: performing image segmentation on the target image to obtain the target object area; performing feature point extraction on the target object area to obtain multiple feature points; clustering the feature points using a clustering algorithm, and performing Gaussian distribution fitting using the clustering results to obtain unstructured Gaussian primitives.

[0093] As one of the possible implementation methods, the structured Gaussian primitives are represented by a structured grid, and the resolution of the structured Gaussian primitives is adjusted according to a scaling factor, where the scaling factor includes the size of the structured grid; or, the structured Gaussian primitives are represented by a point cloud, and the resolution of the structured Gaussian primitives is adjusted according to a scaling factor, where the scaling factor includes the number of samples or the sampling density.

[0094] As one of the feasible ways, when the Gaussian primitive completion unit 304 uses the pre-trained diffusion model to complete the structured Gaussian primitives according to the text description to obtain a complete structured Gaussian primitive, it can be configured as follows: encode the text description to obtain a high-dimensional vector representation corresponding to the text description; fuse the high-dimensional vector and the structured Gaussian primitive to obtain a fused vector; and generate the complete structured Gaussian primitive by the diffusion model according to the fused vector.

[0095] As one of the feasible ways, when encoding the text description and obtaining the high-dimensional vector representation corresponding to the text description, the Gaussian basis element completion unit 304 can be configured as follows: loading the trained low-rank matrix parameters, combining the low-rank matrix parameters with the parameter matrix of the language model to obtain a combined matrix; and using the combined matrix to generate the high-dimensional vector representation corresponding to the text description.

[0096] As one of the feasible ways, when the Gaussian basis element completion unit 304 generates the complete structured Gaussian basis element by the diffusion model according to the fusion vector, it can be configured as follows: obtaining control information, generating guidance information according to the control information by using the control network; and generating a complete structured Gaussian basis element by the diffusion model according to the guidance information and the fusion vector.

[0097] As one of the feasible ways, the three-dimensional reconstruction unit 305 can be configured as follows: using complete structured Gaussian primitives to perform three-dimensional reconstruction to obtain a pre-reconstructed model of the target object; rasterizing the pre-reconstructed model to obtain two-dimensional images at multiple angles, wherein the two-dimensional images include feature information of the corresponding viewpoints; determining the correspondence between different viewpoints based on the feature information; and generating a three-dimensional reconstruction model corresponding to the target object based on the correspondence.

[0098] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the unit described as a separate component may or may not be physically separated, and the component displayed as a unit may or may not be a physical unit, that is, it may be located in one place, or it may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative work.

[0099] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0100] In addition, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of any one of the methods in the aforementioned method embodiments are implemented.

[0101] And an electronic device, comprising: one or more processors; and a memory associated with the one or more processors, the memory being used to store program instructions, and the program instructions, when read and executed by the one or more processors, execute the steps of any of the methods described in the aforementioned method embodiments.

[0102] The present application also provides a computer program product, including a computer program, which implements the steps of any one of the methods in the aforementioned method embodiments when executed by a processor.

[0103] in, Figure 4 The architecture of the electronic device is shown as an example, which may include a processor 410, a video display adapter 411, a disk drive 412, an input / output interface 413, a network interface 414, and a memory 420. The processor 410, the video display adapter 411, the disk drive 412, the input / output interface 413, the network interface 414, and the memory 420 may be communicatively connected via a communication bus 430.

[0104] Among them, the processor 410 can be implemented by a general CPU, a microprocessor, an application specific integrated circuit (ASIC), or one or more integrated circuits, etc., to execute relevant programs to implement the technical solutions provided in this application.

[0105] The memory 420 can be implemented in the form of ROM (Read On ly Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 420 can store an operating system 421 for controlling the operation of the electronic device 400, and a basic input and output system (BIOS) 422 for controlling the low-level operation of the electronic device 400. In addition, a web browser 423, a data storage management system 424, and a three-dimensional reconstruction device 425 based on structured three-dimensional Gaussian representation, etc. can also be stored. The above-mentioned three-dimensional reconstruction device 425 based on structured three-dimensional Gaussian representation can be an application program that specifically implements the operations of the aforementioned steps in the embodiment of the present application. In short, when the technical solution provided by the present application is implemented by software or firmware, the relevant program code is stored in the memory 420 and is called and executed by the processor 410.

[0106] The input / output interface 413 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0107] The network interface 414 is used to connect to a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WI FI, Bluetooth, etc.).

[0108] The bus 430 comprises a pathway for transmitting information between the various components of the device (eg, the processor 410, the video display adapter 411, the disk drive 412, the input / output interface 413, the network interface 414, and the memory 420).

[0109] It should be noted that, although the above device only shows a processor 410, a video display adapter 411, a disk drive 412, an input / output interface 413, a network interface 414, a memory 420, a bus 430, etc., in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include components necessary for implementing the solution of the present application, and does not necessarily include all the components shown in the figure.

[0110] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly contributed to the prior art in the form of a computer program product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0111] The technical solution provided by the present application is described in detail above. The principle and implementation method of the present application are described in detail using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as limiting the present application.

Claims

1. A three-dimensional reconstruction method based on structured three-dimensional Gaussian representation, characterized in that: The method comprises: Acquire a target image of multiple viewing angles including a target object, wherein the target image of the multiple viewing angles has a viewing angle missing relative to the target image of the full viewing angle; Generating an unstructured Gaussian primitive representation corresponding to the target object according to the target image; Establishing a transmission matrix describing the distribution of unstructured Gaussian basis elements and the distribution of structured Gaussian basis elements, solving the transmission matrix using an optimal transmission algorithm to obtain an optimal transmission matrix, and generating a structured Gaussian basis element representation corresponding to the target object from the unstructured Gaussian basis elements according to the optimal transmission matrix; Acquire a text description of the target object, and use a pre-trained diffusion model to complete the structured Gaussian primitives according to the text description to obtain a complete structured Gaussian primitives; The complete structured Gaussian primitives are used to perform three-dimensional reconstruction to obtain a three-dimensional reconstruction model corresponding to the target object.

2. The method according to claim 1, characterized in that Generating an unstructured Gaussian primitive representation corresponding to the target object according to the target image includes: Performing image segmentation on the target image to obtain a target object area; Extracting feature points from the target object area to obtain a plurality of feature points; A clustering algorithm is used to cluster the feature points, and the clustering results are used to perform Gaussian distribution fitting to obtain unstructured Gaussian basis elements.

3. The method according to claim 1, characterized in that The method further comprises: The structured Gaussian primitives are represented by a structured grid, and a resolution of the structured Gaussian primitives is adjusted according to a scaling factor, wherein the scaling factor includes a size of the structured grid; Alternatively, the structured Gaussian primitives are represented by point clouds, and the resolution of the structured Gaussian primitives is adjusted according to a scaling factor, where the scaling factor includes a sampling number or a sampling density.

4. The method according to claim 1, characterized in that: The method of using the diffusion model obtained by pre-training to complete the structured Gaussian primitives according to the text description to obtain a complete structured Gaussian primitives includes: Encoding the text description to obtain a high-dimensional vector representation corresponding to the text description; fusing the high-dimensional vector and the structured Gaussian primitive to obtain a fused vector; The complete structured Gaussian primitives are generated from the diffusion model according to the fused vectors.

5. According to the method of claim 4, encoding the text description to obtain a high-dimensional vector representation corresponding to the text description comprises: Loading the trained low-rank matrix parameters, combining the low-rank matrix parameters with the parameter matrix of the language model to obtain a combined matrix; The combined matrix is ​​used to generate a high-dimensional vector representation corresponding to the text description.

6. The method according to claim 4, characterized in that The generating the complete structured Gaussian primitives from the diffusion model according to the fusion vector comprises: Acquire control information, and generate guidance information based on the control information using a control network; The complete structured Gaussian primitives are generated by the diffusion model according to the guidance information and the fusion vectors.

7. The method according to claim 1, characterized in that The three-dimensional reconstruction is performed using the complete structured Gaussian primitives to obtain a three-dimensional reconstruction model corresponding to the target object, comprising: Performing three-dimensional reconstruction using the complete structured Gaussian primitives to obtain a pre-reconstructed model of the target object; Rasterizing the pre-reconstructed model to obtain two-dimensional images at multiple angles, wherein the two-dimensional images include feature information corresponding to the viewing angles; Determining a correspondence between different perspectives according to the feature information; According to the corresponding relationship, a three-dimensional reconstruction model corresponding to the target object is generated.

8. A three-dimensional reconstruction device based on structured three-dimensional Gaussian representation, characterized in that: The device comprises: A target image acquisition unit is configured to acquire target images of multiple perspectives including a target object, wherein the target images of the multiple perspectives have perspective missing relative to the target image of the full perspective; An unstructured Gaussian primitive generation unit is configured to generate an unstructured Gaussian primitive representation corresponding to the target object according to the target image; The structured Gaussian primitive generation unit is configured to establish a transmission matrix describing the distribution of unstructured Gaussian primitives and the distribution of structured Gaussian primitives, solve the transmission matrix using an optimal transmission algorithm to obtain an optimal transmission matrix, and generate a structured Gaussian primitive representation corresponding to the target object from the unstructured Gaussian primitives according to the optimal transmission matrix; A Gaussian primitive completion unit is configured to obtain a text description of the target object, and complete the structured Gaussian primitive according to the text description using a pre-trained diffusion model to obtain a complete structured Gaussian primitive; The three-dimensional reconstruction unit is configured to perform three-dimensional reconstruction using the complete structured Gaussian primitives to obtain a three-dimensional reconstruction model corresponding to the target object.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. An electronic device, characterized in that: include: one or more processors; And a memory associated with the one or more processors, the memory is used to store program instructions, and when the program instructions are read and executed by the one or more processors, the steps of the method described in any one of claims 1 to 7 are executed.

Images