Training method and device of diffusion model for Gaussian primitive completion

Abstract

The embodiment of the invention discloses a method for training a diffusion model for Gaussian primitive completion, and the method comprises the steps: obtaining a training sample which is an image sample comprising a target object; obtaining a structured Gaussian primitive sample corresponding to the target object by using the image sample; sampling the structured Gaussian primitive sample to obtain a three-dimensional local block, and removing an area corresponding to the three-dimensional local block in the three-dimensional grid space of the target object sample to obtain an input sample; obtaining an output Gaussian primitive which is generated by the diffusion model based on the input sample and aims at the input sample; the training goal includes minimizing differences between output Gaussian primitives for input samples and structured Gaussian primitive samples. According to the method, the input sample is obtained in a mode of sampling and removing the structured Gaussian primitive, and the local missing region is complemented by the diffusion model, so that the generated Gaussian primitive can better fit an actual three-dimensional shape, and complementation with higher quality is realized.

Description

Technical Field

[0001] The present application relates to the technical field of computer vision and image processing, and in particular to a training method and device for a diffusion model for Gaussian basis element completion. Background Art

[0002] 3D reconstruction has a wide range of applications in many fields, including computer graphics, virtual reality, medical imaging, and robotic navigation. In 3D reconstruction tasks, image data is usually used to infer the structure of the target object. However, when the view is missing, it is challenging to directly use the existing view to generate a complete 3D structure. Traditional view completion models usually have the problem of insufficient accuracy, especially when the view is incomplete or data is missing. Summary of the invention

[0003] The present application provides a method for training a diffusion model for Gaussian basis element completion, thereby obtaining a diffusion model with better completion effect.

[0004] This application provides the following solutions:

[0005] According to a first aspect, a training method for a diffusion model for Gaussian primitive completion is provided, the method comprising: acquiring training data comprising a plurality of training samples, the training samples being image samples comprising a target object, the image samples comprising a full-view image representation of the target object samples; obtaining unstructured Gaussian primitives corresponding to the target object using the image samples, structuring the unstructured Gaussian primitives into Gaussian primitives, obtaining structured Gaussian primitive samples corresponding to the target object, the structured Gaussian primitive samples being represented in a three-dimensional grid space; sampling the three-dimensional grid space of the structured Gaussian primitive samples to obtain three-dimensional local blocks, removing the region corresponding to the three-dimensional local blocks in the three-dimensional grid space of the target object samples, obtaining input samples; inputting the input samples into the diffusion model to obtain output Gaussian primitives for the input samples generated by the diffusion model based on the input samples; the objectives of the training comprising: minimizing the difference between the output Gaussian primitives for the input samples and the structured Gaussian primitive samples.

[0006] According to an achievable method in an embodiment of the present application, the unstructured Gaussian primitives are structured into Gaussian primitives to obtain structured Gaussian primitive samples corresponding to the target object, including: establishing a transmission matrix that describes the distribution of unstructured Gaussian primitives and structured Gaussian primitives, solving the transmission matrix using an optimal transmission algorithm to obtain an optimal transmission matrix, and generating structured Gaussian primitive samples corresponding to the target object from the unstructured Gaussian primitives based on the optimal transmission matrix.

[0007] According to an achievable method in an embodiment of the present application, sampling the three-dimensional grid space of the structured Gaussian primitive sample to obtain a three-dimensional local block includes: sampling the three-dimensional grid primitives of the structured Gaussian sample according to a preset viewing angle to obtain a three-dimensional local block; or edge sampling the three-dimensional grid primitives of the structured Gaussian sample to obtain a three-dimensional local block.

[0008] According to an achievable manner in an embodiment of the present application, the method further includes: the training sample also includes a text description sample corresponding to the target object; the inputting the input sample into the diffusion model to obtain the output Gaussian primitives for the input sample generated by the diffusion model based on the input sample includes: encoding the text description sample to obtain a high-dimensional vector representation corresponding to the text description; fusing the high-dimensional vector representation and the input sample to obtain a fused vector; inputting the fused vector into the diffusion model, and generating the output Gaussian primitives by the diffusion model according to the fused vector.

[0009] According to an achievable method in an embodiment of the present application, sampling the three-dimensional grid space of the structured Gaussian primitive samples to obtain a three-dimensional local block includes: determining a sampling size corresponding to the target object based on the text description sample; sampling the three-dimensional grid space of the structured Gaussian primitive samples according to the sampling size to obtain a three-dimensional local block.

[0010] According to an achievable method in an embodiment of the present application, determining the sampling size corresponding to the target object based on the text description sample includes: performing image feature extraction on the image sample to obtain image features; encoding the text description sample using a natural language model to obtain text features; and using a convolutional neural network to generate the sampling size based on the image features and the text features.

[0011] According to a second aspect, a three-dimensional reconstruction method for completing Gaussian primitives based on a diffusion model is provided, the method comprising: acquiring a target object image containing a target object, wherein the target object image has a perspective missing relative to a full perspective representation of the target object; obtaining an unstructured Gaussian primitive corresponding to the target object based on the target object image; structuring the unstructured Gaussian primitives into Gaussian primitives to obtain structured Gaussian primitives corresponding to the target object; completing the structured Gaussian primitives using the diffusion model trained according to the first aspect to obtain a complete structured Gaussian primitive corresponding to the target object; and performing three-dimensional reconstruction using the complete structured Gaussian primitives to obtain a three-dimensional reconstructed model of the target object.

[0012] According to a third aspect, a training device for a diffusion model for Gaussian primitive completion is provided, the device comprising: a training data acquisition unit, configured to acquire training data comprising a plurality of training samples, the training samples being image samples comprising a target object, the image samples comprising a full-view image representation of the target object sample; a structured Gaussian primitive acquisition unit, configured to obtain an unstructured Gaussian primitive corresponding to the target object using the image samples, perform Gaussian primitive structuring on the unstructured Gaussian primitives, and obtain structured Gaussian primitive samples corresponding to the target object, the structured Gaussian primitive samples being represented in a three-dimensional grid space; an input sample acquisition unit, configured to sample the three-dimensional grid space of the structured Gaussian primitive samples, obtain a three-dimensional local block, remove the region corresponding to the three-dimensional local block in the three-dimensional grid space of the target object sample, and obtain an input sample; a model training unit, configured to input the input sample into the diffusion model, and obtain an output Gaussian primitive for the input sample generated by the diffusion model based on the input sample; the training objectives include: minimizing the difference between the output Gaussian primitive for the input sample and the structured Gaussian primitive sample.

[0013] According to a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the steps of the method described in any one of the first aspect or the second aspect are implemented.

[0014] According to a fifth aspect, an electronic device is provided, comprising: one or more processors; and a memory associated with the one or more processors, the memory being used to store program instructions, and the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of the first aspect or the second aspect.

[0015] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0016] 1) This application proposes a complete training method to obtain structured Gaussian primitives based on image samples, and obtain input samples by sampling and removing structured Gaussian primitives. The diffusion model completes the local missing areas of the three-dimensional image samples to generate a complete Gaussian primitive representation. This method ensures that during the training process, the generated Gaussian primitives can better fit the actual three-dimensional shape and achieve higher quality completion.

[0017] 2) This application converts unstructured Gaussian primitives into structured Gaussian primitives through an optimal transfer algorithm, which helps to generate a more regular three-dimensional representation. This enables the model to better utilize three-dimensional structural information during the completion process, thereby obtaining a higher quality completion result.

[0018] 3) By sampling the 3D grid space of the structured Gaussian primitives in a specific way (such as preset view sampling or edge sampling), appropriate 3D local blocks can be generated, allowing the diffusion model to focus on completing local features. This local sampling strategy improves the model's ability to understand local details from different viewpoints, thereby enhancing the model's completion accuracy in different areas and optimizing the local detail performance of 3D completion.

[0019] 4) This application adds text descriptions to the training data, so that the training data contains not only visual information of the target object, but also semantic information. The text description encoding vector is fused with the input sample, which helps the diffusion model learn more details and features about the target object from multimodal data, thereby generating a completed Gaussian primitive that better meets semantic requirements.

[0020] 5) This application determines the sampling size based on the text description, so that the model can flexibly adjust the sampling scale and area size according to the requirements of different text descriptions. This dynamic sampling method can improve the adaptability of the model to different objects and different semantic information, enhance the model's ability to capture detailed features, and make the completion effect more consistent with the size and shape characteristics of the actual object.

[0021] 6) This application combines image features with text features to generate a suitable sampling size, so that the sampling area can more accurately reflect the characteristics of the object. By using convolutional neural networks to jointly encode image and text features, the model can automatically generate the size of the optimal sampling area based on the specific description of the target object. This adaptive sampling further improves the accuracy of completion and the rationality of generation, and enhances the generalization ability of the model.

[0022] 7) This application performs 3D reconstruction based on the trained diffusion model, and can complete the structured Gaussian primitives when the input perspective is missing, thereby generating a complete 3D model. This method solves the reconstruction accuracy problem caused by incomplete 3D perspective, and effectively improves the integrity and visual consistency of 3D reconstruction by completing the Gaussian primitives, making the final generated 3D model closer to the real object.

[0023] Of course, any product implementing the present application does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0025] Figure 1 A flowchart of a method for training a diffusion model for Gaussian basis element completion provided in an embodiment of the present application;

[0026] Figure 2 A schematic diagram of the process of a method for training a diffusion model for Gaussian basis element completion provided in an embodiment of the present application;

[0027] Figure 3 A structural block diagram of a training device for a diffusion model for Gaussian basis element completion provided in an embodiment of the present application;

[0028] Figure 4 A structural block diagram of a three-dimensional reconstruction device for Gaussian basis element completion based on a diffusion model provided in an embodiment of the present application;

[0029] Figure 5 A schematic block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0031] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0032] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0033] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0034] There are already some Gaussian basis element completion models, such as completion based on convolutional neural networks (CNNs) and completion based on variational autoencoders (VAEs). However, these models are not good at capturing global structural relationships, especially when there is a large loss of perspective, the completion quality is poor. The advantages of diffusion models in image generation and completion tasks are gradually being discovered, especially in high-quality detail generation and modeling of complex structures. There is no precedent for training a diffusion model and using it for Gaussian completion in the existing technology.

[0035] In view of this, the present application provides a new idea. Figure 1 A flow chart of a method for training a diffusion model for Gaussian basis element completion provided in an embodiment of the present application, such as Figure 1 As shown in , the method may include the following steps:

[0036] Step 101: Acquire training data including a plurality of training samples, wherein the training samples are image samples including a target object, and the image samples include full-view image representations of the target object samples.

[0037] Step 102: using the image samples to obtain unstructured Gaussian primitives corresponding to the target object, structuring the unstructured Gaussian primitives into Gaussian primitives to obtain structured Gaussian primitive samples corresponding to the target object, wherein the structured Gaussian primitive samples are represented in a three-dimensional grid space.

[0038] Step 103: sampling the three-dimensional grid space of the structured Gaussian primitive samples to obtain three-dimensional local blocks, and removing the area corresponding to the three-dimensional local blocks in the three-dimensional grid space of the target object samples to obtain input samples.

[0039] Step 104: input the input sample into the diffusion model to obtain a complete Gaussian primitive for the input sample generated by the diffusion model based on the input sample; the training goal includes: minimizing the difference between the complete Gaussian primitive for the input sample and the corresponding complete Gaussian primitive sample.

[0040] As can be seen from the above process, this application proposes a complete training method, which obtains structured Gaussian primitives based on image samples, obtains input samples by sampling and removing structured Gaussian primitives, and uses a diffusion model to complete the local missing areas of the three-dimensional image samples, thereby generating a complete Gaussian primitive representation. This method ensures that during the training process, the generated Gaussian primitives can better fit the actual three-dimensional shape and achieve higher quality completion.

[0041] The following describes in detail each step in the above process and the effects that can be further produced in combination with the embodiments. First, the above step 101, namely "obtaining training data including multiple training samples, the training samples are image samples including target objects, and the image samples include full-view image representations of the target object samples", is described in detail in combination with the embodiments.

[0042] Figure 2 A process diagram of a method for training a diffusion model for Gaussian basis element completion provided in an embodiment of the present application. The present application obtains multiple target image samples containing multiple perspectives and provides multi-perspective information through different imaging angles. Among them, the image samples obtained by the present application include full-perspective image representations of target object samples.

[0043] The target image samples can be obtained in a variety of ways, for example, through a public data platform, or by shooting with different imaging angles by a camera. After the image containing the target object is obtained by the data platform or the camera, the image can be pre-processed by cutting, geometric correction, etc., so as to obtain a higher quality target image.

[0044] The above step 102, i.e., "obtaining unstructured Gaussian primitives corresponding to the target object using image samples, structuring the unstructured Gaussian primitives into Gaussian primitives, obtaining structured Gaussian primitive samples corresponding to the target object, wherein the structured Gaussian primitive samples are represented in a three-dimensional grid space" is described in detail below in conjunction with an embodiment.

[0045] Unstructured Gaussian primitives are a probabilistic representation of the distribution of feature points of an object. They are usually composed of parameters such as the mean, variance, and weight of the distribution and are used to describe the characteristics of a local area on the surface of an object. This representation constructs a point cloud or continuous distribution model in three-dimensional space, which can reflect the shape and structure of the object to a certain extent.

[0046] According to the target image samples, obtaining the unstructured Gaussian primitive representation corresponding to the target object can be achieved in a variety of ways. As a feasible way, point clouds can be extracted from multiple target images, and the three-dimensional point cloud representation of the target object can be obtained through multi-view image matching technology, such as Structure from Motion (SFM) or multi-view stereo vision (MVS), and then Gaussian fitting is performed on the point cloud to generate unstructured Gaussian primitives.

[0047] Another possible implementation method includes the following steps:

[0048] First, the target image can be segmented to obtain the target object area. In addition to the target object, the target image may also contain some other objects or background patterns. The target object area in the target image needs to be extracted through image segmentation. The area obtained by image segmentation is the target object area. Among them, image segmentation can be implemented using algorithms such as edge detection and semantic segmentation.

[0049] After that, feature points are extracted from the target object area to obtain multiple feature points. Feature point extraction aims to identify key points that can uniquely describe the characteristics of the object from the two-dimensional image of the target object. These feature points usually have the following properties: stability: they can remain consistent under different viewing angles, lighting and noise conditions; distinguishability: they have good distinguishability and can be used to match the same feature in different images; repeatability: they can appear repeatedly in multiple images, providing necessary data support for three-dimensional reconstruction.

[0050] Feature point extraction can be achieved using a variety of algorithms, for example, using the Harris corner detection method to find corners with strong responses by calculating the autocorrelation matrix of the image. Or using the scale-invariant feature transformation algorithm to detect extreme points in multi-scale space to extract feature points that are invariant to scale and rotation.

[0051] A clustering algorithm is used to cluster the feature points, and the clustering results are used to fit the Gaussian distribution to obtain unstructured Gaussian basis elements. Specifically, a set of feature points extracted from the image is selected as the input of the clustering algorithm; the feature points are clustered using the selected clustering algorithm. The clustering results will form multiple clusters, each of which contains feature points with similar characteristics; to evaluate the effectiveness of clustering, indicators such as the silhouette coefficient and the Davies-Bouldin index can be used to evaluate the quality of clustering.

[0052] After clustering is completed, a Gaussian distribution is fitted based on the clustering results. Specifically, for each cluster, a set of feature points is extracted; for each cluster, its mean, covariance matrix and other statistical parameters are calculated to form the basis of an unstructured Gaussian distribution. These parameters can describe the distribution of feature points in space. Using the obtained parameters, an unstructured Gaussian distribution model is constructed. Each unstructured Gaussian basis element is composed of multiple Gaussian distributions, which are defined by their mean and variance, and describe the distribution of feature points in the area. At the same time, for multiple clustering results, it may be necessary to merge multiple unstructured Gaussian basis elements to form a complete unstructured Gaussian basis element representation. For example, similar Gaussian basis elements can be merged into a larger Gaussian basis element according to needs.

[0053] The present application can also perform standardization processing on unstructured Gaussian basis elements. The standardization processing can balance the differences between different unstructured Gaussian basis elements and avoid the influence of numerical deviation on the transfer matrix solution process, thereby improving the accuracy of the optimal transfer matrix solution.

[0054] After obtaining the unstructured Gaussian primitives, the unstructured Gaussian primitives are structured to obtain structured Gaussian primitive samples corresponding to the target object. The structured Gaussian primitive samples are represented in a three-dimensional grid space.

[0055] Among them, the structuring of Gaussian primitives can be achieved in many ways, for example, using clustering algorithms to group unstructured Gaussian primitives and define the primitives in each group into a three-dimensional grid space; or by building a graph structure containing unstructured Gaussian primitives and using graph neural networks to learn the spatial relationship of Gaussian primitives. The model can generate structured representations based on neighboring relationships.

[0056] As an implementable method, a transmission matrix describing the distribution of unstructured Gaussian primitives and structured Gaussian primitives can be established, and the transmission matrix can be solved using the optimal transmission algorithm to obtain the optimal transmission matrix. According to the optimal transmission matrix, the structured Gaussian primitive samples corresponding to the target object are generated from the unstructured Gaussian primitives.

[0057] The transfer matrix is ​​a mathematical tool used to describe the relationship between two distributions. In this application, the transfer matrix is ​​used to represent the correspondence between the unstructured Gaussian primitive distribution and the structured Gaussian primitive distribution. Specifically, each element of the transfer matrix can be regarded as the "transmission amount" from one primitive distribution (unstructured) to another primitive distribution (structured).

[0058] The steps of constructing the transfer matrix include: first, the feature points in the unstructured Gaussian primitives and the structured Gaussian primitives need to be matched. By calculating the distance or similarity measure between the two, a corresponding relationship can be established; according to the matching of the feature points, the elements of the transfer matrix are assigned values. Generally speaking, each element T of the matrix ij The "transmission amount" from the i-th unstructured Gaussian primitive to the j-th structured Gaussian primitive can be calculated by a distance metric (such as Euclidean distance) or a similarity function. At the same time, in order to ensure the validity of the transmission matrix, it may be necessary to normalize it so that the sum of each row or column is equal to 1. This helps to treat the transmission matrix as a probability distribution in subsequent processing.

[0059] The optimal transmission algorithm is a mathematical tool used to find the best solution to transform one distribution into another. Its goal is to minimize the transmission cost. The optimal transmission algorithm is used to solve the transmission matrix. The optimal transmission matrix can be obtained by the following steps:

[0060] First, the transfer cost function is defined. The transfer cost function is a function based on distance or similarity measure, which represents the "transportation cost" from unstructured primitives to structured primitives.

[0061] After that, the transfer matrix is ​​solved by an optimization algorithm (such as Sinkhorn distance, linear programming, etc.) to find the optimal transfer matrix. Each element in the optimal transfer matrix represents the optimal amount of transmission from each unstructured Gaussian basis element to the structured Gaussian basis element. An iterative algorithm is used to solve the optimal transfer matrix until the convergence condition is reached. The iteration process adjusts the elements of the transfer matrix to reduce the total transmission cost.

[0062] Preferably, a regularization term can be added to the transmission cost function, and the regularization term can introduce prior knowledge or other information (such as marginal distribution, smoothness, sparsity, etc.), so that the optimization problem has a clearer direction.

[0063] After obtaining the optimal transmission matrix, the structured Gaussian representation corresponding to the target object is obtained according to the optimal transmission matrix. This process can be implemented in a variety of ways. For example, the unstructured source Gaussian elements can be directly combined in a weighted manner to obtain the structured Gaussian representation of the target object; or the unstructured Gaussian elements can be weighted and combined into structured Gaussian elements using the weights in the optimal transmission matrix; the expectation maximization algorithm can also be used to iteratively update the parameters of the structured Gaussian elements so that the distribution of the structured Gaussian elements matches the transmission rules guided by the optimal transmission matrix as much as possible.

[0064] The structured Gaussian primitives are represented by a structured grid, which means that the three-dimensional space is divided into regular grid cells, each of which can contain one or more Gaussian primitives for describing the spatial characteristics of an object in the cell, wherein the denser the grid (the higher the resolution), the more detailed information each cell contains. Therefore, the resolution of the structured Gaussian primitive representation can be adjusted according to a scaling factor, which includes the size of the structured grid.

[0065] Resolution here usually refers to the fineness or sampling density of the Gaussian primitives in 3D space. The resolution of the Gaussian primitives can be adjusted by different representation methods to maintain the aspect ratio of the object and achieve different levels of detail through specific scaling factors. The resolution is adjusted by increasing or decreasing the value of the scaling factor. For example, a low resolution can use a smaller voxel grid to represent the overall structure, while a higher resolution uses a denser voxel grid to capture more details.

[0066] The above step 103, i.e., "sampling the three-dimensional grid space of the structured Gaussian primitive samples to obtain three-dimensional local blocks, and removing the areas corresponding to the three-dimensional local blocks in the three-dimensional grid space of the target object samples to obtain input samples" is described in detail below in conjunction with an embodiment.

[0067] The application trains a diffusion model for Gaussian primitive completion, and the input of the model should be Gaussian primitives with perspective loss relative to the full perspective representation of the target object. Therefore, the application obtains structured Gaussian primitives with perspective loss by removing three-dimensional local blocks in a three-dimensional grid space.

[0068] As an implementable method, the three-dimensional grid primitives of the structured Gaussian sample can be sampled according to a preset perspective to obtain a three-dimensional local block. Before sampling, one or more preset perspectives need to be determined according to the task requirements. The preset perspective can be a fixed angle (such as the front, side, top), or a dynamically adjusted angle (such as multiple angles rotating around the target object). The choice of perspective can be determined based on the characteristics of the target object. For example, for a tetrahedral object, sampling can be done from four symmetric directions; for an object with a specific front or main perspective, sampling from the front perspective is usually preferred.

[0069] As another feasible approach, edge sampling can be performed on the three-dimensional grid primitives of the structured Gaussian sample to obtain three-dimensional local blocks. In a three-dimensional grid, the edge region can be defined as the grid primitives located near the surface of the object or the outer contour of the object. The edge region can be determined in a variety of ways: the distance field method can be used to calculate the distance from each grid cell to the surface of the object. The area with a smaller distance is regarded as the edge; the gradient change rate of the grid primitive can be calculated, and the larger gradient usually corresponds to the edge or surface area of ​​the object; the edge region can be identified by calculating the local curvature, and the area with larger curvature is usually the area where the contour or shape changes significantly.

[0070] The present application can also use the text description corresponding to the target object to train the model, determine the sampling size corresponding to the target object based on the text description sample; sample the three-dimensional grid space of the structured Gaussian primitive sample according to the sampling size to obtain a three-dimensional local block.

[0071] Among them, the text description of the target object is a text that can describe the characteristics of the target object, including but not limited to: the text description can be the category information of the target object, such as "cat", "building", "plant", etc., and the category information can help the diffusion model generate Gaussian primitives that meet the characteristics of a specific category; it can be a description of the shape, size, position or spatial structure of the target object, such as "circular", "rectangular", "distributed in the left area", etc., which can affect the distribution parameters of the Gaussian primitives, so that the completion result is more consistent with the geometric characteristics of the object; it can be a description of the visual data, color or texture of the target object, such as "blue and striped", which can help the diffusion model generate more recognizable features when generating completed Gaussian primitives; for target objects with motion characteristics, the text description can be a description of their behavior or posture, such as "moving forward", "leaning", etc., which can help complete the distribution of Gaussian primitives that reflect dynamics.

[0072] Text descriptions can be obtained in a variety of ways, including manual annotation and automated extraction. Manual annotation is a precise description of an object provided directly by a person, which is suitable for more sophisticated scene requirements; automated extraction can use image label generation tools, feature extraction models, or natural language processing models to automatically generate descriptions from data. In addition, you can also use knowledge bases or metadata tags to find detailed descriptions of target objects, or provide descriptions of specific requirements through user input to meet personalized completion requirements. These description methods can provide rich semantic information for the generation model and enhance the accuracy and consistency of the completion results.

[0073] Based on the text description samples, determining the sampling size corresponding to the target object can be achieved in many ways, for example, building or using an existing knowledge graph, marking the standard size information corresponding to different descriptions, matching the description with the concepts or entities in the knowledge graph, and extracting the corresponding size information.

[0074] As an implementable method, image features can be extracted from image samples to obtain image features; the text description samples are encoded using a natural language model to obtain text features; and a convolutional neural network is used to generate the sampling size based on the image features and the text features.

[0075] Specifically, image feature extraction can be achieved through image encoders. Commonly used image encoder networks include ResNet, VGG, EfficientNet, etc., which can convert the input two-dimensional image data into high-dimensional feature vectors. The extracted image feature vector can describe the basic shape, color, outline, complexity and other attributes of the target object. Through the high-dimensional representation of features, the content of the image sample can be effectively represented. Text feature extraction can be achieved through a natural language model, and the text description sample is subjected to feature extraction through a text encoder. The text description is processed into a vectorized feature representation, which can capture the semantic information in the text, including the description of the size, shape, and detail requirements of the object. For example, information such as "smaller" and "wider" in the description will be converted into part of the feature vector to represent the size requirement. The image and text feature vectors are concatenated or fused through weighted averaging, self-attention mechanism, etc. to form a multimodal feature vector. The fused multimodal feature vector is input into a convolutional neural network or a fully connected regression network to output the specific value of the sampling size. The parameters of each layer in the network are trained and optimized to learn to generate a suitable sampling size based on the feature fusion information.

[0076] The above step 104, i.e., "inputting the input sample into the diffusion model to obtain a complete Gaussian primitive for the input sample generated by the diffusion model based on the input sample; the training objectives include: minimizing the difference between the output Gaussian primitive for the input sample and the structured Gaussian primitive sample." is described in detail below in conjunction with an embodiment.

[0077] This application uses the pre-trained diffusion model to complete the structured Gaussian primitives. The diffusion model is a generative model that gradually converts data into noise and then reversely restores the data from the noise. In the scenario of Gaussian primitive structured completion, the diffusion model can complete the missing or incomplete parts from the existing structured Gaussian primitives by simulating the gradual generation process of data.

[0078] During the training process, the diffusion model generates a complete Gaussian primitive based on the input sample, and trains the model by minimizing the difference between the complete Gaussian primitive and the structured Gaussian primitive. The structured Gaussian primitive is the structured Gaussian primitive generated by the image sample in step 102. Since the image sample includes the full-view image representation of the target object sample, the structured Gaussian primitive is a complete Gaussian primitive representation, and the inference accuracy of the diffusion model can be determined by comparing it with the output Gaussian primitive generated by the diffusion model.

[0079] Among them, minimizing the difference usually uses a loss function (such as mean square error, MSE) to measure the difference between the output Gaussian primitives and the structured Gaussian primitives. In this way, the model will be guided to generate outputs that are more similar to the structured Gaussian primitives.

[0080] When the present application uses the diffusion model for completion, the text description of the target object can also be used to guide the reasoning of the diffusion model. Among them, the text description of the target object is a text that can describe the characteristics of the target object, including but not limited to: the text description can be the category information of the target object, such as "cat", "building", "plant", etc., and the category information can help the diffusion model generate Gaussian primitives that meet the characteristics of a specific category; it can be a description of the shape, size, position or spatial structure of the target object, such as "circular", "rectangular", "distributed in the left area", etc., which can affect the distribution parameters of the Gaussian primitives, so that the completion result is more in line with the geometric characteristics of the object; it can be a description of the visual data, color or texture of the target object, such as "blue and striped", which can help the diffusion model generate more recognizable features when generating the completed Gaussian primitives; for target objects with motion characteristics, the text description can be a description of its behavior or posture, such as "moving forward", "leaning", etc., which can help complete the distribution of Gaussian primitives that reflect dynamics.

[0081] After obtaining the text description, the text description and the structured Gaussian primitives can be input into the pre-trained diffusion model together. The diffusion model can complete the structured Gaussian primitives according to the text description to obtain a complete structured Gaussian primitive.

[0082] As an implementable method, the present application may further perform encoding processing on the text description, convert the text description into a high-dimensional vector representation; fuse the high-dimensional vector and the structured Gaussian primitive to obtain a fused vector, and generate the output Gaussian primitive by the diffusion model based on the fused vector.

[0083] Among them, the text description can be further encoded by traditional encoding methods, such as sentence encoding, multimodal encoding, etc.; it can also be implemented by using a language model model; the high-dimensional vector representation of the text description is fused with the structured Gaussian basis unit. This step can be achieved in several ways, such as weighted summation, convolution, attention mechanism, etc. The purpose of fusion is to combine the semantic information of the text with the information of the Gaussian distribution to form a fusion vector that contains the characteristics of both.

[0084] As another feasible method, a large language model can be used to generate a high-dimensional vector representation corresponding to the text description. In particular, the Low-Rank Adaptation of Large Language Models (LoRA) technology of large language models can be used to enable large language models to adapt to specific tasks by loading trained low-rank matrix parameters. Specifically, load the trained low-rank matrix parameters, combine the low-rank matrix parameters with the parameter matrix of the language model to obtain a combined matrix; use the combined matrix to generate a high-dimensional vector representation corresponding to the text description. Among them, the combination of the low-rank matrix parameters and the parameter matrix of the language model can add, weighted sum, convolve, etc. the two matrices, so that the low-rank matrix is ​​superimposed on the parameters of the original model, so that the result generated by the model is more accurate.

[0085] As another practicable manner, when the present application generates a complete structured Gaussian primitive based on the fusion vector using the diffusion model, the completion process of the Gaussian primitive can also be guided by a control network (ControlNet), and ControlNet can be used as an effective control mechanism to make the model follow specific structural information during the generation process. Specifically, control information is obtained, and guidance information is generated based on the control information using the control network; the complete structured Gaussian primitive is generated by the diffusion model based on the guidance information and the fusion vector.

[0086] Among them, the control information is used as the input of the control network to constrain the generation process of the diffusion model. The control information can be a structural diagram of the target object, such as an edge map, a depth map, a posture map, etc., or a Gaussian basis distribution map describing the density or position distribution of the target object in space, etc. The control network outputs guidance information based on the control information, and the guidance information constrains the generation process of the diffusion model to ensure that the generated content meets the input conditional information.

[0087] The present application also provides a three-dimensional reconstruction method for Gaussian primitive completion based on a diffusion model, the method comprising: obtaining a target object image containing a target object, wherein the target object image has a perspective missing relative to the full perspective representation of the target object; obtaining an unstructured Gaussian primitive corresponding to the target object according to the target object image; structuring the unstructured Gaussian primitive into Gaussian primitives to obtain a structured Gaussian primitive corresponding to the target object; completing the structured Gaussian primitive using a diffusion model trained in the above embodiment to obtain a complete structured Gaussian primitive corresponding to the target object; performing three-dimensional reconstruction using the complete structured Gaussian primitive to obtain a three-dimensional reconstruction model of the target object. By using the diffusion model trained in the above embodiment, the quality of Gaussian primitive completion is improved, and the accuracy of three-dimensional reconstruction is further improved.

[0088] This method performs three-dimensional reconstruction on a target image, which is an image containing a target object. This application obtains multiple target images containing multiple perspectives and provides multi-perspective information through different imaging angles. Among them, this application can realize a three-dimensional reconstruction operation when the target image has a perspective loss. Therefore, the target images of multiple perspectives obtained by this application have a perspective loss relative to the target image of the full perspective.

[0089] The target image can be obtained in a variety of ways, for example, through a public data platform, or by shooting with different imaging angles using a camera. After the image containing the target object is obtained by the data platform or the camera, the image can be pre-processed by cutting, geometric correction, etc., to obtain a higher quality target image.

[0090] For steps such as Gaussian structuring, the same method as in the above model training method can be used for structuring, or other methods can be used, which are not limited here.

[0091] When the complete structured Gaussian primitives are used to perform three-dimensional reconstruction to obtain the three-dimensional reconstruction model of the target object, it can be achieved through the following steps: First, the complete structured Gaussian primitives are preprocessed to extract features containing three-dimensional spatial position and distribution information. Then, based on the structured Gaussian primitives, they are converted into three-dimensional point cloud data of the target object through a three-dimensional reconstruction algorithm. In this process, a density estimation algorithm based on Gaussian distribution can be used to achieve point cloud completion and density adjustment of the Gaussian primitives, so that the generated point cloud is more consistent with the morphological characteristics of the target object. Subsequently, a surface reconstruction algorithm is used to convert the point cloud data into a continuous three-dimensional mesh model, for example, a three-dimensional model with a complete surface structure is generated through methods such as Poisson surface reconstruction, Marching Cubes or Delaunay triangulation.

[0092] Preferably, the generated three-dimensional model can be optimized, for example, by removing redundant noise and smoothing, to improve the accuracy and visual effect of the model, thereby obtaining a three-dimensional reconstructed model of the target object that conforms to the structured Gaussian primitives.

[0093] As an implementable method, the present application can optimize the three-dimensional reconstruction model using rasterization technology. Specifically: use the complete structured Gaussian primitives to perform three-dimensional reconstruction to obtain a pre-reconstructed model of the target object; rasterize the pre-reconstructed model to obtain two-dimensional images at multiple angles, the two-dimensional images including feature information of the corresponding viewing angles; determine the correspondence between different viewing angles based on the feature information; and obtain the three-dimensional reconstruction model of the target object based on the correspondence.

[0094] Among them, rasterization technology is widely used in view synthesis and multi-view geometric reconstruction. This technology generates two-dimensional view images at different angles by projecting a three-dimensional object from different perspectives. These view images can represent the depth information and geometric structure of the scene. By adopting a rasterization processing method, the target three-dimensional model is converted into multiple two-dimensional projection images according to the set camera perspective and internal and external parameter parameters, which can effectively capture the features, textures and details of the object surface. The present application first renders the pre-reconstructed model from multiple set angles into a two-dimensional image through rasterization, so that the feature information of different perspectives can be fully displayed in the two-dimensional image. Then, by analyzing and extracting features from the multi-view images one by one, the adjacent views are matched by features using methods such as image registration or stereo vision to determine the correspondence between different perspectives, and the accuracy and stability of reconstruction can be improved by increasing the number of perspectives. Finally, according to the above-mentioned optimized correspondence, the three-dimensional reconstruction model of the target object is obtained. Among them, the generation of the pre-reconstructed model and the generation of the three-dimensional reconstruction model can be obtained by the three-dimensional reconstruction method mentioned above, and no specific restrictions are made here.

[0095] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0096] According to an embodiment of another aspect, a training apparatus for a diffusion model for Gaussian basis completion is provided. Figure 3 A schematic block diagram of a training device for a diffusion model for Gaussian basis element completion according to an embodiment is shown. Figure 3 As shown, the device 300 includes:

[0097] The training data acquisition unit 301 is configured to acquire training data including a plurality of training samples, wherein the training samples are image samples including a target object, and the image samples include a full-view image representation of the target object sample;

[0098] The structured Gaussian primitive acquisition unit 302 is configured to obtain an unstructured Gaussian primitive corresponding to the target object by using the image sample, and perform Gaussian primitive structure on the unstructured Gaussian primitive to obtain a structured Gaussian primitive sample corresponding to the target object, wherein the structured Gaussian primitive sample is represented in a three-dimensional grid space;

[0099] An input sample acquisition unit 303 is configured to sample the three-dimensional grid space of the structured Gaussian primitive sample to obtain a three-dimensional local block, and remove the area corresponding to the three-dimensional local block in the three-dimensional grid space of the target object sample to obtain an input sample;

[0100] The model training unit 304 is configured to input the input sample into the diffusion model to obtain the output Gaussian primitives for the input sample generated by the diffusion model based on the input sample; the training objectives include: minimizing the difference between the output Gaussian primitives for the input sample and the structured Gaussian primitive samples.

[0101] As one of the feasible ways, when the structured Gaussian primitive acquisition unit 302 structures the unstructured Gaussian primitives into Gaussian primitives and obtains the structured Gaussian primitive samples corresponding to the target object, it can be configured as follows: establishing a transmission matrix describing the distribution of unstructured Gaussian primitives and the distribution of structured Gaussian primitives, solving the transmission matrix using an optimal transmission algorithm to obtain an optimal transmission matrix, and generating structured Gaussian primitive samples corresponding to the target object from the unstructured Gaussian primitives according to the optimal transmission matrix.

[0102] As one of the feasible ways, when the input sample acquisition unit 303 samples the three-dimensional grid space of the structured Gaussian primitive sample to obtain a three-dimensional local block, it can be configured as follows: sampling the three-dimensional grid primitives of the structured Gaussian sample according to a preset viewing angle to obtain a three-dimensional local block; or edge sampling the three-dimensional grid primitives of the structured Gaussian sample to obtain a three-dimensional local block.

[0103] As one of the feasible ways, the training samples also include text description samples corresponding to the target object; when the model training unit 304 inputs the input samples into the diffusion model to obtain the output Gaussian primitives for the input samples generated by the diffusion model based on the input samples, it can be configured as follows: encoding the text description samples to obtain the high-dimensional vector representation corresponding to the text description; fusing the high-dimensional vector representation and the input samples to obtain a fused vector; inputting the fused vector into the diffusion model, and generating the output Gaussian primitives by the diffusion model according to the fused vector.

[0104] As one of the feasible ways, when the input sample acquisition unit 303 samples the three-dimensional grid space of the structured Gaussian primitive samples to obtain the three-dimensional local blocks, it can be configured as follows: determining the sampling size corresponding to the target object according to the text description sample; sampling the three-dimensional grid space of the structured Gaussian primitive samples according to the sampling size to obtain the three-dimensional local blocks.

[0105] As one of the feasible ways, when the input sample acquisition unit 303 determines the sampling size corresponding to the target object according to the text description sample, it can be configured as follows: extracting image features from the image sample to obtain image features; encoding the text description sample using a natural language model to obtain text features; and generating the sampling size according to the image features and text features using a convolutional neural network.

[0106] According to an embodiment of another aspect, a three-dimensional reconstruction apparatus for performing Gaussian basis component completion based on a diffusion model is provided. Figure 4 A schematic block diagram of a 3D reconstruction apparatus for Gaussian primitive completion based on a diffusion model according to an embodiment is shown, as shown in FIG. Figure 4 As shown, the device 400 includes:

[0107] The target image acquisition unit 401 is configured to acquire a target object image containing a target object, wherein the target object image has a perspective loss relative to a full perspective representation of the target object.

[0108] The structured Gaussian primitive generation unit 402 is configured to obtain unstructured Gaussian primitives corresponding to the target object according to the target object image; and structure the unstructured Gaussian primitives into Gaussian primitives to obtain structured Gaussian primitives corresponding to the target object.

[0109] The Gaussian primitive completion unit 403 is configured to complete the structured Gaussian primitives using a diffusion model trained by a training device for a diffusion model for Gaussian primitive completion, so as to obtain a complete structured Gaussian primitive corresponding to the target object.

[0110] The 3D reconstruction unit 404 is configured to perform 3D reconstruction using the complete structured Gaussian primitives to obtain a 3D reconstructed model of the target object.

[0111] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the unit described as a separate component may or may not be physically separated, and the component displayed as a unit may or may not be a physical unit, that is, it may be located in one place, or it may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative work.

[0112] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0113] In addition, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of any one of the methods in the aforementioned method embodiments are implemented.

[0114] And an electronic device, comprising:

[0115] One or more processors; and a memory associated with the one or more processors, the memory being used to store program instructions, and the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of the aforementioned method embodiments.

[0116] The present application also provides a computer program product, including a computer program, which implements the steps of any one of the methods in the aforementioned method embodiments when executed by a processor.

[0117] in, Figure 5 The architecture of the electronic device is shown as an example, which may include a processor 510, a video display adapter 511, a disk drive 512, an input / output interface 513, a network interface 514, and a memory 520. The processor 510, the video display adapter 511, the disk drive 512, the input / output interface 513, the network interface 514, and the memory 520 may be communicatively connected via a communication bus 530.

[0118] The processor 510 may be implemented by a general-purpose CPU, a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., and is used to execute relevant programs to implement the technical solutions provided in this application.

[0119] The memory 520 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 520 can store an operating system 521 for controlling the operation of the electronic device 500, and a basic input and output system (BIOS) 522 for controlling the low-level operation of the electronic device 500. In addition, a web browser 523, a data storage management system 524, and a training device 525 for a diffusion model for Gaussian basis element completion, etc. can also be stored. The above-mentioned training device 525 for a diffusion model for Gaussian basis element completion can be an application program that specifically implements the operations of the aforementioned steps in the embodiment of the present application. In short, when the technical solution provided by the present application is implemented by software or firmware, the relevant program code is stored in the memory 520 and is called and executed by the processor 510.

[0120] The input / output interface 513 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0121] The network interface 514 is used to connect to a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0122] The bus 530 comprises a pathway for transmitting information between the various components of the device (eg, the processor 510, the video display adapter 511, the disk drive 512, the input / output interface 513, the network interface 514, and the memory 520).

[0123] It should be noted that, although the above device only shows the processor X10, the video display adapter 511, the disk drive 512, the input / output interface 513, the network interface 514, the memory 520, the bus 530, etc., in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the solution of the present application, and does not necessarily include all the components shown in the figure.

[0124] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly contributed to the prior art in the form of a computer program product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0125] The technical solution provided by the present application is described in detail above. The principle and implementation method of the present application are described in detail using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as limiting the present application.

Claims

1. A method for training a diffusion model for Gaussian basis completion, characterized in that: The method comprises: Acquire training data including a plurality of training samples, wherein the training samples are image samples including a target object, and the image samples include full-view image representations of the target object samples; Obtaining an unstructured Gaussian primitive corresponding to the target object by using the image sample, structuring the unstructured Gaussian primitive into a Gaussian primitive to obtain a structured Gaussian primitive sample corresponding to the target object, wherein the structured Gaussian primitive sample is represented in a three-dimensional grid space; Sampling the three-dimensional grid space of the structured Gaussian primitive sample to obtain a three-dimensional local block, and removing the area corresponding to the three-dimensional local block in the three-dimensional grid space of the target object sample to obtain an input sample; The input sample is input into the diffusion model to obtain an output Gaussian primitive for the input sample generated by the diffusion model based on the input sample. The training objective includes: minimizing the difference between the output Gaussian primitive for the input sample and the structured Gaussian primitive sample.

2. The method according to claim 1, characterized in that The step of structuring the unstructured Gaussian primitives to obtain structured Gaussian primitive samples corresponding to the target object comprises: A transmission matrix describing the distribution of unstructured Gaussian primitives and structured Gaussian primitives is established, and the transmission matrix is ​​solved by using an optimal transmission algorithm to obtain an optimal transmission matrix. According to the optimal transmission matrix, the structured Gaussian primitive samples corresponding to the target object are generated by the unstructured Gaussian primitives.

3. The method according to claim 1, characterized in that: The sampling of the three-dimensional grid space of the structured Gaussian primitive samples to obtain a three-dimensional local block comprises: Sampling the three-dimensional grid primitives of the structured Gaussian sample according to a preset viewing angle to obtain a three-dimensional local block; or, Edge sampling is performed on the three-dimensional grid primitives of the structured Gaussian sample to obtain a three-dimensional local block.

4. The method according to claim 1, characterized in that: The method further comprises: The training samples also include text description samples corresponding to the target objects; Inputting the input sample into the diffusion model to obtain an output Gaussian basis element for the input sample generated by the diffusion model based on the input sample comprises: Encoding the text description sample to obtain a high-dimensional vector representation corresponding to the text description; Fusing the high-dimensional vector representation and the input sample to obtain a fused vector; The fused vector is input into the diffusion model, and the output Gaussian primitive is generated by the diffusion model according to the fused vector.

5. The method according to claim 4, characterized in that The sampling of the three-dimensional grid space of the structured Gaussian primitive samples to obtain a three-dimensional local block comprises: Determining a sampling size corresponding to the target object according to the text description sample; The three-dimensional grid space of the structured Gaussian primitive samples is sampled according to the sampling size to obtain a three-dimensional local block.

6. The method according to claim 5, wherein determining the sampling size corresponding to the target object according to the text description sample comprises: Performing image feature extraction on the image sample to obtain image features; Encoding the text description sample using a natural language model to obtain text features; The sampling size is generated according to the image features and the text features by using a convolutional neural network.

7. A three-dimensional reconstruction method based on a diffusion model for Gaussian basis element completion, characterized in that: The method comprises: Acquire a target object image including the target object, wherein the target object image has a perspective loss relative to a full perspective representation of the target object; According to the target object image, obtaining an unstructured Gaussian basis element corresponding to the target object; The unstructured Gaussian primitives are structured into Gaussian primitives to obtain structured Gaussian primitives corresponding to the target object; The structured Gaussian primitives are completed using the diffusion model obtained by training according to claims 1 to 6 to obtain a complete structured Gaussian primitive corresponding to the target object; The complete structured Gaussian primitives are used to perform three-dimensional reconstruction to obtain a three-dimensional reconstructed model of the target object.

8. A training device for a diffusion model for Gaussian basis completion, characterized in that: The device comprises: A training data acquisition unit is configured to acquire training data including a plurality of training samples, wherein the training samples are image samples including a target object, and the image samples include a full-view image representation of the target object sample; a structured Gaussian primitive acquisition unit, configured to obtain an unstructured Gaussian primitive corresponding to the target object by using the image sample, and to structure the unstructured Gaussian primitive into a Gaussian primitive to obtain a structured Gaussian primitive sample corresponding to the target object, wherein the structured Gaussian primitive sample is represented in a three-dimensional grid space; An input sample acquisition unit is configured to sample the three-dimensional grid space of the structured Gaussian primitive sample to obtain a three-dimensional local block, and remove the area corresponding to the three-dimensional local block in the three-dimensional grid space of the target object sample to obtain an input sample; The model training unit is configured to input the input sample into the diffusion model to obtain the output Gaussian primitives for the input sample generated by the diffusion model based on the input sample, and the training objectives include: minimizing the difference between the output Gaussian primitives for the input sample and the structured Gaussian primitives sample.

9. An electronic device, characterized in that: include: one or more processors; And a memory associated with the one or more processors, the memory is used to store program instructions, and when the program instructions are read and executed by the one or more processors, the steps of the method described in any one of claims 1 to 7 are executed.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images