A training method and device of a diffusion model for Gaussian cell completion

Abstract

The embodiment of the application discloses a training method of a diffusion model for Gaussian cell completion, which comprises the following steps: obtaining a training sample, wherein the training sample is an image sample comprising a target object; obtaining a structured Gaussian cell sample corresponding to the target object by using the image sample; sampling the structured Gaussian cell sample to obtain a three-dimensional local block, removing a region corresponding to the three-dimensional local block in a three-dimensional grid space of the target object sample to obtain an input sample; obtaining an output Gaussian cell generated by the diffusion model based on the input sample; and training a target, which comprises minimizing the difference between the output Gaussian cell and the structured Gaussian cell sample. The input sample is obtained by sampling and removing the structured Gaussian cell, the local missing region is completed by the diffusion model, the generated Gaussian cell can better fit the actual three-dimensional shape, and higher quality completion is achieved.

Description

Technical Field

[0001] This application relates to the fields of computer vision and image processing technology, and in particular to a training method and apparatus for a diffusion model for Gaussian primitive completion. Background Technology

[0002] 3D reconstruction has wide applications in many fields, including computer graphics, virtual reality, medical imaging, and robot navigation. In 3D reconstruction tasks, image data is typically used to infer the structure of a target object. However, generating a complete 3D structure directly from the existing viewpoint is challenging when the viewpoint is missing. Traditional viewpoint completion models often suffer from insufficient accuracy, especially when the viewpoint is incomplete or data is missing. Summary of the Invention

[0003] This application provides a training method for a diffusion model for Gaussian primitive completion, thereby obtaining a diffusion model with better completion performance.

[0004] This application provides the following solution:

[0005] According to a first aspect, a training method for a diffusion model for Gaussian complement completion is provided. The method includes: acquiring training data comprising multiple training samples, wherein the training samples are image samples including a target object, and the image samples include a full-view image representation of the target object sample; obtaining unstructured Gaussian complements corresponding to the target object using the image samples; structuring the unstructured Gaussian complements to obtain structured Gaussian complement samples corresponding to the target object, wherein the structured Gaussian complement samples are represented in a three-dimensional mesh space; sampling the three-dimensional mesh space of the structured Gaussian complement samples to obtain three-dimensional local blocks; removing the regions corresponding to the three-dimensional local blocks from the three-dimensional mesh space of the target object sample to obtain an input sample; inputting the input sample into the diffusion model to obtain output Gaussian complements generated by the diffusion model based on the input sample for the input sample; the training objective includes minimizing the difference between the output Gaussian complements for the input sample and the structured Gaussian complement samples.

[0006] According to one achievable method in an embodiment of this application, the step of structuring the unstructured Gaussian elements to obtain the structured Gaussian sample corresponding to the target object includes: establishing a transfer matrix describing the distribution of unstructured Gaussian elements and the distribution of structured Gaussian elements; solving the transfer matrix using an optimal transfer algorithm to obtain an optimal transfer matrix; and generating the structured Gaussian sample corresponding to the target object from the unstructured Gaussian elements based on the optimal transfer matrix.

[0007] According to one achievable method in an embodiment of this application, sampling the three-dimensional mesh space of the structured Gaussian primitive sample to obtain a three-dimensional local block includes: sampling the three-dimensional mesh primitives of the structured Gaussian sample according to a preset viewpoint to obtain a three-dimensional local block; or, performing edge sampling on the three-dimensional mesh primitives of the structured Gaussian sample to obtain a three-dimensional local block.

[0008] According to one achievable embodiment of this application, the method further includes: the training samples further include text description samples corresponding to the target object; the step of inputting the input samples into the diffusion model to obtain the output Gaussian units generated by the diffusion model based on the input samples includes: encoding the text description samples to obtain a high-dimensional vector representation corresponding to the text description; fusing the high-dimensional vector representation and the input samples to obtain a fused vector; inputting the fused vector into the diffusion model, and generating the output Gaussian units by the diffusion model based on the fused vector.

[0009] According to one achievable method in an embodiment of this application, sampling the three-dimensional mesh space of the structured Gaussian meta sample to obtain a three-dimensional local block includes: determining the sampling size corresponding to the target object based on the text description sample; and sampling the three-dimensional mesh space of the structured Gaussian meta sample based on the sampling size to obtain a three-dimensional local block.

[0010] According to one achievable method in an embodiment of this application, determining the sampling size corresponding to the target object based on the text description sample includes: extracting image features from the image sample to obtain image features; encoding the text description sample using a natural language model to obtain text features; and generating the sampling size based on the image features and the text features using a convolutional neural network.

[0011] According to the second aspect, a 3D reconstruction method based on diffusion model for Gaussian element completion is provided. The method includes: acquiring a target object image containing a target object, wherein the target object image has a viewpoint missing relative to the full viewpoint representation of the target object; obtaining unstructured Gaussian elements corresponding to the target object based on the target object image; structuring the unstructured Gaussian elements to obtain structured Gaussian elements corresponding to the target object; completing the structured Gaussian elements using the diffusion model trained as described in the first aspect above to obtain complete structured Gaussian elements corresponding to the target object; and performing 3D reconstruction using the complete structured Gaussian elements to obtain a 3D reconstruction model of the target object.

[0012] According to a third aspect, a training apparatus for a diffusion model for Gaussian primitive completion is provided, the apparatus comprising: a training data acquisition unit configured to acquire training data including multiple training samples, wherein the training samples are image samples including a target object, and the image samples include a full-view image representation of the target object sample; a structured Gaussian primitive acquisition unit configured to obtain unstructured Gaussian primitives corresponding to the target object using the image samples, and to perform Gaussian primitive structuring on the unstructured Gaussian primitives to obtain structured Gaussian primitive samples corresponding to the target object, wherein the structured Gaussian primitive samples are represented in a three-dimensional mesh space; an input sample acquisition unit configured to sample the three-dimensional mesh space of the structured Gaussian primitive samples to obtain three-dimensional local blocks, and to remove the regions corresponding to the three-dimensional local blocks from the three-dimensional mesh space of the target object sample to obtain input samples; and a model training unit configured to input the input samples into the diffusion model to obtain output Gaussian primitives generated by the diffusion model based on the input samples for the input samples; wherein the training objective includes minimizing the difference between the output Gaussian primitives for the input samples and the structured Gaussian primitive samples.

[0013] According to a fourth aspect, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps of the method described in any one of the first or second aspects above.

[0014] According to a fifth aspect, an electronic device is provided, comprising: one or more processors; and a memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method described in any one of the first or second aspects.

[0015] According to the specific embodiments provided in this application, the following technical effects are disclosed:

[0016] 1) This application proposes a complete training method that obtains structured Gaussian primitives from image samples, and obtains input samples by sampling and removing these structured Gaussian primitives. A diffusion model then fills in the missing local regions of the 3D image samples, thereby generating a complete Gaussian primitive representation. This method ensures that the generated Gaussian primitives can better fit the actual 3D shape during training, achieving higher quality completion.

[0017] 2) This application transforms unstructured Gaussian units into structured Gaussian units using an optimal transfer algorithm, which helps generate more regular 3D representations. This allows the model to better utilize 3D structural information during the completion process, resulting in higher quality completion results.

[0018] 3) By sampling the 3D mesh space of the structured Gaussian elements in a specific way (such as preset viewpoint sampling or edge sampling), appropriate 3D local blocks can be generated, allowing the diffusion model to focus on completing local features. This local sampling strategy improves the model's ability to understand local details under different viewpoints, thereby enhancing the model's completion accuracy in different regions and optimizing the local detail representation of 3D completion.

[0019] 4) This application adds textual descriptions to the training data, so that the training data includes not only visual information of the target object but also semantic information. The fusion of the textual description encoding vector with the input samples helps the diffusion model learn more details and features about the target object from multimodal data, thereby generating complete Gaussian units that better meet semantic requirements.

[0020] 5) This application determines the sampling size based on the text description, enabling the model to flexibly adjust the sampling scale and region size according to the requirements of different text descriptions. This dynamic sampling method can improve the model's adaptability to different objects and different semantic information, enhance the model's ability to capture detailed features, and make the completion effect more consistent with the size and shape characteristics of actual objects.

[0021] 6) This application generates a suitable sampling size by combining image features and text features, enabling the sampling region to more accurately reflect the object features. By using a convolutional neural network to jointly encode image and text features, the model can automatically generate the optimal sampling region size based on the specific description of the target object. This adaptive sampling further improves the accuracy and rationality of the completion, and enhances the model's generalization ability.

[0022] 7) This application performs 3D reconstruction based on a trained diffusion model, and can complete the structured Gaussian units even when the input viewpoint is missing, thereby generating a complete 3D model. This method solves the reconstruction accuracy problem caused by incomplete 3D viewpoints, and effectively improves the integrity and visual consistency of 3D reconstruction through Gaussian unit completion, making the final generated 3D model closer to the real object.

[0023] Of course, any product implementing this application does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 A flowchart illustrating the training method for the diffusion model used for Gaussian primitive completion provided in this application embodiment;

[0026] Figure 2 A schematic diagram illustrating the process of training a diffusion model for Gaussian primitive completion provided in an embodiment of this application;

[0027] Figure 3 A structural block diagram of a training device for a diffusion model for Gaussian element completion provided in an embodiment of this application;

[0028] Figure 4 A structural block diagram of a three-dimensional reconstruction device for Gaussian element completion based on a diffusion model, provided in an embodiment of this application;

[0029] Figure 5 A schematic block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0030] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0031] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” as used in the embodiments of this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0032] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0033] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0034] Several Gaussian incomplete models already exist, such as those based on convolutional neural networks (CNNs) and variational autoencoders (VAEs). However, these models are not adept at capturing global structural relationships, especially when there are significant viewpoint gaps, resulting in poor incomplete quality. The advantages of diffusion models in image generation and incomplete tasks are gradually being discovered, particularly their superior performance in generating high-quality details and modeling complex structures. Currently, there is no precedent for training a diffusion model for Gaussian incomplete.

[0035] In view of this, this application provides a new approach. Figure 1 A flowchart illustrating a training method for a diffusion model for Gaussian meta-completion provided in this application is shown below. Figure 1 As shown, the method may include the following steps:

[0036] Step 101: Obtain training data including multiple training samples, wherein the training samples are image samples including target objects, and the image samples include full-view image representations of the target object samples.

[0037] Step 102: Obtain the unstructured Gaussian elements corresponding to the target object using the image samples, and perform Gaussian structuring on the unstructured Gaussian elements to obtain the structured Gaussian element samples corresponding to the target object. The structured Gaussian element samples are represented in a three-dimensional mesh space.

[0038] Step 103: Sample the three-dimensional mesh space of the structured Gaussian meta sample to obtain a three-dimensional local block, and remove the region corresponding to the three-dimensional local block from the three-dimensional mesh space of the target object sample to obtain the input sample.

[0039] Step 104: Input the input sample into the diffusion model to obtain the complete Gaussian primitives generated by the diffusion model based on the input sample; the training objective includes minimizing the difference between the complete Gaussian primitives for the input sample and the corresponding complete Gaussian primitive sample.

[0040] As can be seen from the above process, this application proposes a complete training method. Structured Gaussian primitives are obtained from image samples, and input samples are obtained by sampling and removing these structured Gaussian primitives. A diffusion model then fills in the missing local regions of the 3D image samples, thereby generating a complete Gaussian primitive representation. This method ensures that the generated Gaussian primitives can better fit the actual 3D shape during training, achieving higher quality completion.

[0041] The following describes in detail each step of the above process and the effects that can be further produced, with reference to the embodiments. First, step 101, namely "acquiring training data including multiple training samples, wherein the training samples are image samples including target objects, and the image samples include a full-view image representation of the target object samples", will be described in detail with reference to the embodiments.

[0042] Figure 2 This is a schematic diagram illustrating the training method for the diffusion model used for Gaussian element completion provided in this application embodiment. This application acquires multiple target image samples containing multiple viewpoints, providing multi-view information through different imaging angles. The image samples acquired by this application include full-view image representations of the target object samples.

[0043] Target image samples can be obtained through various means, such as public data platforms or by taking pictures from different imaging angles using imaging devices. After obtaining images containing the target object from the data platform or imaging device, preprocessing operations such as cropping and geometric correction can be performed on the images to obtain higher quality target images.

[0044] The following describes in detail step 102, namely, "obtaining unstructured Gaussian elements corresponding to the target object using image samples, structuring the unstructured Gaussian elements to obtain structured Gaussian elements corresponding to the target object, wherein the structured Gaussian elements are represented in a three-dimensional mesh space," with reference to the embodiments.

[0045] Unstructured Gaussian elements are a probabilistic representation of the distribution of feature points on an object. They are typically composed of parameters such as the mean, variance, and weights of the distribution and are used to describe the features of local regions on the object's surface. This representation constructs a point cloud or a continuous distribution model in three-dimensional space, which can reflect the object's shape and structure to a certain extent.

[0046] Based on the target image samples, the unstructured Gaussian representation of the target object can be obtained in a variety of ways. One feasible approach is to extract point clouds from multiple target images and obtain a 3D point cloud representation of the target object through multi-view image matching techniques, such as Structure from Motion (SFM) or Multi-View Stereo (MVS). Then, Gaussian fitting is performed on the point cloud to generate unstructured Gaussian units.

[0047] As another possible approach, the following steps are included:

[0048] First, image segmentation can be performed on the target image to obtain the target object region. Besides the target object, the target image may also contain other objects or background patterns. Image segmentation is needed to extract the target object region from the target image; the segmented region is the target object region. Image segmentation can be achieved using algorithms such as edge detection and semantic segmentation.

[0049] Next, feature points are extracted from the target object region to obtain multiple feature points. Feature point extraction aims to identify key points from the two-dimensional image of the target object that can uniquely describe the object's features. These feature points typically possess the following properties: stability: they remain consistent under different viewpoints, lighting, and noise conditions; discriminability: they have good discriminability and can be used to match the same feature in different images; repeatability: they can appear repeatedly in multiple images, providing necessary data support for 3D reconstruction.

[0050] Feature point extraction can be achieved using various algorithms. For example, the Harris corner detection method can be used to find corners with strong responses by calculating the autocorrelation matrix of the image. Alternatively, scale-invariant feature transformation algorithms can be used to extract feature points that are invariant to scale and rotation by detecting extreme points in multi-scale space.

[0051] Clustering algorithms are used to cluster feature points, and the clustering results are fitted with a Gaussian distribution to obtain unstructured Gaussian units. Specifically, a set of feature points extracted from the image is selected as the input to the clustering algorithm; the selected clustering algorithm is used to cluster the feature points. The clustering results will form multiple clusters, each containing feature points with similar characteristics; the effectiveness of the clustering can be evaluated using metrics such as the silhouette coefficient and the Davies-Bouldin index to assess the quality of the clustering.

[0052] After clustering, a Gaussian distribution is fitted based on the clustering results. Specifically, for each cluster, the set of feature points is extracted; for each cluster, its mean, covariance matrix, and other statistical parameters are calculated to form the basis of the unstructured Gaussian distribution. These parameters describe the spatial distribution of feature points. Using the obtained parameters, an unstructured Gaussian distribution model is constructed. Each unstructured Gaussian unit consists of multiple Gaussian distributions, defined by its mean and variance, describing the distribution of feature points in that region. Furthermore, for multiple clustering results, it may be necessary to merge multiple unstructured Gaussian units to form a complete unstructured Gaussian unit representation. For example, similar Gaussian units can be merged into a larger Gaussian unit, depending on the requirements.

[0053] This application can also standardize unstructured Gaussian elements. Standardization can balance the differences between different unstructured Gaussian elements, avoid the influence of numerical deviations on the solution process of the transfer matrix, and thus improve the accuracy of the optimal transfer matrix solution.

[0054] After obtaining the unstructured Gaussian elements, the unstructured Gaussian elements are structured to obtain the structured Gaussian element samples corresponding to the target object. The structured Gaussian element samples are represented in a three-dimensional mesh space.

[0055] The structuring of Gaussian elements can be achieved in several ways. For example, clustering algorithms can be used to group unstructured Gaussian elements, defining the primitives within each group in a three-dimensional mesh space; or a graph structure containing unstructured Gaussian elements can be constructed, and a graph neural network can be used to learn the spatial relationships between the Gaussian elements. The model can then generate structured representations based on proximity relationships.

[0056] As an feasible approach, a transfer matrix describing the distribution of unstructured Gaussian elements and the distribution of structured Gaussian elements can be established. The optimal transfer matrix can be obtained by solving the transfer matrix using the optimal transfer algorithm. Based on the optimal transfer matrix, structured Gaussian elements corresponding to the target object can be generated from the unstructured Gaussian elements.

[0057] A transfer matrix is ​​a mathematical tool used to describe the relationship between two distributions. In this application, the transfer matrix is ​​used to represent the correspondence between unstructured Gaussian distributions and structured Gaussian distributions. Specifically, each element of the transfer matrix can be viewed as a "transfer amount" from one primitive distribution (unstructured) to another primitive distribution (structured).

[0058] The steps for constructing a transfer matrix include: First, matching feature points in unstructured Gaussian elements and structured Gaussian elements. A correspondence can be established by calculating the distance or similarity metric between them; based on the feature point matching, the elements of the transfer matrix are assigned values. Generally, each element T of the matrix... ij The "transfer amount" representing the distance from the i-th unstructured Gaussian unit to the j-th structured Gaussian unit can be calculated using a distance metric (e.g., Euclidean distance) or a similarity function. To ensure the validity of the transfer matrix, it may be necessary to normalize it so that the sum of each row or column equals 1. This helps to treat the transfer matrix as a probability distribution in subsequent processing.

[0059] The optimal transmission algorithm is a mathematical tool used to find the optimal solution to transform one distribution into another. Its goal is to minimize transmission cost. The optimal transmission matrix can be obtained by solving for the transmission matrix using the optimal transmission algorithm, which can be achieved through the following steps:

[0060] First, we define a transmission cost function, which is a function based on distance or similarity metrics, representing the "transportation cost" from unstructured primitives to structured primitives.

[0061] Next, the transfer matrix is ​​solved using optimization algorithms (such as Sinkhorn distance, linear programming, etc.) to find the optimal transfer matrix. Each element in the optimal transfer matrix represents the optimal transfer amount from each unstructured Gaussian cell to a structured Gaussian cell. An iterative algorithm is used to solve for the optimal transfer matrix until convergence is achieved. The iterative process adjusts the elements of the transfer matrix to reduce the total transfer cost.

[0062] Preferably, a regularization term can be added to the transmission cost function. The regularization term can introduce prior knowledge or other information (such as marginal distribution, smoothness, sparsity, etc.), thereby giving the optimization problem a clearer direction.

[0063] After obtaining the optimal transfer matrix, the structured Gaussian representation of the target object is obtained based on the optimal transfer matrix. This process can be achieved in several ways. For example, the unstructured source Gaussian units can be directly combined using a weighted method to obtain the structured Gaussian representation of the target object; or the unstructured Gaussian units can be weighted and combined into structured Gaussian units using the weights in the optimal transfer matrix; or the parameters of the structured Gaussian units can be iteratively updated using the expectation-maximization algorithm to make the distribution of the structured Gaussian units match the transfer rules guided by the optimal transfer matrix as closely as possible.

[0064] Structured Gaussian elements are represented by structured meshes, which divide three-dimensional space into regular grid cells. Each grid cell can contain one or more Gaussian elements used to describe the spatial characteristics of an object within that cell. The denser the mesh (higher resolution), the more detailed information each cell contains. Therefore, the resolution of the structured Gaussian element representation can be adjusted according to a scaling factor, which includes the size of the structured mesh.

[0065] Resolution here typically refers to the fineness or sampling density of Gaussian pixels distributed in 3D space. The resolution of Gaussian pixels can be adjusted using different representation methods to maintain the aspect ratio of an object, and different levels of detail can be achieved through specific scaling factors. Resolution is adjusted by increasing or decreasing the value of the scaling factor. For example, low resolution can use fewer voxel meshes to represent the overall structure, while high resolution uses a denser voxel mesh to capture more detail.

[0066] The following describes in detail step 103, namely, "sampling the three-dimensional mesh space of the structured Gaussian sample to obtain a three-dimensional local block, removing the region corresponding to the three-dimensional local block from the three-dimensional mesh space of the target object sample to obtain the input sample," with reference to the embodiments.

[0067] This application trains a diffusion model for Gaussian primitive completion. The input to the model should be Gaussian primitives with viewpoint missingness relative to the full viewpoint representation of the target object. Therefore, this application obtains structured Gaussian primitives with viewpoint missingness by removing 3D local blocks in 3D mesh space.

[0068] As a feasible approach, 3D mesh primitives of structured Gaussian samples can be sampled according to a preset viewpoint to obtain 3D local blocks. Before sampling, one or more preset viewpoints need to be determined based on task requirements. The preset viewpoint can be a fixed angle (such as front, side, or top) or a dynamically adjusted angle (such as multiple angles rotated around the target object). The choice of viewpoint can be determined based on the characteristics of the target object. For example, for a tetrahedral object, sampling can be performed from four symmetrical directions; while for an object with a specific front or main viewpoint, sampling from the front viewpoint is usually preferred.

[0069] As another feasible approach, edge sampling can be performed on the 3D mesh primitives of the structured Gaussian sample to obtain 3D local blocks. In a 3D mesh, edge regions can be defined as mesh primitives located near the object's surface or outer contour. Edge regions can be determined in several ways: the distance field method can be used to calculate the distance from each mesh cell to the object's surface; regions with smaller distances are considered edges; the gradient rate of change of the mesh primitives can be calculated, with larger gradients typically corresponding to edges or surface regions of the object; edge regions can also be identified by calculating local curvature, with areas of greater curvature usually representing regions with significant contour or shape changes.

[0070] This application can also use the text description corresponding to the target object to train the model, determine the sampling size corresponding to the target object based on the text description sample, and sample the three-dimensional mesh space of the structured Gaussian meta sample based on the sampling size to obtain a three-dimensional local block.

[0071] The text description of the target object can be any text that describes the characteristics of the target object, including but not limited to: the text description can be the category information of the target object, such as "cat", "building", "plant", etc., which can help the diffusion model generate Gaussian elements that conform to specific category characteristics; it can be a description of the shape, size, position or spatial structure of the target object, such as "circle", "rectangle", "distributed in the left area", etc., which can affect the distribution parameters of the Gaussian elements, making the completion result more consistent with the geometric features of the object; it can be a description of the visual data, color or texture of the target object, such as "blue and striped", which can help the diffusion model generate more distinctive features when generating the completed Gaussian elements; for target objects with motion characteristics, the text description can be a description of its behavior or posture, such as "moving forward", "tilting", etc., which can help complete the distribution of Gaussian elements that reflects the dynamics.

[0072] Text descriptions can be obtained in various ways, including manual annotation and automated extraction. Manual annotation involves humans providing precise object descriptions, suitable for more detailed scene requirements. Automated extraction utilizes image tagging tools, feature extraction models, or natural language processing models to automatically generate descriptions from data. Furthermore, knowledge bases or metadata tags can be used to find detailed descriptions of the target object, or users can provide descriptions based on specific needs to meet personalized completion requirements. These description methods provide rich semantic information to the generative model, enhancing the accuracy and consistency of the completion results.

[0073] Determining the sampling size of a target object based on textual description samples can be achieved in several ways. For example, it can be done by constructing or utilizing existing knowledge graphs, labeling the standard size information corresponding to different descriptions, matching the descriptions with concepts or entities in the knowledge graph, and extracting the corresponding size information.

[0074] As an feasible approach, image features can be obtained by extracting image features from image samples; text features can be obtained by encoding the text description samples using a natural language model; and the sampling size can be generated using a convolutional neural network based on the image features and the text features.

[0075] Specifically, image feature extraction can be achieved through image encoders. Commonly used image encoder networks include ResNet, VGG, and EfficientNet, which can transform input two-dimensional image data into high-dimensional feature vectors. The extracted image feature vectors can describe the basic shape, color, contour, complexity, and other attributes of the target object. Through the high-dimensional representation of features, the content of the image sample can be effectively represented. Text feature extraction can be achieved through natural language models. Text description samples are processed by a text encoder to extract features. The text description is processed into a vectorized feature representation that can capture the semantic information in the text, including descriptions of object size, shape, detail requirements, etc. For example, information such as "small" or "wide" in the description will be transformed into part of the feature vector to represent size requirements. Image and text feature vectors are concatenated or fused through weighted averaging, self-attention mechanisms, etc., to form a multimodal feature vector. The fused multimodal feature vector is input into a convolutional neural network or a fully connected regression network to output the specific numerical value of the sampling size. The parameters of each layer in the network are trained and optimized to learn to generate a suitable sampling size based on the feature fusion information.

[0076] The following describes in detail step 104, namely, "Inputting the input sample into the diffusion model to obtain the complete Gaussian units generated by the diffusion model based on the input sample; the training objective includes minimizing the difference between the output Gaussian units for the input sample and the structured Gaussian unit samples," with reference to an embodiment.

[0077] This application utilizes a pre-trained diffusion model to complete structured Gaussian primitives. The diffusion model is a generative model that progressively transforms data into noise and then reconstructs the data from the noise. In the scenario of Gaussian primitive structured completion, the diffusion model can complete missing or incomplete parts from existing structured Gaussian primitives by simulating the progressive generation process of data.

[0078] During training, the diffusion model generates a complete Gaussian primitive (GMT) based on the input sample. The model is trained by minimizing the difference between this complete GMT and the structured GMT. The structured GMT is the one generated from the image sample in step 102. Since the image sample includes a full-view image representation of the target object sample, the structured GMT is a complete GMT representation. Comparing it with the output GMT generated by the diffusion model determines the inference accuracy of the diffusion model.

[0079] Minimizing the difference typically involves using a loss function (such as mean squared error, MSE) to measure the difference between the output Gaussian units and the structured Gaussian units. In this way, the model is guided to generate outputs that are more similar to the structured Gaussian units.

[0080] This application, when using a diffusion model for completion, can also utilize the textual description of the target object to guide the inference of the diffusion model. The textual description of the target object can be text that describes the characteristics of the target object, including but not limited to: the textual description can be category information of the target object, such as "cat," "building," "plant," etc., which helps the diffusion model generate Gaussian elements that conform to specific category characteristics; it can be a description of the shape, size, position, or spatial structure of the target object, such as "circle," "rectangle," "distributed in the left area," etc., which can affect the distribution parameters of the Gaussian elements, making the completion result more consistent with the geometric characteristics of the object; it can be a description of the visual data, color, or texture of the target object, such as "blue with stripes," which helps the diffusion model generate more distinctive features when generating the Gaussian elements for completion; for target objects with motion characteristics, the textual description can be a description of its behavior or posture, such as "moving forward," "tilting," etc., which helps complete the distribution of Gaussian elements that reflects dynamics.

[0081] After obtaining the text description, the text description and the structured Gaussian units can be input together into the pre-trained diffusion model. The diffusion model will then complete the structured Gaussian units based on the text description, thus obtaining the complete structured Gaussian units.

[0082] As one possible implementation, this application can further encode the text description to convert it into a high-dimensional vector representation; fuse the high-dimensional vector and the structured Gaussian units to obtain a fused vector; and generate the output Gaussian units from the diffusion model based on the fused vector.

[0083] Further encoding of the text description can be achieved using traditional methods such as sentence encoding and multimodal encoding; alternatively, a language model can be used. This involves fusing the high-dimensional vector representation of the text description with structured Gaussian primitives. This step can be implemented in several ways, such as weighted summation, convolution, and attention mechanisms. The goal of this fusion is to combine the semantic information of the text with the information from the Gaussian distribution, forming a fused vector that incorporates features from both.

[0084] As another feasible approach, a high-dimensional vector representation of the text description can be generated using a large language model. Specifically, the Low-Rank Adaptation of Large Language Models (LoRA) technique can be utilized by loading pre-trained low-rank matrix parameters to enable the large language model to adapt to specific tasks. Specifically, the pre-trained low-rank matrix parameters are loaded and combined with the language model's parameter matrix to obtain a combined matrix; this combined matrix is ​​then used to generate the high-dimensional vector representation of the text description. The combination of the low-rank matrix parameters and the language model's parameter matrix can involve adding, weighted summing, convolution, etc., to superimpose the low-rank matrix onto the original model's parameters, resulting in higher accuracy of the generated results.

[0085] As another feasible approach, this application, when generating complete structured Gaussian elements from the fusion vector using a diffusion model, can also guide the Gaussian element completion process through a ControlNet. ControlNet serves as an effective control mechanism, ensuring the model follows specific structural information during generation. Specifically, control information is acquired, and guidance information is generated using the control network based on this control information; the complete structured Gaussian elements are then generated by the diffusion model based on the guidance information and the fusion vector.

[0086] In this system, control information serves as the input to the control network, constraining the generation process of the diffusion model. This control information can be a structural map of the target object, such as an edge map, depth map, or attitude map, or a Gaussian element distribution map describing the density or positional distribution of the target object in space. Based on the control information, the control network outputs guiding information, which constrains the generation process of the diffusion model, ensuring that the generated content conforms to the input conditions.

[0087] This application also provides a 3D reconstruction method based on a diffusion model for Gaussian primitive completion. The method includes: acquiring an image of a target object containing a target object, wherein the target object image has a missing viewpoint relative to the full-view representation of the target object; obtaining unstructured Gaussian primitives corresponding to the target object based on the target object image; structuring the unstructured Gaussian primitives to obtain structured Gaussian primitives corresponding to the target object; completing the structured Gaussian primitives using the diffusion model trained in the above embodiments to obtain complete structured Gaussian primitives corresponding to the target object; and performing 3D reconstruction using the complete structured Gaussian primitives to obtain a 3D reconstruction model of the target object. By utilizing the diffusion model trained in the above embodiments, the quality of Gaussian primitive completion is improved, further enhancing the accuracy of 3D reconstruction.

[0088] This method performs 3D reconstruction on a target image, which is an image containing the target object. This application acquires multiple target images containing multiple viewpoints, providing multi-view information through different imaging angles. Notably, this application can perform 3D reconstruction even when the target image has missing viewpoints. Therefore, the target images acquired by this application with multiple viewpoints have missing viewpoints compared to the full-view target image.

[0089] Target images can be acquired through various means, such as public data platforms or by taking pictures from different imaging angles using imaging devices. After obtaining an image containing the target object from a data platform or imaging device, preprocessing operations such as cropping and geometric correction can be performed on the image to obtain a higher quality target image.

[0090] For steps such as Gaussian structuring, the same method as in the model training method above can be used for structuring, or other methods can be used; no restrictions are placed here.

[0091] When using the complete structured Gaussian primitives (GRPs) for 3D reconstruction to obtain a 3D reconstructed model of the target object, the following steps can be taken: First, the complete GRPs are preprocessed to extract features containing 3D spatial location and distribution information. Then, based on the GRPs, they are converted into 3D point cloud data of the target object using a 3D reconstruction algorithm. During this process, a Gaussian distribution-based density estimation algorithm can be used to complete the point cloud and adjust its density, making the generated point cloud more consistent with the morphological characteristics of the target object. Subsequently, a surface reconstruction algorithm is used to convert the point cloud data into a continuous 3D mesh model. For example, methods such as Poisson surface reconstruction, Marching Cubes, or Delaunay triangulation can be used to generate a 3D model with a complete surface structure.

[0092] Preferably, the generated 3D model can be optimized, for example, by removing redundant noise or smoothing, to improve the model's accuracy and visual effect, thereby obtaining a 3D reconstruction model of the target object that conforms to the structured Gaussian elements.

[0093] As an feasible approach, this application utilizes rasterization technology to optimize the 3D reconstruction model. Specifically: 3D reconstruction is performed using complete structured Gaussian units to obtain a pre-reconstructed model of the target object; the pre-reconstructed model is rasterized to obtain 2D images from multiple angles, wherein the 2D images include feature information of their corresponding viewpoints; the correspondence between different viewpoints is determined based on the feature information; and the 3D reconstruction model of the target object is obtained based on the correspondence.

[0094] Rasterization technology is widely used in view synthesis and multi-view geometric reconstruction. This technology projects a 3D object from different viewpoints to generate 2D view images at different angles. These view images can represent the depth information and geometric structure of a scene. By employing rasterization, the target 3D model is converted into multiple 2D projected images according to set camera viewpoints and intrinsic / extrinsic parameters, effectively capturing the features, textures, and details of the object's surface. This application first renders the pre-reconstructed model into 2D images from multiple set angles using rasterization, allowing the feature information from different viewpoints to be fully displayed in the 2D images. Next, by analyzing and extracting features from each of the multi-view images, image registration or stereo vision methods are used to match features between adjacent views to determine the correspondence between different viewpoints. Increasing the number of viewpoints can also improve the accuracy and stability of the reconstruction. Finally, based on the optimized correspondence, a 3D reconstructed model of the target object is obtained. Both the generation of the pre-reconstructed model and the generation of the 3D reconstructed model can be obtained using the 3D reconstruction methods mentioned above, without specific limitations.

[0095] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0096] According to another embodiment, a training apparatus for a diffusion model for Gaussian element completion is provided. Figure 3 A schematic block diagram of a training apparatus for a diffusion model for Gaussian complementation according to one embodiment is shown, as follows: Figure 3 As shown, the device 300 includes:

[0097] The training data acquisition unit 301 is configured to acquire training data including multiple training samples, wherein the training samples are image samples including target objects, and the image samples include a full-view image representation of the target object samples;

[0098] The structured Gaussian element acquisition unit 302 is configured to obtain unstructured Gaussian elements corresponding to the target object using the image samples, perform Gaussian element structuring on the unstructured Gaussian elements, and obtain structured Gaussian element samples corresponding to the target object, wherein the structured Gaussian element samples are represented in a three-dimensional mesh space.

[0099] The input sample acquisition unit 303 is configured to sample the three-dimensional mesh space of the structured Gaussian meta sample to obtain a three-dimensional local block, and remove the region corresponding to the three-dimensional local block from the three-dimensional mesh space of the target object sample to obtain the input sample;

[0100] The model training unit 304 is configured to input the input sample into the diffusion model to obtain the output Gaussian units generated by the diffusion model based on the input sample for the input sample; the training objective includes minimizing the difference between the output Gaussian units for the input sample and the structured Gaussian unit sample.

[0101] As one possible implementation method, the structured Gaussian element acquisition unit 302 can be configured to: establish a transfer matrix describing the distribution of unstructured Gaussian elements and the distribution of structured Gaussian elements; solve the transfer matrix using an optimal transfer algorithm to obtain an optimal transfer matrix; and generate a structured Gaussian element sample corresponding to the target object from the unstructured Gaussian elements based on the optimal transfer matrix.

[0102] As one possible implementation method, the input sample acquisition unit 303 can be configured to: sample the three-dimensional mesh primitives of the structured Gaussian sample according to a preset viewpoint to obtain a three-dimensional local block; or, perform edge sampling on the three-dimensional mesh primitives of the structured Gaussian sample to obtain a three-dimensional local block.

[0103] As one possible approach, the training samples also include text description samples corresponding to the target object; when the model training unit 304 inputs the input samples into the diffusion model and obtains the output Gaussian units generated by the diffusion model based on the input samples, it can be configured to: encode the text description samples to obtain a high-dimensional vector representation corresponding to the text description; fuse the high-dimensional vector representation and the input samples to obtain a fused vector; input the fused vector into the diffusion model, and generate the output Gaussian units by the diffusion model based on the fused vector.

[0104] As one possible implementation method, the input sample acquisition unit 303 can be configured to: determine the sampling size corresponding to the target object based on the text description sample; and sample the three-dimensional mesh space of the structured Gaussian meta sample based on the sampling size to obtain the three-dimensional local block.

[0105] As one possible implementation method, the input sample acquisition unit 303 can be configured to: extract image features from the image sample to obtain image features; encode the text description sample using a natural language model to obtain text features; and generate the sampling size using a convolutional neural network based on the image features and text features.

[0106] According to another embodiment, a three-dimensional reconstruction apparatus for Gaussian element completion based on a diffusion model is provided. Figure 4 A schematic block diagram of a 3D reconstruction apparatus for Gaussian element completion based on a diffusion model, according to one embodiment, is shown. Figure 4 As shown, the device 400 includes:

[0107] The target image acquisition unit 401 is configured to acquire a target object image containing the target object, wherein the target object image has a missing viewpoint relative to the full viewpoint representation of the target object.

[0108] The structured Gaussian unit 402 is configured to obtain unstructured Gaussian elements corresponding to the target object based on the target object image; and to perform Gaussian structuring on the unstructured Gaussian elements to obtain the structured Gaussian elements corresponding to the target object.

[0109] The Gaussian complement unit 403 is configured to complete the structured Gaussian units using a diffusion model trained by a training device for a diffusion model used for Gaussian complement, thereby obtaining the complete structured Gaussian units corresponding to the target object.

[0110] The 3D reconstruction unit 404 is configured to perform 3D reconstruction using complete structured Gaussian units to obtain a 3D reconstruction model of the target object.

[0111] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0112] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0113] In addition, embodiments of this application also provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method described in any of the foregoing method embodiments.

[0114] And an electronic device, comprising:

[0115] One or more processors; and a memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method described in any of the foregoing method embodiments.

[0116] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in any of the foregoing method embodiments.

[0117] in, Figure 5 An exemplary architecture of an electronic device is shown, which may include a processor 510, a video display adapter 511, a disk drive 512, an input / output interface 513, a network interface 514, and a memory 520. The processor 510, video display adapter 511, disk drive 512, input / output interface 513, network interface 514, and memory 520 can communicate with each other via a communication bus 530.

[0118] The processor 510 can be implemented using a general-purpose CPU, microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits to execute relevant programs in order to implement the technical solution provided in this application.

[0119] The memory 520 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 520 can store the operating system 521 for controlling the operation of the electronic device 500, and the basic input / output system (BIOS) 522 for controlling the low-level operations of the electronic device 500. Additionally, it can store a web browser 523, a data storage management system 524, and a training device 525 for the diffusion model of Gaussian primitive completion, etc. The aforementioned training device 525 for the diffusion model of Gaussian primitive completion can be the application program that specifically implements the aforementioned steps in this embodiment. In summary, when implementing the technical solution provided in this application through software or firmware, the relevant program code is stored in the memory 520 and is called and executed by the processor 510.

[0120] Input / output interface 513 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touch screens, microphones, various sensors, etc., and output devices may include displays, speakers, vibrators, indicator lights, etc.

[0121] Network interface 514 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0122] Bus 530 includes a pathway for transmitting information between various components of the device, such as processor 510, video display adapter 511, disk drive 512, input / output interface 513, network interface 514, and memory 520.

[0123] It should be noted that although the above-described device only shows the processor X10, video display adapter 511, disk drive 512, input / output interface 513, network interface 514, memory 520, bus 530, etc., in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the solution of this application, and does not necessarily include all the components shown in the figures.

[0124] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer program product. This computer program product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0125] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A training method of a diffusion model for Gaussian cell completion, characterized in that, The method comprises: obtaining training data comprising a plurality of training samples, the training samples being image samples comprising a target object, the image samples comprising a full-view image representation of the target object sample; obtaining an unstructured Gaussian cell corresponding to the target object using the image sample, structuring the unstructured Gaussian cell to obtain a structured Gaussian cell sample corresponding to the target object, the structured Gaussian cell sample being represented in a three-dimensional grid space; sampling the three-dimensional grid space of the structured Gaussian cell sample to obtain a three-dimensional local block, removing the region corresponding to the three-dimensional local block in the three-dimensional grid space of the target object sample to obtain an input sample; inputting the input sample into the diffusion model to obtain an output Gaussian cell generated by the diffusion model for the input sample, the training target comprising minimizing the difference between the output Gaussian cell for the input sample and the structured Gaussian cell sample.

2. The method of claim 1, wherein, The structured Gaussian cell is obtained by: establishing a transport matrix describing the distribution of the unstructured Gaussian cell and the structured Gaussian cell, solving the transport matrix using an optimal transport algorithm to obtain an optimal transport matrix, and generating the structured Gaussian cell sample corresponding to the target object from the unstructured Gaussian cell according to the optimal transport matrix.

3. The method of claim 1, wherein, The three-dimensional local block is obtained by: sampling the three-dimensional grid cell of the structured Gaussian cell sample according to a predetermined view angle to obtain a three-dimensional local block; or, edge sampling the three-dimensional grid cell of the structured Gaussian cell sample to obtain a three-dimensional local block.

4. The method of claim 1, wherein, The method further comprises: The training sample further comprises a textual description sample corresponding to the target object; The input sample is input into the diffusion model to obtain an output Gaussian cell generated by the diffusion model for the input sample, which comprises: encoding the textual description sample to obtain a high-dimensional vector representation corresponding to the textual description sample; fusing the high-dimensional vector representation and the input sample to obtain a fused vector; inputting the fused vector into the diffusion model to generate the output Gaussian cell from the diffusion model according to the fused vector.

5. The method of claim 4, wherein, The three-dimensional local block is obtained by: determining a sampling size corresponding to the target object according to the textual description sample; sampling the three-dimensional grid space of the structured Gaussian cell sample according to the sampling size to obtain a three-dimensional local block.

6. The method of claim 5, wherein the sampling size corresponding to the target object is determined according to the textual description sample, comprising: performing image feature extraction on the image sample to obtain image features; encoding the textual description sample using a natural language model to obtain text features; generate the sampling size according to the image feature and the text feature by using a convolutional neural network.

7. A three-dimensional reconstruction method for Gaussian cell completion based on a diffusion model, characterized in that, The method comprises: an image of a target object is obtained, the image of the target object having a view angle loss relative to a full view angle representation of the target object; an unstructured Gaussian cell corresponding to the target object is obtained according to the image of the target object; the unstructured Gaussian cell is structured to obtain a structured Gaussian cell corresponding to the target object; the structured Gaussian cell is completed by using a diffusion model obtained by the training method of any one of claims 1-6 to obtain a complete structured Gaussian cell corresponding to the target object; three-dimensional reconstruction is performed by using the complete structured Gaussian cell to obtain a three-dimensional reconstruction model of the target object.

8. A device for training a diffusion model for Gaussian cell completion, characterized in that, The device comprises: a training data acquisition unit configured to acquire training data comprising a plurality of training samples, the training samples being image samples comprising a target object, the image samples comprising a full view angle image representation of the target object sample; a structured Gaussian cell acquisition unit configured to obtain an unstructured Gaussian cell corresponding to the target object by using the image sample, structure the unstructured Gaussian cell to obtain a structured Gaussian cell sample corresponding to the target object, the structured Gaussian cell sample being represented by a three-dimensional grid space; an input sample acquisition unit configured to sample the three-dimensional grid space of the structured Gaussian cell sample to obtain a three-dimensional local block, remove the corresponding region of the three-dimensional local block in the three-dimensional grid space of the target object sample to obtain an input sample; a model training unit configured to input the input sample into the diffusion model to obtain an output Gaussian cell generated by the diffusion model for the input sample, the training target comprising minimizing the difference between the output Gaussian cell for the input sample and the structured Gaussian cell sample.

9. An electronic device, comprising: comprise: one or more processors; and a memory associated with the one or more processors, the memory being configured to store program instructions, the program instructions being configured to perform the steps of the method of any one of claims 1-7 when read and executed by the one or more processors.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the steps of the method of any one of claims 1-7.

Images