Model updating method, three-dimensional reconstruction method, equipment, medium and program product

Abstract

The invention provides a model updating method, a three-dimensional reconstruction method, equipment, a medium and a program product, and relates to the technical field of computers. The model updating method comprises the following steps: constructing a loss function based on a first probability density field, a directed distance and a confidence coefficient corresponding to the directed distance; wherein the first probability density field is generated based on a preset three-dimensional Gaussian model and is used for expressing a probability density field corresponding to a Gaussian primitive of the target scene; the directed distance is a directed distance parameter output by the target model to be updated and from the spatial point to the target scene, and the confidence coefficient is a confidence coefficient parameter output by the target model to be updated for the directed distance; and sending the loss function to the target model, so that the target model is updated based on the loss function. According to the method, the updating precision and updating efficiency of the target model are greatly improved.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, specifically to a model updating method, a three-dimensional reconstruction method, an apparatus, a medium, and a program product. Background Technology

[0002] In the field of 3D reconstruction, different 3D descriptions have their own advantages for the same target scene. Using 3DGS (3D Gaussian Splatting) for 3D reconstruction is the benchmark method for differentiable rendering in terms of both performance and results, and has become the absolute mainstream method. Using the SDF (Signed Distance Function) model for 3D reconstruction also has its inherent advantages. First, the SDF model can obtain clear surface information, thus reducing many semi-transparent floating objects. Second, the SDF model can easily extract meshes and is compatible with traditional rendering.

[0003] However, training an SDF model using a neural network is significantly inferior to 3DGS in both performance and effectiveness. In particular, in terms of performance, training a 3DGS model using a network currently only takes a few minutes, while training an SDF model often takes more than ten hours. Summary of the Invention

[0004] This disclosure presents a model update method, a three-dimensional reconstruction method, an apparatus, a medium, and a program product.

[0005] In a first aspect, embodiments of this disclosure propose a model update method, the method comprising: constructing a loss function based on a first probability density field, a directed distance, and a confidence level corresponding to the directed distance; wherein the first probability density field is generated based on a preset three-dimensional Gaussian model and is used to express the probability density field corresponding to the Gaussian units of the target scene; the directed distance is the directed distance parameter from a spatial point to the target scene output by the target model to be updated, and the confidence level is the confidence level parameter output by the target model to be updated for the directed distance; and the loss function is sent to the target model so that the target model is updated based on the loss function.

[0006] In some embodiments, a loss function is constructed based on a first probability density field, a directed distance, and a confidence level corresponding to the directed distance, including: constructing a second probability density field based on the directed distance and the confidence level; wherein the second probability density field is used to describe the probability distribution that the directed distance follows at a spatial point; and constructing a loss function based on the first probability density field and the second probability density field.

[0007] In some embodiments, constructing a loss function based on a first probability density field and a second probability density field includes: constructing a relationship function based on the relationship between the first probability density field and the second probability density field; obtaining an initial loss function based on the relationship function; and optimizing the initial loss function through a score matching method to obtain a final loss function.

[0008] In some embodiments, a loss function is constructed based on a first probability density field, directed distances, and confidence levels corresponding to those directed distances; this includes: constructing weight parameters corresponding to each sampled Gaussian element based on the first probability density field; each sampled Gaussian element is obtained by randomly sampling all Gaussian elements corresponding to the three-dimensional Gaussian model; constructing feature differences between the sampled spatial points and the sampled Gaussian elements based on the directed distances and corresponding confidence levels of multiple sampled spatial points, as well as the center position parameters of each sampled Gaussian element; multiple sampled spatial points are obtained by randomly sampling all spatial points corresponding to the target model; and constructing a loss function based on the expected values, weight parameters, and feature differences of the multiple sampled spatial points in the three-dimensional spatial distribution.

[0009] In some embodiments, a feature difference between a sampling spatial point and a sampling Gaussian element is constructed based on the directed distances and corresponding confidence levels of multiple sampling spatial points, as well as the center position parameters of each sampling Gaussian element. This includes: constructing a first feature corresponding to a sampling spatial point based on the first gradient of the directed distance and the second gradient of the confidence level, the directed distance, and the confidence level; constructing a second feature based on the offset of the center position parameters of multiple sampling Gaussian elements relative to the center position of the sampling spatial point; and constructing a vector distance between the first feature and the second feature based on the norm operation between the first feature and the second feature, wherein the vector distance is the feature difference.

[0010] In some embodiments, the first gradient is orthogonal to the second gradient.

[0011] In some embodiments, a loss function is constructed based on a first probability density field, a directed distance, and a confidence level corresponding to the directed distance, including: adjusting the confidence level corresponding to the directed distance according to the current iteration number and the total iteration number of the target model to obtain an adjusted confidence level corresponding to the directed distance; and constructing a loss function based on the first probability density field, the directed distance, and the adjusted confidence level corresponding to the directed distance.

[0012] In some embodiments, the confidence level corresponding to the directed distance is adjusted based on the current iteration number and the total iteration number of the target model to obtain the adjusted confidence level corresponding to the directed distance, including: determining a first ratio between the current iteration number and the preset total iteration number; and adjusting the confidence level corresponding to the directed distance to a second preset value if the first ratio is less than a first preset value to obtain the adjusted confidence level corresponding to the directed distance.

[0013] Secondly, this disclosure also provides a three-dimensional reconstruction method, the method comprising: inputting multiple three-dimensional coordinates of the spatial region to be reconstructed into a trained target model to obtain multiple directed distance values ​​output by the target model corresponding to the multiple three-dimensional coordinates; wherein the target model is obtained according to any one of the methods in the first aspect above; performing zero isosurface extraction on the directed distance values ​​to obtain a three-dimensional model of the spatial region to be reconstructed.

[0014] Thirdly, embodiments of this disclosure provide an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to implement the method described in either the first or second aspect.

[0015] Fourthly, embodiments of this disclosure provide a non-transitory computer-readable storage medium storing computer instructions that enable a computer to perform the method as described in either the first or second aspect.

[0016] Fifthly, embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the steps of the method as described in either the first or second aspect.

[0017] The target model update method disclosed herein utilizes Gaussian primitives representing the target scene in a pre-trained 3D Gaussian model and their corresponding first probability density field. Combined with the directed distances from spatial points predicted by the target model to the surface of the target scene and their corresponding confidence levels, a loss function for the target model is constructed. This allows the use of the first probability density field provided by the 3D Gaussian model as a supervision signal to drive the distance field output by the target model to update in a direction consistent with the preset geometric features of the 3D Gaussian model, thereby improving the update accuracy and iteration efficiency of the target model. This disclosure can directly extract the target model from the 3D Gaussian model, reducing model update time and improving update accuracy and iteration efficiency compared to updating the target model using a separate network. Attached Figure Description

[0018] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 A flowchart of a model update method provided in this disclosure embodiment; Figure 2a A schematic diagram of a Gaussian ellipse provided in an embodiment of this disclosure; Figure 2b for Figure 2a A schematic diagram of the probability density corresponding to a Gaussian ellipse; Figure 2c This is a schematic diagram of the confidence level, directed distance value, directed distance value greater than 0, second probability density corresponding to SDF model, third probability density corresponding to 3D Gaussian model, and Gaussian ellipse provided in an embodiment of this disclosure; Figure 2d To Figure 2c A schematic diagram showing the result after reducing the eikonal loss; Figure 2e This is a schematic diagram of the result after normalizing the Gaussian ellipse; Figure 2f To Figure 2e The result is illustrated after weight reduction; Figure 2g This is a schematic diagram of the results after optimization using score matching; Figure 2h A schematic diagram illustrating the results of non-normalized random score matching; Figure 2i To Figure 2h A schematic diagram showing the results of normalized recovery; Figure 3 A flowchart of a three-dimensional reconstruction method provided in this disclosure embodiment; Figure 4 A structural block diagram of a model update device is also provided as an embodiment of this disclosure; Figure 5 A structural block diagram of a three-dimensional reconstruction device provided in an embodiment of this disclosure; Figure 6 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0019] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding; these should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description. It should be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0020] Figure 1 A flowchart of a model update method provided in an embodiment of this disclosure.

[0021] See Figure 1 The process specifically includes the following steps: Step 101: Construct a loss function based on the first probability density field, the directed distance, and the confidence level corresponding to the directed distance.

[0022] The first probability density field is generated based on a preset three-dimensional Gaussian model and is used to express the probability density field corresponding to the Gaussian elements of the target scene; the directed distance is the directed distance parameter from the spatial point to the target scene output by the target model to be updated, and the confidence level is the confidence level parameter output by the target model to be updated for the directed distance.

[0023] Specifically, the first probability density field represents the overall probability density at any point in three-dimensional space, contributed by all Gaussian elements corresponding to the target scene. The target scene can be modeled, represented, rendered, or generated using a 3D Gaussian model to represent a specific 3D visual environment or object. The 3D Gaussian model contains multiple Gaussian elements (also called Gaussian spheres, Gaussian ellipsoids, or Gaussian points), each of which can characterize a local geometry within the target scene; for each Gaussian element, its corresponding local probability density function can be represented based on its parameters. The first probability density field comprises all the local probability density functions corresponding to the multiple Gaussian elements in the 3D Gaussian model.

[0024] The parameters of a Gaussian element can include position, shape, and opacity parameters. The position parameter represents the center position of the Gaussian element in three-dimensional space; the shape parameter represents the ellipsoidal shape and orientation of the Gaussian element in three-dimensional space; and the opacity parameter represents the weight of the Gaussian element's contribution to the final color. Of course, the parameters of a Gaussian element can also include other parameters, which are not limited in this disclosure.

[0025] A target model is a mathematical model or neural network used to accurately represent a three-dimensional geometric surface. Its core idea is to use a continuous function to describe the shortest distance from any point in space to the surface of the target scene, and to distinguish whether the point is inside or outside the target scene by the sign of this distance. Specifically, a positive directed distance indicates that the corresponding spatial point is outside the target scene surface; a directed distance of 0 indicates that the corresponding spatial point is on the target scene surface; and a directed distance of negative indicates that the corresponding spatial point is inside the target scene surface.

[0026] In other words, the input to the target model is the coordinates of a point in three-dimensional space, and the output is the directed distance from that point to the surface of the target scene. Furthermore, the target model can also output a confidence score, which corresponds to the directed distance and is used to characterize the reliability of the obtained directed distance. Optionally, depending on the needs of the actual scene, the target model can also output some other additional information, such as the feature vector and color of the point in space; this disclosure does not limit this.

[0027] In some embodiments, the target model can be an SDF (Signed Distance Function) model. When the target model is an SDF model, the value of the directed distance can be a signed distance value, or simply an SDF value. Of course, the target model can also be other models that can accurately represent the three-dimensional geometric surface, and this disclosure does not limit it.

[0028] The 3D Gaussian model fits the target scene using Gaussian primitives, representing its three-dimensional geometric representation. The first probability density field, derived from the 3D Gaussian model, represents the probability density at the corresponding spatial points. The directed distances and confidence scores predicted by the target model, describing the target scene through the distances between spatial points and the scene, represent its three-dimensional geometric surface representation. The loss function of the target model, constructed based on the 3D Gaussian model, measures the consistency between the distance field predicted by the target model and the geometry represented by the 3D Gaussian model, while also considering confidence scores to handle the uncertainty of directed distances.

[0029] In some embodiments, when constructing the loss function, at least one of the following constraints may be further satisfied: (1) Consistency constraint between the directed distance output by the target model and the first probability density field of the 3D Gaussian model: the region with high density (near the surface) should have a small |d(x)|, where |d(x)| represents the directed distance, that is, the directed distance of the region with high density can be close to 0.

[0030] Confidence weighting: Using confidence weighting loss makes the constraints in high-confidence regions stronger.

[0031] (2) Eikonal constraints are used to ensure that the gradient norm of the target model is close to 1, so as to obtain a valid directed distance. That is: through |||| d||-1|| constrains the target model for spatial points at scale x.

[0032] In some embodiments, other constraints can be added according to the actual scenario. For example, when a spatial point moves towards the target scene surface, the corresponding movement distance value is positive; conversely, when a spatial point moves in the opposite direction to the target scene surface, the corresponding movement distance value is negative, i.e.: ||dμi +λ dμ i -λ∣∣+∣∣dμ i -λ dμ i +λ∣∣, where μ is the center coordinate of the Gaussian point. i As a constraint, the directed distance to the target model is dμ. i This disclosure does not impose any restrictions on the constraints and boundary conditions.

[0033] Step 102: Send the loss function to the target model so that the target model can be updated based on the loss function.

[0034] Specifically, the loss function constructed in step 101 above is associated with a preset 3D Gaussian model. Its core function is to quantify the degree of matching between the distance field output by the target model and the geometry represented by the preset 3D Gaussian model. Updating the target model based on this loss function can drive the distance field output by the target model to be updated in a direction that is more consistent with the geometric features of the preset 3D Gaussian model, thereby improving the update accuracy and iteration efficiency of the target model.

[0035] The target model update method disclosed herein utilizes Gaussian primitives representing the target scene in a pre-trained 3D Gaussian model and their corresponding first probability density field. Combined with the directed distances from spatial points predicted by the target model to the surface of the target scene and their corresponding confidence levels, a loss function for the target model is constructed. This allows the use of the first probability density field provided by the 3D Gaussian model as a supervision signal to drive the distance field output by the target model to update in a direction consistent with the preset geometric features of the 3D Gaussian model, thereby improving the update accuracy and iteration efficiency of the target model. This disclosure can directly extract the target model from the 3D Gaussian model, reducing model update time and improving update accuracy and iteration efficiency compared to updating the target model using a separate network.

[0036] Since the 3D Gaussian model and the target model are different representations of the same target scene, there should be a certain correlation between the first probability density field corresponding to the Gaussian elements used to represent the target scene and the probability density represented by the spatial points corresponding to the target model. Figure 1 In step 101, the loss function can be constructed using two probability densities. A specific implementation method is given below.

[0037] In some embodiments, constructing a loss function based on a first probability density field, a directed distance, and a confidence level corresponding to the directed distance includes: constructing a second probability density field based on the directed distance and the confidence level; wherein the second probability density field is used to describe the probability distribution followed by the directed distance at a spatial point; and constructing a loss function based on the first probability density field and the second probability density field.

[0038] Among them, the second probability density field can be constructed using NeuS (Neural Implicit Surface Reconstruction Model) through directed distance and confidence.

[0039] Specifically, directed distance can be combined with a truncated exponential function to transform distance information into "density," which can then be used to construct a distribution through probability integration. The closer the distance to the surface, the higher the density and the higher the sampling probability. Of course, other methods can also be used to convert directed distance into density, thereby constructing a second probability density field; this disclosure does not limit this approach.

[0040] Then, the loss function of the target model can be constructed using the first probability density and the second probability density to minimize the difference between the first probability density field and the second probability density field, thus obtaining the target model.

[0041] This disclosure constructs a second probability density field by converting directed distance into density, and uses the first and second probability densities to construct the loss function of the target model. This allows the training of the target model using a pre-trained explicit Gaussian model as supervision. The embodiments of this disclosure combine the advantages of two density representations: the first probability density of the 3D Gaussian model excels at capturing high-frequency geometric details and ensuring consistency in real-time rendering, while the second probability density of the target model possesses strict symbolic geometric constraints and continuous smoothness. By fusing the first probability density of the 3D Gaussian model and the second probability density of the target model to construct the loss function, the detail advantages of 3DGS can compensate for the smoothness deficiencies of the target model, and the symbolic constraints of the target model can solve the geometric topological ambiguity problem of 3DGS. Ultimately, this results in a multi-dimensional improvement in geometric accuracy, convergence efficiency, and rendering quality of the target model, making it particularly suitable for scenarios requiring a balance between geometric accuracy and rendering efficiency.

[0042] Furthermore, in some embodiments, constructing a loss function for the target model based on a first probability density field and a second probability density field includes: constructing a relationship function based on the relationship between the first probability density field and the second probability density field; obtaining an initial loss function for the target model based on the relationship function; and optimizing the initial loss function through a score matching method to obtain the final loss function for the target model.

[0043] Specifically, the relationship between the 3D Gaussian model and the target model is first established. The 3D Gaussian model can be viewed as a mixture Gaussian distribution. Then, the first probability density field corresponding to the 3D Gaussian model and the second probability density field corresponding to the target model are probably identically distributed. Thus, the relationship function is constructed as follows: ;in, This represents the first probability density corresponding to the target model. This represents the second probability density corresponding to the 3D Gaussian model, where α represents a constant coefficient used for balancing. and The relationship between them.

[0044] Furthermore, based on the aforementioned relational function, the initial loss function (loss) of the target model can be obtained as follows: ,in, This represents the expectation of the spatial points corresponding to the target model.

[0045] Furthermore, due to the existence of α, the initial loss function can be optimized through score matching to process α.

[0046] Specifically, score matching can be used, which involves gradient operations. The optimized loss function is then: ; also, Where s represents the directed distance (s is the SDF value when the target model is an SDF model), and d represents the confidence level; ;in, This represents the density of the i-th Gaussian element at point x in space. = ; The opacity / scaling factor of the i-th Gaussian element controls the "intensity weight" of the Gaussian in space, where x is any point in space (a three-dimensional vector with coordinates (x, y, z)). This represents the center position of the i-th Gaussian element (a three-dimensional vector, which is the point with the highest Gaussian density). The inverse (or its simplified form) of the covariance matrix of the i-th Gaussian element is used to describe the shape, size, and orientation of the Gaussian. This represents the square of the Mahalanobis distance, describing the "weighted distance" of point x relative to the Gaussian center μi (reflecting the influence of the Gaussian shape on the distance).

[0047] Going a step further, ;in, It represents its derivative with respect to x.

[0048] Since the 3D Gaussian model is a known model that has already been trained, while the target model is an unknown model to be trained, the loss can be further optimized as follows: ; Going a step further, remove The term is an unbiased estimate of the result, affecting only the optimization weights between different x values. This can be further... Elimination yields the final loss function of the target model:

[0049] In this embodiment, the unknown normalization constant can be eliminated through score matching. Simultaneously, score matching transforms the first probability density, which is determined by millions of Gaussian elements, into a defined gradient, thus solving the problem that the first probability density corresponding to the 3D Gaussian model cannot be directly obtained due to the large number of Gaussian elements. Furthermore, since the first probability density corresponding to the 3D Gaussian model cannot be directly calculated due to the large number of Gaussian elements, this disclosure removes weight normalization. The method provided by this disclosure can construct a loss function for the target model based on the trained 3D Gaussian model, minimizing the difference between the logarithm of the gradient of the first probability density field and the logarithm of the gradient of the second probability density field, directly obtaining the target model and improving the efficiency and accuracy of target model training.

[0050] Furthermore, this disclosure can also determine the loss function of the target model directly based on the first probability density field corresponding to the 3D Gaussian model, the directed distance predicted by the target model, and the confidence level, without relying on the correlation between probability densities. A specific implementation method is given below.

[0051] In some embodiments, a loss function is constructed based on a first probability density field, directed distances, and confidence levels corresponding to those directed distances, including: constructing weight parameters corresponding to each sampled Gaussian element based on the first probability density field; constructing feature differences between sampled spatial points and sampled Gaussian elements based on directed distances between multiple sampled spatial points and their corresponding confidence levels, as well as center position parameters of each sampled Gaussian element; and constructing a loss function for the target model based on the expected values ​​of multiple sampled spatial points in the 3D spatial distribution, weight parameters, and feature differences.

[0052] Among them, each sampled Gaussian element is obtained by randomly sampling all Gaussian elements corresponding to the three-dimensional Gaussian model; multiple sampled spatial points are obtained by randomly sampling all spatial points corresponding to the target model.

[0053] Specifically, the number of Gaussian elements corresponding to the 3D Gaussian model is in the millions. Therefore, in the process of constructing the loss function, only a portion of the Gaussian elements corresponding to the 3D Gaussian model can be used to construct the weight parameters in the loss function. Similarly, only the directed distance and confidence of a portion of the spatial points corresponding to the target model can be used to construct the feature difference in the loss function.

[0054] The weight parameter corresponding to the sampled Gaussian element is the density / intensity of the corresponding sampled Gaussian element at a spatial point. The center position parameter of the sampled Gaussian element refers to the point where the density of that sampled Gaussian element is the highest.

[0055] The feature difference between the sampled spatial points and the sampled Gaussian units can be used to measure the mismatch between the sampled spatial points and the sampled Gaussian units in terms of space / features; the expected value of the 3D spatial distribution is used to describe the average positional characteristics of the sampled spatial points in space, and can be used to reflect the overall distribution trend of the sampled spatial points.

[0056] This disclosure uses 3D Gaussian primitives as the basis for fitting the target model. The loss function constrains the matching degree between a single spatial point and a Gaussian primitive through feature difference, and constrains the overall spatial distribution of Gaussians through the expectation of sampling point distribution. Finally, the contribution of each Gaussian is allocated by weight parameters, forcing the zero isosurface of the target model to be close to the surface and uniformly distributed. The loss function obtained in this way can better guide the update training of the target model, and ultimately make the fitted target model more accurate.

[0057] Furthermore, in some embodiments, the feature difference between the sampling spatial points and the sampling Gaussian units is determined based on the directed distances and corresponding confidence levels of multiple sampling spatial points, as well as the center position parameters of each sampling Gaussian unit. This includes: constructing a first feature corresponding to the sampling spatial point based on the first gradient of the directed distance and the second gradient of the confidence level, the directed distance, and the confidence level; constructing a second feature based on the offset of the center position parameters of the multiple sampling Gaussian units relative to the center position of the sampling Gaussian unit; and constructing a vector distance between the first feature and the second feature based on the norm operation between the first feature and the second feature, where the vector distance is the feature difference.

[0058] Specifically, the first gradient vector represents the direction and rate of the fastest change in directed distance at the sampling point, and its magnitude should theoretically be 1 (satisfying the Eikonal equation). The second gradient reflects the change in the model's perception of the reliability of the prediction at the sampling point in space. The confidence gradient is large in regions closer to the object surface and small in regions farther away; in regions where confidence changes drastically, the gradient of directed distance needs to be given more attention. The first and second gradients are fused with the original directed distance and confidence to form a feature vector (i.e., the first feature) that comprehensively describes the geometric properties and prediction certainty of the sampling point. Optionally, these elements can be concatenated or weighted to form the feature vector, which is the first feature. This first feature simultaneously encodes the implicit surface geometric information (position, normal) of the sampling point and the uncertainty information of the network prediction.

[0059] Specifically, the first feature can be: ; d represents the directed distance, s represents the confidence level, These are their respective gradients.

[0060] For each sampled Gaussian element associated with a sampling point in the sampling space, the offset vector of the Gaussian element's center relative to the sampling point directly reflects the spatial relationship between the local geometric prior represented by the Gaussian model and the current sampling point. Based on this offset, a second feature can be constructed through a feature extraction or transformation process. This process may include normalizing the offset vector, mapping it to a high-dimensional space (e.g., through a small neural network or a fixed encoding function such as positional encoding), or further computation in conjunction with other properties of the Gaussian element (such as scale or rotation). The final constructed second feature aims to effectively characterize the structured prior information provided by the Gaussian elements observed from the sampling point in the sampling space.

[0061] The second feature can be H i Let be a positive definite matrix representing the i-th Gaussian element, used to define the orientation and shape of the Gaussian element; x represents a sampling point in the space. This represents the center position of the i-th Gaussian element.

[0062] After constructing a first feature representing the state of a point in the sampling space and a second feature representing the prior information of the associated Gaussian units, the difference between them is quantified by constructing the vector distance between them in the feature space. Specifically, the vector distance between the two features in the feature space can be constructed through norm operations, such as L1 norm, L2 norm, or cosine distance, to measure the distance between the first and second features, which is defined as the feature difference degree.

[0063] Among them, the characteristic differences can be: ; Furthermore, the final loss function of the target model can be: .

[0064] In the embodiments provided in this disclosure, the feature dissimilarity directly measures the degree of mismatch between the features predicted by the target model at the current spatial point and the geometric prior features provided by the 3D Gaussian model near that spatial point. This disclosure constructs a loss function based on this feature dissimilarity, enabling the learning of geometric features by the target model to maintain consistency with the reliable geometric distribution expressed by the 3D Gaussian sputtering model during the updating of the target model according to this loss function by minimizing such dissimilarity.

[0065] Furthermore, in some embodiments, the first gradient is orthogonal to the second gradient.

[0066] That is to say This ensures that, during the process of updating the target model based on the constructed loss function, the directed distance and confidence can be optimized separately, thereby improving the accuracy of the target model update.

[0067] In some embodiments, a loss function is constructed based on a first probability density field, directed distance, and the confidence level corresponding to the directed distance. This includes: adjusting the confidence level corresponding to the directed distance according to the current iteration number and the total iteration number of the target model to obtain an adjusted confidence level corresponding to the directed distance; and constructing the loss function based on the first probability density field, directed distance, and the adjusted confidence level corresponding to the directed distance. Specifically, the confidence level of the directed distance corresponding to the spatial point output by the target model can be used as the initial confidence level; then, during each update of the target model, the initial confidence level can be adjusted according to the training stage of the target model, and the loss function is constructed using the adjusted confidence level.

[0068] Specifically, during the target model update training process, in the early stages of training, the model has a low fit to directed distances. The core objective is to quickly capture the approximate shape and spatial distribution of objects to rapidly establish the correct spatial occupancy relationships and basic topological structure of the scene or objects. In the middle stages of training, the model has a general shape, and the core objective is to refine the surface details of objects, improving the accuracy of the target model. The training focus shifts to refining geometric details and improving the quality of the directed distance field. In the later stages of training, the model has richer details, and the core objective is to correct local errors and improve the global consistency and robustness of the target model. The goal at this stage is to make the model stably converge to a high-fidelity, robust, and easy-to-use representation.

[0069] Therefore, in the early stages of training, a loss function can be constructed by setting the confidence level to a fixed value. This allows for a faster and more accurate capture of the object's approximate shape and spatial distribution during model updates. In the middle and later stages of model updates, the loss function can be directly constructed using the confidence level corresponding to the directed distance, based on the target model's output confidence level. This allows the target model to refine the object's surface details, improving update accuracy and ultimately leading to a stable convergence of the model to a high-fidelity, robust, and easy-to-use representation.

[0070] Furthermore, this disclosure can determine the current training stage based on the total number of iterations and the current iteration number of the target model training, thereby setting an appropriate confidence level.

[0071] In some embodiments, the initial confidence level is adjusted based on the current iteration number and the total iteration number of the target model to obtain the confidence level corresponding to the directed distance, including: determining a first ratio of the current iteration number to the total iteration number; and if the first ratio is less than a first preset value, adjusting the initial confidence level to a second preset value to obtain the confidence level corresponding to the directed distance.

[0072] The training stage of the target model can be determined by the ratio of the current iteration number to the preset total iteration number. The first preset value can generally be 20%, 30%, or set according to actual needs; this disclosure does not limit this.

[0073] The target model update method disclosed herein utilizes Gaussian primitives representing the target scene in a pre-trained 3D Gaussian model and their corresponding first probability density field. Combined with the directed distances from spatial points predicted by the target model to the surface of the target scene and their corresponding confidence levels, a loss function for the target model is constructed. This allows the use of the first probability density field provided by the 3D Gaussian model as a supervision signal to drive the distance field output by the target model to update in a direction consistent with the preset geometric features of the 3D Gaussian model, thereby improving the update accuracy and iteration efficiency of the target model. This disclosure can directly extract the target model from the 3D Gaussian model, reducing model update time and improving update accuracy and iteration efficiency compared to updating the target model using a separate network.

[0074] To verify the effectiveness of the above scheme, this disclosure illustrates the scheme in conjunction with specific experiments.

[0075] See Figures 2a-2b For visualization, 2D ellipses can be used as Gaussian units to represent the target scene. Taking the SDF model as the target model as an example, the effectiveness of the above scheme is illustrated by randomly generating 2D Gaussian ellipses on a unit circle and obtaining the probability density. Among them, Figure 2a This is a schematic diagram of a Gaussian ellipse. Figure 2b for Figure 2a A schematic diagram of the probability density corresponding to a Gaussian ellipse.

[0076] Next, this embodiment will start from the most basic probability distribution consistency, and conduct experiments step by step to show that the result of using scorematching is consistent with the result of directly constraining the probability distribution.

[0077] The essential problem that needs to be optimized in the embodiments of this disclosure is the correspondence between the 3D Gaussian model and the SDF model. Since the two probability density functions are consistent, the embodiments of this disclosure can first directly calculate the probability density function with a finite number of Gaussian points. Used for optimization.

[0078] The loss is optimized using Monte Carlo KL divergence, i.e. ; The parameter α was also included in the optimization; the results can be found in [reference needed]. Figure 2c ,in, Figure 2c The six figures in the image, from left to right and from top to bottom, represent the confidence level after adding parameters, the directed distance value, the case where the directed distance value is greater than 0, the second probability density corresponding to the SDF model, the third probability density corresponding to the 3D Gaussian model, and a schematic diagram of the Gaussian ellipse.

[0079] Among them, the fitting accuracy is directly determined by the eikonal loss. Figure 2d for Figure 2c A schematic diagram showing the results corresponding to reducing the eikonal loss.

[0080] Inheriting the stability loss derived from experiments on KL divergence, the main loss is updated to score matching: ; Furthermore, softmax is first used for calculation. See results Figure 2e ,according to Figure 2e We can see that a very faithful probability density fit has been obtained.

[0081] Furthermore, converting from KL divergence to score matching, the loss threshold in the gradient domain is significantly higher than that in the probability domain. After reducing the weights, a result almost identical to that of KL divergence is obtained. See the results below. Figure 2f .

[0082] Then, verify. Randomly selecting a batch is used to optimize score matching, and the result is as follows: Figure 2g ,according to Figure 2gAs can be seen, the embodiments disclosed herein can be successfully optimized, but there is a slight loss in fitting accuracy.

[0083] Next, we consider non-normalized random score matching, which allows us to optimize independently for each Gaussian point. See [link to relevant documentation]. Figure 2h As can be seen, there was a significant loss of accuracy this time.

[0084] This is mainly because without normalization, the weights are significantly reduced, requiring more iterations. See also Figure 2i Normalization can restore the normal optimization results.

[0085] The above embodiments illustrate how to update the target model from various aspects. In order to highlight the effect of updating the trained target model as much as possible in actual use scenarios, this disclosure also provides a specific scheme for using the trained target model for three-dimensional reconstruction.

[0086] Please refer to Figure 3 , Figure 3 A flowchart of a three-dimensional reconstruction method provided in this disclosure embodiment, wherein process 300 includes the following steps: Step 301: Input multiple three-dimensional coordinates of the spatial region to be reconstructed into the trained target model to obtain multiple directed distance values ​​output by the target model corresponding to the multiple three-dimensional coordinates.

[0087] The target model is obtained according to any of the methods described in the above embodiments.

[0088] Step 302: Perform zero isosurface extraction on the directed distance values ​​to obtain a three-dimensional model of the spatial region to be reconstructed.

[0089] Since the target model in this embodiment is trained based on a 3D Gaussian model, the directed distance value obtained through the target model is more accurate, thereby improving the accuracy of 3D reconstruction.

[0090] Based on the same inventive concept as the methods described above, this disclosure also provides a training device and a three-dimensional reconstruction device.

[0091] Further reference Figure 4 and Figure 5 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a model updating device and an embodiment of a three-dimensional reconstruction device, respectively. The model updating device embodiment and... Figure 1 Corresponding to the illustrated model update method embodiment, the 3D reconstruction device embodiment is similar to... Figure 3 The illustrated three-dimensional reconstruction method corresponds to the above embodiment. The aforementioned device can be specifically applied to various electronic devices.

[0092] like Figure 4 As shown in the embodiments of this disclosure, a model update apparatus is also provided, the apparatus including: a construction module 401 and a sending module 402.

[0093] The construction module 401 is used to construct a loss function based on a first probability density field, a directed distance, and a confidence level corresponding to the directed distance. The first probability density field is generated based on a preset three-dimensional Gaussian model and is used to express the probability density field corresponding to the Gaussian units of the target scene. The directed distance is the directed distance parameter from the spatial point to the target scene output by the target model to be updated, and the confidence level is the confidence level parameter output by the target model to be updated for the directed distance. The update module 402 is used to send the loss function to the target model so that the target model can be updated based on the loss function.

[0094] In this embodiment, the specific processing of the construction module 401 and the sending module 402 in the model update device and the resulting technical effects can be referred to respectively. Figure 1 The relevant descriptions of steps 101-102 in the corresponding embodiments will not be repeated here.

[0095] In some embodiments, the construction module 401 is specifically used to: construct a second probability density field based on the directed distance and the confidence level; wherein the second probability density field is used to describe the probability distribution that the directed distance follows at a spatial point; and construct a loss function based on the first probability density field and the second probability density field.

[0096] In some embodiments, the construction module 401 is specifically used to: construct a relationship function based on the relationship between the first probability density field and the second probability density field; obtain an initial loss function based on the relationship function; and optimize the initial loss function by a score matching method to obtain the final loss function.

[0097] In some embodiments, the construction module 401 is specifically used to: construct weight parameters corresponding to each sampled Gaussian element based on the first probability density field; each sampled Gaussian element is obtained by randomly sampling all Gaussian elements corresponding to the three-dimensional Gaussian model; construct feature difference between the sampled spatial points and the sampled Gaussian elements based on the directed distance and corresponding confidence of multiple sampled spatial points, as well as the center position parameters of each sampled Gaussian element; multiple sampled spatial points are obtained by randomly sampling all spatial points corresponding to the target model; and construct a loss function based on the expected value, weight parameters, and feature difference of the multiple sampled spatial points in the three-dimensional spatial distribution.

[0098] In some embodiments, the construction module 401 is specifically used to: construct a first feature corresponding to a sampling space point based on the first gradient of the directed distance and the second gradient of the confidence level, the directed distance and the confidence level; construct a second feature based on the offset of the center position parameters of multiple sampling Gaussian units relative to the center position of the sampling space point; and construct a vector distance between the first feature and the second feature based on the norm operation between the first feature and the second feature, wherein the vector distance is the feature difference degree.

[0099] In some embodiments, the first gradient is orthogonal to the second gradient.

[0100] In some embodiments, the construction module 401 is specifically used to: adjust the confidence level corresponding to the directed distance according to the current iteration number and the total iteration number of the target model to obtain the adjusted confidence level corresponding to the directed distance; and construct a loss function based on the first probability density field, the directed distance, and the adjusted confidence level corresponding to the directed distance.

[0101] In some embodiments, the construction module 401 is specifically used to: determine a first ratio between the current iteration number and the preset total iteration number; and, if the first ratio is less than a first preset value, adjust the confidence level corresponding to the directed distance to a second preset value to obtain the adjusted confidence level corresponding to the directed distance.

[0102] like Figure 5 As shown, this disclosure also provides a three-dimensional reconstruction apparatus, which includes a calculation unit 501 and a reconstruction unit 502. The calculation unit 501 is used to input multiple three-dimensional coordinates of the spatial region to be reconstructed into a trained target model to obtain multiple directed distance values ​​output by the target model corresponding to the multiple three-dimensional coordinates; wherein the target model is obtained according to any of the method embodiments described above; the reconstruction unit 502 is used to perform zero isosurface extraction on the directed distance values ​​to obtain a three-dimensional model of the spatial region to be reconstructed.

[0103] In this embodiment, the specific processing of the calculation unit 501 and the reconstruction unit 502 in the three-dimensional reconstruction device and the resulting technical effects can be respectively corresponding to... Figure 3 The relevant descriptions in the method embodiments are not repeated here.

[0104] This embodiment exists as a device embodiment corresponding to the above method embodiment. The model update device and 3D reconstruction device provided in this embodiment... By using Gaussian primitives representing the target scene in a pre-trained 3D Gaussian model and their corresponding first probability density field, combined with the directed distances from spatial points predicted by the target model to the surface of the target scene and their corresponding confidence levels, a loss function for the target model is constructed. This allows the use of the first probability density field provided by the 3D Gaussian model as a supervisory signal to drive the distance field output by the target model to update in a direction consistent with the preset geometric features of the 3D Gaussian model, thereby improving the update accuracy and iteration efficiency of the target model. This disclosure can directly extract the target model from the 3D Gaussian model, reducing model update time and improving update accuracy and iteration efficiency compared to updating the target model separately using a network. Furthermore, obtaining the directed distance values ​​corresponding to the three-dimensional coordinates through the above target model for 3D reconstruction improves the accuracy and efficiency of 3D reconstruction.

[0105] According to embodiments of the present disclosure, the present disclosure also provides an electronic device, the electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to implement the methods described in any of the above embodiments.

[0106] Figure 6 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this disclosure. For example... Figure 6 As shown, the electronic device 600 of this embodiment includes a processor 601 and a memory 602; wherein, the memory 602 is used to store computer execution instructions; the processor 601 is used to execute the computer execution instructions stored in the memory to implement the various steps performed by the electronic device in the above embodiment. For details, please refer to the relevant descriptions in the foregoing method embodiments. For example, the electronic device 600 can be a general-purpose processor, a graphics processing device, a neural network computing device, or a graph neural network computing device.

[0107] In some embodiments, the memory 602 can be either standalone or integrated with the processor 601.

[0108] When the memory 602 is set up independently, the electronic device also includes a bus 603 for connecting the memory 602 and the processor 601.

[0109] It should be understood that the processor 601 described above can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly manifested as being executed by a hardware processor, or being executed by a combination of hardware and software modules within the processor.

[0110] The memory 602 may include high-speed RAM memory, and may also include non-volatile memory NVM, such as at least one disk storage device, and may also be a USB flash drive, portable hard drive, read-only memory, disk or optical disc, etc.

[0111] Bus 603 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0112] This disclosure also provides a computer storage medium storing computer execution instructions, which, when executed by a processor, implement the steps of any of the above method embodiments.

[0113] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method according to any of the above embodiments.

[0114] In the several embodiments provided by this invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0115] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to implement the solution of this embodiment according to actual needs.

[0116] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The unit composed of the above modules can be implemented in hardware or in the form of hardware plus software functional units.

[0117] The integrated modules described above, implemented as software functional modules, can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute partial steps of the methods in the various embodiments of this application.

[0118] The aforementioned storage medium can be implemented from any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium accessible to general-purpose or special-purpose computers.

[0119] An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor. Both the processor and the storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and storage medium can exist as discrete components in an electronic device or host device.

[0120] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0121] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0122] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0123] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A model update method, characterized in that, The method includes: A loss function is constructed based on a first probability density field, a directed distance, and the confidence level corresponding to the directed distance; wherein, the first probability density field is generated based on a preset three-dimensional Gaussian model and is used to express the probability density field corresponding to the Gaussian units of the target scene; the directed distance is the directed distance parameter from the spatial point to the target scene output by the target model to be updated, and the confidence level is the confidence level parameter output by the target model to be updated for the directed distance; The loss function is sent to the target model so that the target model is updated based on the loss function.

2. The method according to claim 1, characterized in that, The loss function is constructed based on the first probability density field, the directed distance, and the confidence level corresponding to the directed distance, including: A second probability density field is constructed based on the directed distance and the confidence level; wherein the second probability density field is used to describe the probability distribution that the directed distance follows at the spatial point; The loss function is constructed based on the first probability density field and the second probability density field.

3. The method according to claim 2, characterized in that, The step of constructing the loss function based on the first probability density field and the second probability density field includes: Based on the relationship between the first probability density field and the second probability density field, a relationship function is constructed; Based on the aforementioned relationship function, the initial loss function is obtained; The initial loss function is optimized by score matching to obtain the final loss function.

4. The method according to claim 1, characterized in that, The loss function is constructed based on the first probability density field, the directed distance, and the confidence level corresponding to the directed distance; including: The weight parameters corresponding to each sampled Gaussian element are constructed based on the first probability density field; each sampled Gaussian element is obtained by randomly sampling all Gaussian elements corresponding to the three-dimensional Gaussian model. The feature difference between the sampled spatial points and the sampled Gaussian elements is constructed based on the directed distances and corresponding confidence levels of multiple sampled spatial points, as well as the center position parameters of each sampled Gaussian element; the multiple sampled spatial points are obtained by randomly sampling all spatial points corresponding to the target model. The loss function is constructed based on the expected values ​​of the multiple sampling spatial points in the three-dimensional spatial distribution, the weight parameters, and the feature difference degree.

5. The method according to claim 4, characterized in that, The step of constructing the feature difference between the sampling spatial points and the sampling Gaussian units based on the directed distances and corresponding confidence levels of multiple sampling spatial points, and the center position parameters of each sampling Gaussian unit, includes: Based on the first gradient of the directed distance and the second gradient of the confidence level of each of the sampling spatial points, the directed distance and the confidence level, a first feature corresponding to the sampling spatial point is constructed; A second feature is constructed based on the offset of the center position parameters of the multiple sampled Gaussian elements relative to the sampled spatial points, corresponding to the offset of the sampled spatial points relative to the center position of the sampled Gaussian elements. The vector distance between the first feature and the second feature is constructed based on the norm operation between the first feature and the second feature, and the vector distance is the feature difference degree.

6. The method according to claim 5, characterized in that, The first gradient is orthogonal to the second gradient.

7. The method according to claim 1, characterized in that, The loss function is constructed based on the first probability density field, the directed distance, and the confidence level corresponding to the directed distance, including: Based on the current iteration number and total iteration number of the target model, the confidence level corresponding to the directed distance is adjusted to obtain the adjusted confidence level corresponding to the directed distance; A loss function is constructed based on the first probability density field, the directed distance, and the adjusted confidence level corresponding to the directed distance.

8. The method according to claim 7, characterized in that, The step of adjusting the confidence level corresponding to the directed distance based on the current iteration number and the total iteration number of the target model to obtain the adjusted confidence level corresponding to the directed distance includes: Determine a first ratio between the current iteration number and the preset total iteration number; If the first ratio is less than a first preset value, the confidence level corresponding to the directed distance is adjusted to a second preset value to obtain the adjusted confidence level corresponding to the directed distance.

9. A three-dimensional reconstruction method, characterized in that, The method includes: Multiple three-dimensional coordinates of the spatial region to be reconstructed are input into the trained target model to obtain multiple directed distance values ​​output by the target model corresponding to the multiple three-dimensional coordinates; wherein, the target model is obtained according to the method described in any one of claims 1-8; Zero isosurface extraction is performed on the directed distance values ​​to obtain a three-dimensional model of the spatial region to be reconstructed.

10. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-9.

11. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-9.

12. A computer program product, characterized in that, Includes a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1-9.

Images