Image processing method and device and computer program product

Abstract

The invention discloses an image processing method and apparatus, and a computer program product. The method comprises the steps of obtaining a three-dimensional Gaussian field for expressing a target scene; for each three-dimensional Gaussian function in the three-dimensional Gaussian field, calculating the uncertainty of a target parameter in the three-dimensional Gaussian function; and in the three-dimensional Gaussian field, deleting the three-dimensional Gaussian functions of which the uncertainty exceeds a preset threshold. By calculating the uncertainty of the target parameter in each three-dimensional Gaussian function, whether each three-dimensional Gaussian function can accurately describe the three-dimensional distribution of the corresponding position in the scene can be evaluated, the inaccurate description of the scene can be eliminated by eliminating the three-dimensional Gaussian function with relatively high uncertainty, and the accuracy of the scene description can be improved. The accuracy of reconstruction of the three-dimensional scene is improved, and the accuracy of further rendering the two-dimensional image based on the three-dimensional scene is also improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, device, and computer program product. Background Art

[0002] The 3D Gaussian splatting method is a 3D reconstruction method commonly used in graphics. The 3D Gaussian splatting method recovers a scene expressed as a 3D Gaussian field from a set of images with determined camera parameters, and can achieve efficient rasterization rendering based on Gaussian splatting. However, the 3D Gaussian splatting method in related technologies relies on dense training perspectives. In areas with sparse perspectives, due to the lack of multi-view constraints, scene areas that are not covered by sufficiently dense training perspectives will also produce serious reconstruction errors, and the geometric structure will have serious defects. When the observation perspective deviates from the training perspective, the rendering result will be significantly different from the actual result. Summary of the Invention

[0003] The purpose of this application is to propose an image processing method, device and computer program product to address the deficiencies of the above-mentioned prior art, so as to at least solve the deficiencies of the above-mentioned prior art. This purpose is achieved through the following technical solutions.

[0004] A first aspect of the present application provides an image processing method, the method comprising:

[0005] Obtain a three-dimensional Gaussian field for expressing the target scene;

[0006] For each three-dimensional Gaussian function in the three-dimensional Gaussian field, calculating the uncertainty of the target parameter in the three-dimensional Gaussian function;

[0007] In the three-dimensional Gaussian field, the three-dimensional Gaussian functions whose uncertainty exceeds a preset threshold are deleted.

[0008] Optionally, calculating the uncertainty of a target parameter in each three-dimensional Gaussian function in the three-dimensional Gaussian field includes:

[0009] For each of the three-dimensional Gaussian functions, based on a known target image, a Laplace approximation algorithm is used to perform a posteriori estimation on the target parameters to obtain a posterior probability distribution of the target parameters; the target image is obtained by photographing the target scene at a target perspective;

[0010] The uncertainty of the target parameter in the three-dimensional Gaussian function is calculated according to the posterior probability distribution of the target parameter.

[0011] Optionally, performing a posteriori estimation on the target parameters using a Laplace approximation algorithm based on the known target image to obtain a posterior probability distribution of the target parameters includes:

[0012] performing a reparameterization operation on the target parameter in the three-dimensional Gaussian function to obtain an updated target parameter;

[0013] Replacing the target parameter in the three-dimensional Gaussian function with the updated target parameter to obtain an updated three-dimensional Gaussian function;

[0014] estimating a Hessian matrix of the updated three-dimensional Gaussian function based on a diagonal estimation algorithm;

[0015] The posterior probability distribution of the target parameter is determined according to the Hessian matrix of the updated three-dimensional Gaussian function.

[0016] Optionally, estimating the Hessian matrix of the updated three-dimensional Gaussian function based on a diagonal estimation algorithm includes:

[0017] The Hessian matrix of the updated three-dimensional Gaussian function is estimated based on the generalized Gauss-Newton matrix.

[0018] Optionally, determining the posterior probability distribution of the target parameter according to the updated Hessian matrix of the three-dimensional Gaussian function includes:

[0019] Determining a covariance matrix in a posterior probability distribution of the target parameter according to the Hessian matrix of the updated three-dimensional Gaussian function to obtain the determined posterior probability distribution;

[0020] Calculating the uncertainty of the target parameter in the three-dimensional Gaussian function according to the posterior probability distribution of the target parameter includes:

[0021] The uncertainty is determined based on the covariance matrix.

[0022] Optionally, the target parameter of the three-dimensional Gaussian function is the center point position of the three-dimensional Gaussian function; and performing a reparameterization operation on the target parameter in the three-dimensional Gaussian function to obtain an updated target parameter includes:

[0023] Determining a three-dimensional grid in a gridded space where a center point of the three-dimensional Gaussian function is located to obtain a target three-dimensional grid; the gridded space is obtained by dividing the three-dimensional space where the target scene is located into a plurality of three-dimensional grids of a preset size, wherein each vertex in each of the three-dimensional grids is provided with a corresponding three-dimensional perturbation amount;

[0024] An updated center point position is generated based on the trilinear interpolation of the three-dimensional perturbation amount of each vertex in the target three-dimensional mesh and the center point position of the three-dimensional Gaussian function; the updated center point position is the center point position in the updated three-dimensional Gaussian function.

[0025] Optionally, determining the posterior probability distribution of the target parameter according to the updated Hessian matrix of the three-dimensional Gaussian function includes:

[0026] estimating a posterior probability distribution of each vertex in the target three-dimensional grid according to the updated Hessian matrix of the three-dimensional Gaussian function;

[0027] According to the posterior probability distribution of each vertex in the target three-dimensional grid, the posterior probability distribution of the center point position is calculated by trilinear interpolation.

[0028] Optionally, obtaining a three-dimensional Gaussian field for expressing a target scene includes:

[0029] Obtaining a set of initial three-dimensional Gaussian functions; each of the initial three-dimensional Gaussian functions includes a plurality of parameters, and the plurality of parameters include the target parameter;

[0030] According to the target image, iterative training is performed on the parameters of the initial three-dimensional Gaussian function to obtain the three-dimensional Gaussian field.

[0031] A second aspect of the present application provides an image processing device, comprising:

[0032] an acquisition unit, configured to acquire a three-dimensional Gaussian field for expressing a target scene;

[0033] a calculation unit, configured to calculate, for each three-dimensional Gaussian function in the three-dimensional Gaussian field, an uncertainty of a target parameter in the three-dimensional Gaussian function;

[0034] A deleting unit is used to delete the three-dimensional Gaussian function whose uncertainty exceeds a preset threshold in the three-dimensional Gaussian field.

[0035] A third aspect of the present application provides a computer program product, comprising a computer program, wherein the computer program is executed by a processor to implement the method described in the first aspect of the present application.

[0036] Based on the method described in the first aspect above, this application has at least the following beneficial effects or advantages:

[0037] After obtaining a three-dimensional Gaussian field used to express a target scene, the uncertainty of the target parameters in each three-dimensional Gaussian function in the three-dimensional Gaussian field is calculated. Furthermore, three-dimensional Gaussian functions whose uncertainty exceeds a preset threshold can be deleted from the three-dimensional Gaussian field. By calculating the uncertainty of the target parameters in each three-dimensional Gaussian function, the present application can evaluate whether each three-dimensional Gaussian function can accurately describe the three-dimensional distribution at the corresponding position in the scene. By eliminating three-dimensional Gaussian functions with higher uncertainties, inaccurate descriptions of the scene can be eliminated, thereby improving the accuracy of three-dimensional scene reconstruction and the accuracy of further rendering of two-dimensional images based on the three-dimensional scene.

[0038] The above description is only an overview of the technical solution of this application. In order to more clearly understand the technical means of this application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of this application more obvious and easy to understand, the specific implementation methods of this application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0040] Figure 1 A schematic diagram of the principle of a three-dimensional Gaussian splattering method in an image processing method according to some embodiments;

[0041] Figure 2 is a flowchart illustrating an image processing method according to some embodiments;

[0042] Figure 3 is a schematic diagram showing the effect of an image processing method according to some embodiments;

[0043] Figure 4 is a schematic structural diagram of an image processing device according to some embodiments;

[0044] Figure 5 A schematic diagram of the hardware structure of an electronic device according to some embodiments is shown. DETAILED DESCRIPTION

[0045] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0046] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. As used in this application and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0047] It should be understood that although the terms first, second, third, etc. may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0048] 3D reconstruction and new perspective rendering are key technologies in graphics. The 3D Gaussian splatting method is one of the commonly used methods in 3D reconstruction and new perspective rendering. The 3D Gaussian splatting method uses a set of images with certain camera parameters to reconstruct the 3D scene, recovering the 3D structure expressed by the 3D Gaussian field, and then implements efficient rasterization rendering based on the 3D Gaussian splatting, rendering the 3D scene into a 2D image. Figure 1 The three-dimensional Gaussian field on the right includes multiple three-dimensional Gaussian functions. The three-dimensional Gaussian function is the primitive for expressing the three-dimensional scene. Specifically, the three-dimensional scene is represented as a set of three-dimensional Gaussian functions.

[0049] An embodiment of the present application provides an image processing method, in which a three-dimensional Gaussian field of a three-dimensional scene of a two-dimensional image under a known partial viewing angle can be further processed, and three-dimensional Gaussian functions whose uncertainty of target parameters of the three-dimensional Gaussian functions in the three-dimensional Gaussian field exceeds a threshold are deleted, thereby deleting the three-dimensional Gaussian functions that do not accurately express the three-dimensional scene, improving the accuracy of three-dimensional reconstruction, and facilitating further rendering.

[0050] refer to Figure 2 The image processing method provided in this embodiment includes the following steps:

[0051] Step 101: Obtain a three-dimensional Gaussian field for expressing a target scene.

[0052] Step 102 : For each three-dimensional Gaussian function in the three-dimensional Gaussian field, calculate the uncertainty of the target parameter in the three-dimensional Gaussian function.

[0053] Step 103: In the three-dimensional Gaussian field, delete the three-dimensional Gaussian functions whose uncertainty exceeds a preset threshold.

[0054] The target scene is a three-dimensional scene, and the three-dimensional Gaussian field used to express the target scene includes multiple three-dimensional Gaussian functions. In some embodiments, step 101 may be to read a known three-dimensional Gaussian field. In other embodiments, step 101 may include performing three-dimensional reconstruction on the target scene to obtain the three-dimensional Gaussian field.

[0055] Specifically, the process of reconstructing a 3D scene based on a 2D image at a known viewing angle includes:

[0056] The following parameters are initialized randomly: the center point μ, size S, rotation R, red, green, and blue (RGB) color c, and opacity o of the 3D Gaussian function. A 2D image of the 3D scene at a known viewing angle is obtained, and the 3D Gaussian function is used to represent the projection of the 3D scene in 2D space at the same viewing angle, resulting in a rendered image. The color difference between the rendered image and the real image at the corresponding viewing angle is used as a loss function. These parameters are then iteratively trained to obtain the optimized updated parameters, thereby determining the 3D Gaussian function.

[0057] The following describes in detail the principle of iterative training of the above parameters.

[0058] First, each three-dimensional Gaussian function can be expressed by the following formula:

[0059]

[0060] Among them, the 3×3 covariance matrix Σ of each three-dimensional Gaussian function is obtained by rotation and size combination:

[0061] Σ=RSS T R T Formula 2

[0062] The covariance matrix Σ of each three-dimensional Gaussian function describes the distribution characteristics of the three-dimensional Gaussian function and defines the extension degree and direction of the three-dimensional Gaussian function in the three-dimensional space.

[0063] Secondly, project the three-dimensional Gaussian function along the same viewing direction as the known viewing angle, referring to Figure 1 , thus obtaining a two-dimensional Gaussian function on the viewing plane. Due to the properties of the three-dimensional Gaussian function itself, the two-dimensional Gaussian function can be easily derived in analytical form and rasterized according to the RGB color c and opacity o on the viewing plane to obtain the final two-dimensional RGB image. Among them, the two-dimensional Gaussian distribution φ k The covariance matrix Σ′ of (x) k It can be estimated by local affine transformation. Specifically, at the center point x of the two-dimensional Gaussian distribution kAt , perform a second-order Taylor expansion and get:

[0064] φ k (x)=φ k (x k )+J k (xx k ) Formula 3

[0065] in:

[0066]

[0067] The covariance matrix Σ′ of the projected two-dimensional Gaussian distribution k It can be expressed as:

[0068]

[0069] Where M represents the matrix of projection transformation.

[0070] Through the above rendering method, the rendering result is differentiable with respect to the various properties of the three-dimensional Gaussian itself, so the parameters can be optimized by the gradient descent method. Specifically, the optimization algorithm uses the color difference between the rendered image and the real image as the loss function, and jointly optimizes the parameters of the three-dimensional Gaussian function by the gradient descent method. At the same time, for the loss function used to represent the color difference, the parameters of the center point position are gradient detected. The specific steps of gradient detection include taking the partial differentiation of the color difference between the rendered image and the real image with respect to the parameters of the center point position to obtain the gradient of the center point position. If the gradient of the center point position of a three-dimensional Gaussian function is greater than the threshold λ, it means that the fitting of the three-dimensional Gaussian function to the three-dimensional scene is not accurate enough, and the number of three-dimensional Gaussian functions needs to be increased. Therefore, the three-dimensional Gaussian function can be copied and further split, so that the three-dimensional Gaussian function can more accurately restore the geometric structure in the three-dimensional scene. The above optimization algorithm can efficiently complete the expression of the three-dimensional scene.

[0071] During the actual optimization process, the learning rates can be set to 0.00016, 0.0025, 0.005, 0.001, and 0.05 for the center point μ, RGB color c, size information S, rotation information R, and opacity o of the 3D Gaussian function, respectively. The gradient threshold λ can be set to 0.1 for general scenarios, and a smaller value, such as 0.005, can be used for outdoor scenes with high precision requirements.

[0072] Optionally, step 101 obtains a three-dimensional Gaussian field for expressing the target scene, which may specifically include executing the following steps: obtaining a set of initial three-dimensional Gaussian functions; each initial three-dimensional Gaussian function includes multiple parameters, and the multiple parameters include target parameters; according to the target image, performing iterative training on each parameter in the initial three-dimensional Gaussian function to obtain a three-dimensional Gaussian field.

[0073] Through iterative training of parameters, the initial parameters can be adjusted according to the data of the target image to better adapt to the data of the target image, thereby improving the expression accuracy of the three-dimensional Gaussian field to the target scene.

[0074] By optimizing the initialized parameters through the above-mentioned iterative training method, the three-dimensional reconstruction result, namely each three-dimensional Gaussian function, can be obtained.

[0075] After obtaining the three-dimensional Gaussian function, the uncertainty of the target parameter in the three-dimensional Gaussian function is calculated for each three-dimensional Gaussian function in the three-dimensional Gaussian field. The three-dimensional Gaussian function includes multiple related parameters, and the target parameter is some or all of the parameters in the three-dimensional Gaussian function. In some embodiments, the center point position of the three-dimensional Gaussian function is selected as the target parameter, and the uncertainty of the center point position of the three-dimensional Gaussian function is calculated. The calculation of uncertainty can be achieved by calculating the variance and standard deviation, or by using calculation methods such as the Monte Carlo algorithm, the empirical Bayesian algorithm, and the interval estimation algorithm.

[0076] In some implementations of the present application, the uncertainty of the target parameter in the three-dimensional Gaussian function is calculated by the Laplace approximation algorithm. The Laplace approximation algorithm is a method based on Bayesian statistics for estimating uncertainty. Specifically, it is assumed that the three-dimensional Gaussian field in step 101 is obtained by a known training data set. With parameters As a function of the parameter to be optimized In a known training dataset Above Train the loss function to obtain the maximum a posteriori estimate θ of the parameter θ MaP :

[0077]

[0078] Among them, x n Represents the training dataset For each input sample in y n Represents the training dataset Each output sample in, N represents the number of samples, represents an M-dimensional vector space, Representing a C-dimensional vector space, the parameter θ can be part or all of the center point position μ, RGB color c, size information S, rotation information R and opacity o of the above-mentioned three-dimensional Gaussian function. Optionally, in some embodiments, other parameters may also be included.

[0079] The posterior probability distribution of the parameter θ satisfies:

[0080]

[0081] For the loss function In the maximum a posteriori estimate θ MAP At , performing a second-order Taylor expansion, we can get the following loss function The following approximate expression:

[0082]

[0083] Therefore, the loss function It is formally equivalent to the probability density function of the K-dimensional Gaussian distribution, which leads to the Gaussian distribution approximation of the posterior probability distribution:

[0084]

[0085] The second-order Hessian matrix term in Formula 9 It can be estimated by generalizing the Gauss-Newton matrix, which is an approximate Hessian matrix used to estimate the covariance matrix of the parameters or for optimization. The formula estimated by generalizing the Gauss-Newton matrix is:

[0086]

[0087] In formula 10, the Jacobian matrix is the log-likelihood function logp(y n |f) About output f The second-order derivative calculated at , so that the uncertainty of the parameter θ can be estimated. Since the posterior probability distribution of formula 9 is approximately consistent with the Gaussian distribution, the covariance matrix in the Gaussian distribution of formula 9 is That is, it reflects the uncertainty of the parameter θ.

[0088] Based on the above principles, in some specific implementations, step 102 calculates the uncertainty of the target parameters (which may be part or all of the above θ) in the three-dimensional Gaussian function for each three-dimensional Gaussian function in the three-dimensional Gaussian field, and may specifically include the following steps: for each three-dimensional Gaussian function, based on a known target image, a Laplace approximation algorithm is used to perform a posteriori estimation of the target parameters to obtain a posterior probability distribution of the target parameters; the target image is obtained by photographing the target scene from a target perspective; and based on the posterior probability distribution of the target parameters, the uncertainty of the target parameters in the three-dimensional Gaussian function is calculated.

[0089] The above-described specific implementation uses the Laplace approximation algorithm to provide a method for approximating the posterior probability distribution to a Gaussian distribution within a local region. This helps find the most likely parameter value and its distribution characteristics in the parameter space, thereby improving the accuracy of the estimation. Compared to full Bayesian methods or other global optimization methods, the Laplace approximation algorithm is computationally less complex because it only considers the area near the peak of the posterior probability distribution, resulting in faster optimization results.

[0090] Since the actual solution of the covariance matrix in the Laplace approximation algorithm often faces the problem of the Hessian matrix being too large in dimension but with limited computing resources, in some implementations provided in the embodiments of the present application, a diagonal estimation algorithm is used to approximate the Hessian matrix. The diagonal estimation algorithm requires a weak correlation between the parameters θ, while the parameters of each three-dimensional Gaussian function are strongly correlated. In order to reduce the correlation between the parameters and thus ensure the sparsity of the Hessian matrix, a reparameterization operation can be used to convert the parameters of the three-dimensional Gaussian function into updated parameters to keep the approximation error within an acceptable range.

[0091] In some specific implementations of the embodiments of the present application, the above-mentioned step of performing a posteriori estimation on the target parameters based on the known target image using the Laplace approximation algorithm to obtain the posterior probability distribution of the target parameters can specifically include the following steps: performing a reparameterization operation on the target parameters in the three-dimensional Gaussian function to obtain updated target parameters; replacing the target parameters in the three-dimensional Gaussian function with the updated target parameters to obtain an updated three-dimensional Gaussian function; estimating the Hessian matrix of the updated three-dimensional Gaussian function based on the diagonal estimation algorithm; and determining the posterior probability distribution of the target parameters based on the Hessian matrix of the updated three-dimensional Gaussian function.

[0092] Using a diagonal estimation algorithm to approximate the Hessian matrix can reduce computational complexity while maintaining reasonable estimates under the assumption of weak correlation between parameters. By considering only the diagonal elements of the Hessian matrix, the diagonal estimation algorithm simplifies the computational process, making uncertainty estimates more efficient. By reducing the complexity of the Hessian matrix calculation, the demand on computing resources can be reduced, leading to more efficient use of computing resources.

[0093] In some specific implementations of the embodiments of the present application, the Hessian matrix of the updated three-dimensional Gaussian function is estimated based on the diagonal estimation algorithm. Specifically, the generalized Gauss-Newton matrix can be used to estimate the Hessian matrix of the updated three-dimensional Gaussian function. The generalized Gauss-Newton matrix takes into account the second-order sensitivity of the model output to parameter changes, which helps to estimate the uncertainty of the parameters more robustly during the optimization process. The Hessian matrix can be approximated as a diagonal matrix with only non-zero diagonals through Formula 10, and the Hessian matrix can be estimated using the generalized Gauss-Newton matrix.

[0094] In some specific implementations of the embodiments of the present application, the above-mentioned step of determining the posterior probability distribution of the target parameter based on the updated Hessian matrix of the three-dimensional Gaussian function can be specifically performed as follows: determining the covariance matrix in the posterior probability distribution of the target parameter based on the updated Hessian matrix of the three-dimensional Gaussian function to obtain a determined posterior probability distribution; accordingly, the above-mentioned step of calculating the uncertainty of the target parameter in the three-dimensional Gaussian function based on the posterior probability distribution of the target parameter can be determined based on the covariance matrix. This specific implementation method provides an accurate measure of the posterior probability distribution of the target parameter through the covariance matrix determined by the Hessian matrix, which helps to more accurately describe the uncertainty of the target parameter. Based on the approximate formula of the posterior probability distribution provided by Formula 9, it can be seen that its covariance matrix reflects the relative size of the uncertainty.

[0095] In some specific implementations of the present application, the target parameter is the center point position μ of the three-dimensional Gaussian function. k The above-mentioned step of performing a reparameterization operation on the target parameters in the three-dimensional Gaussian function to obtain updated target parameters can specifically include performing the following steps: determining the three-dimensional grid in which the center point position of the three-dimensional Gaussian function is located in the gridded space to obtain a target three-dimensional grid; the gridded space is obtained by dividing the three-dimensional space where the target scene is located into multiple three-dimensional grids of preset sizes, and each vertex in each three-dimensional grid is set with a corresponding three-dimensional perturbation amount; generating an updated center point position based on trilinear interpolation of the three-dimensional perturbation amount of each vertex in the target three-dimensional grid and the center point position of the three-dimensional Gaussian function; the updated center point position is the center point position in the updated three-dimensional Gaussian function.

[0096] Specifically, first, the three-dimensional space is divided into three-dimensional grids of fixed size. Each three-dimensional grid can be a cube / cuboid with a total of 8 vertices. Secondly, a three-dimensional perturbation amount σ is assigned to each vertex. Finally, for the center point position μ of any three-dimensional Gaussian function k , determine the center point position after reparameterization as μ′ k =μ k +σ k , where σ k is μk The three-dimensional perturbation of the eight nearest vertices of the grid is performed by trilinear interpolation. The updated center point position μ′ k To replace the original center point position μ k , represents the updated three-dimensional Gaussian function G′ k (x) is:

[0097]

[0098] By assigning a three-dimensional perturbation to each location in the grid space and replacing the original center point position with the reparameterized center point position, the correlation of the center point position parameters of the three-dimensional Gaussian function is weakened, providing the prerequisite for simplifying the Hessian matrix and diagonalizing the matrix, and improving the approximation accuracy of the Hessian matrix approximation matrix.

[0099] Optionally, the above-mentioned step of determining the posterior probability distribution of the target parameters based on the Hessian matrix of the updated three-dimensional Gaussian function may specifically include the following steps: estimating the posterior probability distribution of each vertex in the target three-dimensional grid based on the Hessian matrix of the updated three-dimensional Gaussian function; calculating the posterior probability distribution of the center point position by trilinear interpolation based on the posterior probability distribution of each vertex in the target three-dimensional grid.

[0100] Based on the posterior probability distribution of each vertex in the target 3D mesh, the posterior probability distribution of each point in the 3D space of the target 3D mesh can be calculated. Therefore, at the original center point of the 3D Gaussian function, trilinear interpolation is performed based on the distances between this center point and each vertex in the target 3D mesh to obtain the posterior probability distribution of the original center point of the 3D Gaussian function. Trilinear interpolation provides smooth and continuous probability estimates for each internal point in the mesh through trilinear interpolation based on the posterior probability distribution of each vertex, thereby improving the accuracy of the posterior probability distribution at the center point.

[0101] refer to Figure 3 , Figure 3 The right side of the figure is a real image of a 3D scene at a known viewing angle, and the left side is a rendered image obtained by performing 3D Gaussian splashing based on a 3D Gaussian field at an unknown viewing angle. Through the image processing method provided in the embodiment of the present application, the uncertainty of each 3D Gaussian function in the 3D Gaussian field can be calculated. Figure 3 The area with larger uncertainty is eliminated on the left side, and the area with smaller uncertainty is retained, thereby removing the defects of 3D reconstruction and improving the image rendering quality of the 3D scene in a new perspective.

[0102] In some embodiments of the embodiments of the present application, the uncertainty of the three-dimensional Gaussian splash is analyzed using a Laplace approximation algorithm, the three-dimensional Gaussian splash is reparameterized based on a trilinear interpolation grid method, and the Hessian matrix of the three-dimensional Gaussian is estimated using a generalized Gauss-Newton matrix method. Through a series of reparameterization, generalized Gauss-Newton matrix and diagonalization estimation algorithms, a more accurate estimation of the uncertainty of the three-dimensional Gaussian splash can be provided, and the three-dimensional Gaussian function that inaccurately expresses the scene can be eliminated based on the results of the uncertainty estimation, thereby improving the geometric quality and rendering quality of the three-dimensional Gaussian splash in new perspectives and sparse observation areas.

[0103] The embodiment of the present application is applied to the reconstruction of real scenes based on three-dimensional Gaussian splashing, which solves the problem of serious geometric defects in the reconstruction results of the three-dimensional Gaussian splashing method in the related art in areas where the perspective is not dense enough. The three-dimensional Gaussian splashing in the related art relies on dense training perspectives, and in areas with sparse perspectives, serious defects in the geometric structure occur due to the lack of multi-view constraints. Based on the three-dimensional Gaussian splashing in the related art, the embodiment of the present application further proposes uncertainty perception based on the Laplace approximation algorithm, which can analyze the geometric uncertainty of a set of optimized three-dimensional Gaussians, estimate the geometric quality of the scene block area based on the Laplace approximation algorithm, and has the ability to clean up floating objects and geometric defects obtained by erroneous reconstruction based on uncertainty values. Compared with the three-dimensional Gaussian splashing method in the related art, the embodiment of the present application solves the problem of insufficient three-dimensional reconstruction accuracy in areas with sparse perspectives, and has a significant improvement in ensuring the rendering quality of new perspectives and the reconstruction quality of sparse observation areas.

[0104] Furthermore, the embodiments of the present application can be applied to the field of autonomous driving. In the field of autonomous driving, high-precision three-dimensional environment reconstruction can help autonomous driving systems understand and predict the surrounding environment more accurately. The embodiments of the present application can reconstruct high-quality three-dimensional maps with fewer viewing angles, thereby providing more reliable data support for autonomous driving vehicles and further improving their navigation and obstacle avoidance capabilities. The embodiments of the present application can also be applied to the field of digital twins. The digital twin field mainly requires simulation analysis and predictive management by creating virtual copies of physical entities. The embodiments of the present application can provide more accurate and detailed data support for digital twin models by improving the accuracy and quality of three-dimensional reconstruction, especially when building digital models of complex industrial facilities or urban infrastructure, which can effectively improve the reliability and application value of the model. The embodiments of the present application can also be applied to the field of smart city construction. In the field of smart city construction, accurate three-dimensional city models are the key to achieving urban management and planning. The embodiments of the present application can achieve high-quality three-dimensional reconstruction in areas with uneven viewing angle coverage in the city, helping urban planners and managers to better perform tasks such as urban design, disaster management, and resource allocation.

[0105] Some embodiments of the present application also provide an image processing device corresponding to the image processing method provided in the aforementioned embodiments. The image processing device can be used to perform the aforementioned image processing method. For portions not fully described in the image processing device provided in this embodiment, reference can be made to the relevant description of the aforementioned image processing method. The beneficial effects of the image processing device provided in the embodiments of the present application can be referred to the relevant description of the image processing method provided in the embodiments of the present application.

[0106] like Figure 4 As shown, the image processing device provided in the embodiment of the present application includes an acquisition unit 401, a calculation unit 402 and a deletion unit 403.

[0107] An acquisition unit 401 is configured to acquire a three-dimensional Gaussian field for expressing a target scene;

[0108] A calculation unit 402 is configured to calculate, for each three-dimensional Gaussian function in the three-dimensional Gaussian field, an uncertainty of a target parameter in the three-dimensional Gaussian function;

[0109] The deleting unit 403 is configured to delete the three-dimensional Gaussian function whose uncertainty exceeds a preset threshold in the three-dimensional Gaussian field.

[0110] Optionally, the calculation unit 402 may be specifically configured to:

[0111] For each three-dimensional Gaussian function, based on the known target image, the target parameters are estimated a posteriori using the Laplace approximation algorithm to obtain the posterior probability distribution of the target parameters; the target image is obtained by photographing the target scene from the target perspective;

[0112] The uncertainty of the target parameters in the three-dimensional Gaussian function is calculated based on the posterior probability distribution of the target parameters.

[0113] Optionally, the calculation unit 402 may be further configured to:

[0114] Perform a reparameterization operation on the target parameters in the three-dimensional Gaussian function to obtain updated target parameters;

[0115] Replacing the target parameters in the three-dimensional Gaussian function with updated target parameters to obtain an updated three-dimensional Gaussian function;

[0116] Estimate the updated Hessian matrix of the three-dimensional Gaussian function based on the diagonal estimation algorithm;

[0117] The posterior probability distribution of the target parameters is determined based on the Hessian matrix of the updated three-dimensional Gaussian function.

[0118] Optionally, the calculation unit 402 may be further configured to:

[0119] Estimate the Hessian matrix of the updated three-dimensional Gaussian function based on the generalized Gauss-Newton matrix.

[0120] Optionally, the calculation unit 402 may be further configured to:

[0121] According to the updated Hessian matrix of the three-dimensional Gaussian function, the covariance matrix in the posterior probability distribution of the target parameter is determined to obtain a determined posterior probability distribution;

[0122] According to the posterior probability distribution of the target parameters, the uncertainty of the target parameters in the three-dimensional Gaussian function is calculated, including:

[0123] Based on the covariance matrix, the uncertainty is determined.

[0124] Optionally, the target parameter of the three-dimensional Gaussian function is the center point position of the three-dimensional Gaussian function; the calculation unit 402 may also be specifically configured to:

[0125] Determine the three-dimensional grid where the center point of the three-dimensional Gaussian function is located in the gridded space to obtain a target three-dimensional grid; the gridded space is obtained by dividing the three-dimensional space where the target scene is located into multiple three-dimensional grids of preset sizes, and each vertex in each three-dimensional grid is set with a corresponding three-dimensional perturbation amount;

[0126] An updated center point position is generated based on trilinear interpolation of the three-dimensional perturbation amount of each vertex in the target three-dimensional mesh and the center point position of the three-dimensional Gaussian function; the updated center point position is the center point position in the updated three-dimensional Gaussian function.

[0127] Optionally, the calculation unit 402 may be further configured to:

[0128] Estimate the posterior probability distribution of each vertex in the target three-dimensional mesh based on the updated Hessian matrix of the three-dimensional Gaussian function;

[0129] According to the posterior probability distribution of each vertex in the target three-dimensional grid, the posterior probability distribution of the center point position is calculated by trilinear interpolation.

[0130] Optionally, the acquiring unit 401 may be specifically configured to:

[0131] Obtaining a set of initial three-dimensional Gaussian functions; each initial three-dimensional Gaussian function includes multiple parameters, and the multiple parameters include a target parameter;

[0132] According to the target image, iterative training is performed on the parameters of the initial three-dimensional Gaussian function to obtain a three-dimensional Gaussian field.

[0133] Some embodiments of the present application further provide an electronic device corresponding to the image processing method provided in the aforementioned embodiment, so as to execute the aforementioned image processing method.

[0134] Figure 5 This is a hardware structure diagram of an electronic device according to an exemplary embodiment. The electronic device includes: a communication interface 501, a processor 502, a memory 503, and a bus 504. The communication interface 501, the processor 502, and the memory 503 communicate with each other via the bus 504. The processor 502 can execute the image processing method described above by reading and executing machine-executable instructions corresponding to the control logic of the image processing method in the memory 503. The details of the method are described in the above embodiments and will not be repeated here.

[0135] The memory 503 mentioned in this application can be any electronic, magnetic, optical or other physical storage device, and can contain stored information, such as executable instructions, data, etc. Specifically, the memory 503 can be RAM (Random Access Memory), flash memory, a storage drive (such as a hard disk drive), any type of storage disk (such as an optical disk, DVD, etc.), or a similar storage medium, or a combination thereof. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 501 (which can be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. can be used.

[0136] The bus 504 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory 503 is used to store programs, and the processor 502 executes the programs after receiving an execution instruction.

[0137] The processor 502 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by an integrated logic circuit of the hardware in the processor 502 or an instruction in the form of software. The above-mentioned processor 502 can be a general-purpose processor, including a network processor (NP), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc. The various methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly embodied as a hardware decoding processor for execution, or can be executed by a combination of hardware and software modules in the decoding processor.

[0138] The electronic device provided in the embodiment of the present application and the image processing method provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, operated or implemented by them.

[0139] An embodiment of the present application also provides a computer-readable storage medium corresponding to the image processing method provided in the aforementioned embodiment, on which a computer program (i.e., a program product) is stored. When the computer program is run by a processor, it will execute the image processing method provided in any of the aforementioned embodiments.

[0140] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

[0141] The computer-readable storage medium provided in the above-mentioned embodiments of the present application and the image processing method provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0142] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of the present application and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered merely as exemplary, and the true scope and spirit of the present application are indicated by the claims.

[0143] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0144] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

[0145] If the solutions in this application specification and examples involve the processing of personal information, they will be processed on the premise of having a legal basis (such as obtaining the consent of the personal information subject or being necessary for the performance of a contract) and will only be processed within the scope of regulations or agreements. The user's refusal to process personal information other than the necessary information for basic functions will not affect the user's use of basic functions.

Claims

1. An image processing method, characterized in that: The method comprises: Obtain a three-dimensional Gaussian field for expressing the target scene; For each three-dimensional Gaussian function in the three-dimensional Gaussian field, calculating the uncertainty of the target parameter in the three-dimensional Gaussian function; In the three-dimensional Gaussian field, the three-dimensional Gaussian function whose uncertainty exceeds a preset threshold is deleted.

2. The method according to claim 1, characterized in that The step of calculating the uncertainty of a target parameter in each three-dimensional Gaussian function in the three-dimensional Gaussian field includes: For each of the three-dimensional Gaussian functions, based on a known target image, a Laplace approximation algorithm is used to perform a posteriori estimation on the target parameter to obtain a posteriori probability distribution of the target parameter; the target image is obtained by photographing the target scene at a target viewing angle; The uncertainty of the target parameter in the three-dimensional Gaussian function is calculated according to the posterior probability distribution of the target parameter.

3. The method according to claim 2, characterized in that The method of performing a posteriori estimation on the target parameters based on the known target image using the Laplace approximation algorithm to obtain the posterior probability distribution of the target parameters includes: Performing a reparameterization operation on the target parameter in the three-dimensional Gaussian function to obtain an updated target parameter; Replacing the target parameter in the three-dimensional Gaussian function with the updated target parameter to obtain an updated three-dimensional Gaussian function; estimating a Hessian matrix of the updated three-dimensional Gaussian function based on a diagonal estimation algorithm; The posterior probability distribution of the target parameter is determined according to the Hessian matrix of the updated three-dimensional Gaussian function.

4. The method according to claim 3, characterized in that The estimating the Hessian matrix of the updated three-dimensional Gaussian function based on a diagonal estimation algorithm comprises: The Hessian matrix of the updated three-dimensional Gaussian function is estimated based on the generalized Gauss-Newton matrix.

5. The method according to claim 3, characterized in that: Determining the posterior probability distribution of the target parameter according to the Hessian matrix of the updated three-dimensional Gaussian function includes: Determine the covariance matrix in the posterior probability distribution of the target parameter according to the Hessian matrix of the updated three-dimensional Gaussian function, and obtain the determined posterior probability distribution; Calculating the uncertainty of the target parameter in the three-dimensional Gaussian function according to the posterior probability distribution of the target parameter includes: The uncertainty is determined based on the covariance matrix.

6. The method according to claim 3, characterized in that The target parameter of the three-dimensional Gaussian function is the center point position of the three-dimensional Gaussian function; performing a reparameterization operation on the target parameter in the three-dimensional Gaussian function to obtain an updated target parameter includes: Determine the three-dimensional grid where the center point of the three-dimensional Gaussian function is located in the gridded space to obtain a target three-dimensional grid; the gridded space is obtained by dividing the three-dimensional space where the target scene is located into a plurality of three-dimensional grids of a preset size, and each vertex in each of the three-dimensional grids is provided with a corresponding three-dimensional disturbance amount; An updated center point position is generated based on trilinear interpolation of the three-dimensional disturbance amount of each vertex in the target three-dimensional mesh and the center point position of the three-dimensional Gaussian function; the updated center point position is the center point position in the updated three-dimensional Gaussian function.

7. The method according to claim 6, characterized in that Determining the posterior probability distribution of the target parameter according to the Hessian matrix of the updated three-dimensional Gaussian function includes: Estimate the posterior probability distribution of each vertex in the target three-dimensional grid according to the Hessian matrix of the updated three-dimensional Gaussian function; According to the posterior probability distribution of each vertex in the target three-dimensional grid, the posterior probability distribution of the center point position is calculated by trilinear interpolation.

8. The method according to claim 2, characterized in that: The step of obtaining a three-dimensional Gaussian field for expressing a target scene includes: Acquire a set of initial three-dimensional Gaussian functions; each of the initial three-dimensional Gaussian functions includes a plurality of parameters, and the plurality of parameters include the target parameter; According to the target image, iterative training is performed on each parameter in the initial three-dimensional Gaussian function to obtain the three-dimensional Gaussian field.

9. An image processing device, characterized in that: The device comprises: An acquisition unit, used for acquiring a three-dimensional Gaussian field for expressing a target scene; A calculation unit, configured to calculate the uncertainty of a target parameter in each three-dimensional Gaussian function in the three-dimensional Gaussian field; A deleting unit is used to delete the three-dimensional Gaussian function whose uncertainty exceeds a preset threshold in the three-dimensional Gaussian field.

10. A computer program product, comprising a computer program, characterized in that The computer program is executed by a processor to implement the method according to any one of claims 1 to 8.

Images