Three-dimensional model optimization method and apparatus, and electronic device

Abstract

The invention relates to a three-dimensional model optimization method and device and electronic equipment, and the method comprises the steps: generating a plurality of three-dimensional Gaussian balls as an initial three-dimensional model; executing at least one parameter optimization operation for a plurality of three-dimensional Gaussian balls in the initial three-dimensional model to obtain an optimized three-dimensional model; the parameter optimization operation comprises the following steps: generating a two-dimensional image based on the plurality of three-dimensional Gaussian balls; and according to the difference between the two-dimensional image and the real image, adjusting parameters of the plurality of three-dimensional Gaussian balls so as to reduce the difference. According to the embodiment of the invention, the construction efficiency of the three-dimensional model is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a three-dimensional model optimization method and device, electronic equipment, and storage medium. Background Art

[0002] 3D reconstruction technology has made significant progress in the field of computer vision. Its core goal is to accurately restore the geometric form of a 3D scene from 2D images taken from multiple different angles. This technology integrates knowledge from multiple disciplines such as image processing, machine learning, and optimization algorithms, and has demonstrated its important application value in multiple fields such as virtual reality, augmented reality, and unmanned driving. The new perspective generation technology further simulates 2D images observed from a completely new perspective on a known 3D model or scene, providing a rich visual experience for industries such as 3D movies, games, and virtual reality.

[0003] In the related art, when constructing a three-dimensional model based on a two-dimensional image, it is necessary to provide accurate camera pose (position and attitude) information in advance, and then preliminarily estimate the initial three-dimensional model based on the camera pose, and then optimize the initial three-dimensional model.

[0004] However, obtaining camera pose information often requires a lot of calculations, which may take several hours to several days. This time-consuming process leads to low efficiency in building three-dimensional models. In addition, related technologies often fail to reconstruct camera poses, resulting in low practicality of three-dimensional model reconstruction technology. Summary of the invention

[0005] The present disclosure proposes a three-dimensional model optimization technical solution.

[0006] According to one aspect of the present disclosure, a three-dimensional model optimization method is provided, comprising:

[0007] Generate multiple three-dimensional Gaussian spheres as initial three-dimensional models;

[0008] For the multiple three-dimensional Gaussian balls in the initial three-dimensional model, perform at least one parameter optimization operation to obtain an optimized three-dimensional model;

[0009] The parameter optimization operation includes:

[0010] generating a two-dimensional image based on the plurality of three-dimensional Gaussian spheres;

[0011] According to the difference between the two-dimensional image and the real image, the parameters of the multiple three-dimensional Gaussian spheres are adjusted to reduce the difference.

[0012] In a possible implementation, generating a plurality of three-dimensional Gaussian spheres as an initial three-dimensional model includes:

[0013] Multiple three-dimensional Gaussian spheres are randomly generated as the initial three-dimensional model.

[0014] In a possible implementation, the generating of multiple three-dimensional Gaussian spheres in a random manner includes:

[0015] The parameters of a plurality of three-dimensional Gaussian spheres are randomly initialized, wherein the parameters include at least one of the following:

[0016] Position parameters, shape parameters, opacity parameters, and color parameters.

[0017] In a possible implementation, the generating of multiple three-dimensional Gaussian spheres in a random manner includes:

[0018] The position parameters are uniformly set within a unit sphere in the world coordinate system.

[0019] In a possible implementation manner, generating a two-dimensional image based on the multiple three-dimensional Gaussian spheres includes:

[0020] According to the camera posture, the three-dimensional Gaussian sphere is projected onto the image plane to obtain a two-dimensional Gaussian sphere;

[0021] The two-dimensional Gaussian sphere is rendered to obtain a two-dimensional image.

[0022] In a possible implementation, the method further includes:

[0023] Initialize the camera pose;

[0024] The parameter optimization operation further includes:

[0025] According to the difference between the two-dimensional image and the real image, the parameters of the camera pose are adjusted to reduce the difference.

[0026] In a possible implementation, the initializing the camera pose includes:

[0027] The position and posture of the camera are randomly set; or the position and posture of the camera are set to a preset position and posture.

[0028] In a possible implementation, adjusting the parameters of the plurality of three-dimensional Gaussian spheres according to the difference between the two-dimensional image and the real image to reduce the difference includes:

[0029] The loss function is obtained based on the difference between the two-dimensional image and the real image;

[0030] Based on the loss function, the parameters of the three-dimensional Gaussian sphere are adjusted by gradient descent to reduce the value of the loss function.

[0031] In a possible implementation manner, the parameter optimization operation further includes:

[0032] According to the difference between the two-dimensional image and the real image, the density distribution of the multiple three-dimensional Gaussian balls in space is adjusted.

[0033] According to one aspect of the present disclosure, there is provided a three-dimensional model optimization device, comprising:

[0034] A generation module, used for generating multiple three-dimensional Gaussian spheres as an initial three-dimensional model;

[0035] An optimization module, used for performing at least one parameter optimization operation on a plurality of three-dimensional Gaussian spheres in the initial three-dimensional model to obtain an optimized three-dimensional model;

[0036] The parameter optimization operation includes:

[0037] generating a two-dimensional image based on the plurality of three-dimensional Gaussian spheres;

[0038] According to the difference between the two-dimensional image and the real image, the parameters of the multiple three-dimensional Gaussian spheres are adjusted to reduce the difference.

[0039] In a possible implementation, the generation module is used to generate a plurality of three-dimensional Gaussian spheres in a random manner as an initial three-dimensional model.

[0040] In a possible implementation, the generating module is used to randomly initialize parameters of a plurality of three-dimensional Gaussian spheres, and the parameters include at least one of the following:

[0041] Position parameters, shape parameters, opacity parameters, and color parameters.

[0042] In a possible implementation, the generating module is used to uniformly set the position parameters within a unit sphere in a world coordinate system.

[0043] In a possible implementation manner, generating a two-dimensional image based on the multiple three-dimensional Gaussian spheres includes:

[0044] According to the camera posture, the three-dimensional Gaussian sphere is projected onto the image plane to obtain a two-dimensional Gaussian sphere;

[0045] The two-dimensional Gaussian sphere is rendered to obtain a two-dimensional image.

[0046] In a possible implementation manner, the device further includes:

[0047] Initialization module, used to initialize the camera pose;

[0048] The parameter optimization operation further includes:

[0049] According to the difference between the two-dimensional image and the real image, the parameters of the camera pose are adjusted to reduce the difference.

[0050] In a possible implementation, the initialization module is used to:

[0051] The position and posture of the camera are randomly set; or the position and posture of the camera are set to a preset position and posture.

[0052] In a possible implementation, adjusting the parameters of the plurality of three-dimensional Gaussian spheres according to the difference between the two-dimensional image and the real image to reduce the difference includes:

[0053] The loss function is obtained based on the difference between the two-dimensional image and the real image;

[0054] Based on the loss function, the parameters of the three-dimensional Gaussian sphere are adjusted by gradient descent to reduce the value of the loss function.

[0055] In a possible implementation manner, the parameter optimization operation further includes:

[0056] According to the difference between the two-dimensional image and the real image, the density distribution of the multiple three-dimensional Gaussian balls in space is adjusted.

[0057] According to one aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to call the instructions stored in the memory to execute the above method.

[0058] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor.

[0059] In the disclosed embodiment, multiple three-dimensional Gaussian spheres are generated as the initial three-dimensional model; at least one parameter optimization operation is performed on the multiple three-dimensional Gaussian spheres in the initial three-dimensional model to obtain an optimized three-dimensional model, and the parameter optimization operation includes: generating a two-dimensional image based on the multiple three-dimensional Gaussian spheres; adjusting the parameters of the multiple three-dimensional Gaussian spheres according to the difference between the two-dimensional image and the real image to reduce the difference. Since the initial three-dimensional model is a generated three-dimensional Gaussian sphere, there is no need to provide an accurate camera pose in advance to estimate the initial three-dimensional model. Instead, the two-dimensional image is generated based on the multiple three-dimensional Gaussian spheres, and the parameters of the multiple three-dimensional Gaussian spheres are adjusted according to the difference between the two-dimensional image and the real image, thereby optimizing the three-dimensional model. As a result, the construction efficiency of the three-dimensional model is improved, and there is no failure in the reconstruction of the camera pose, and the practicality is high.

[0060] It should be understood that the above general description and the following detailed description are exemplary and explanatory only and do not limit the present disclosure. Other features and aspects of the present disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used to illustrate the technical solutions of the present disclosure together with the specification.

[0062] Figure 1 A flowchart of a three-dimensional model optimization method according to an embodiment of the present disclosure is shown.

[0063] Figure 2 A flowchart of a three-dimensional model optimization process according to an embodiment of the present disclosure is shown.

[0064] Figure 3 A block diagram of a three-dimensional model optimization device according to an embodiment of the present disclosure is shown.

[0065] Figure 4 A block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0066] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0067] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0068] The term "and / or" herein is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can represent including any one or more elements selected from the set consisting of A, B, and C.

[0069] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0070] Currently, camera pose information is usually obtained through incremental Structure from Motion (SfM) technology. However, there are some problems with this method. First, SfM is a computationally intensive process that usually takes hours to days to complete, which greatly limits the efficiency of 3D model construction. Second, SfM technology may fail in some cases, such as when there is insufficient texture in the scene or the camera motion is too complex, which will make it impossible to obtain accurate camera pose information, resulting in low practicality of 3D model reconstruction technology.

[0071] Based on this, the present disclosure provides a three-dimensional model optimization method, which generates multiple three-dimensional Gaussian balls as the initial three-dimensional model; performs at least one parameter optimization operation on the multiple three-dimensional Gaussian balls in the initial three-dimensional model to obtain an optimized three-dimensional model, and the parameter optimization operation includes: generating a two-dimensional image based on the multiple three-dimensional Gaussian balls; adjusting the parameters of the multiple three-dimensional Gaussian balls according to the difference between the two-dimensional image and the real image to reduce the difference. Since the initial three-dimensional model is a generated three-dimensional Gaussian ball, it is not necessary to provide an accurate camera pose in advance to estimate the initial three-dimensional model. Instead, the two-dimensional image is generated based on the multiple three-dimensional Gaussian balls, and the parameters of the multiple three-dimensional Gaussian balls are adjusted according to the difference between the two-dimensional image and the real image, thereby optimizing the three-dimensional model. As a result, the construction efficiency of the three-dimensional model is improved, and there is no failure in the reconstruction of the camera pose, and the practicality is high.

[0072] Figure 1 A flowchart of a three-dimensional model optimization method according to an embodiment of the present disclosure is shown. Figure 1 As shown, the method includes:

[0073] In step S11, a plurality of three-dimensional Gaussian spheres are generated as an initial three-dimensional model.

[0074] A three-dimensional Gaussian sphere can be a spherical region in a three-dimensional space. A Gaussian sphere can represent a three-dimensional object with spherical symmetry, and an ellipsoid set composed of multiple three-dimensional Gaussian spheres can be used to represent a three-dimensional model.

[0075] In a possible implementation, generating multiple three-dimensional Gaussian spheres as the initial three-dimensional model includes: generating multiple three-dimensional Gaussian spheres as the initial three-dimensional model in a random manner. The shape of the three-dimensional model can be adjusted by adjusting parameters such as the position, size, and color of each sphere in the ellipsoid set.

[0076] When generating multiple three-dimensional Gaussian balls in a random manner, the parameters of the multiple three-dimensional Gaussian balls can be randomly initialized, and the parameters of the three-dimensional Gaussian balls include at least one of the following: position parameter, shape parameter, opacity parameter and color parameter. When constructing a geometric model of a three-dimensional Gaussian ball, the position parameter is the center point coordinate x of the three-dimensional Gaussian ball, the shape parameter is the three-dimensional covariance matrix ∑, the color parameter is the spherical harmonic function c, and the opacity parameter is the optimizable parameter α.

[0077] When the parameters of multiple three-dimensional Gaussian spheres are randomly initialized, the parameters of the three-dimensional Gaussian spheres can be randomly distributed and generated. Since the initial three-dimensional model is a randomly generated three-dimensional Gaussian sphere, there is no need to provide accurate camera poses in advance to estimate the initial three-dimensional model. In addition, Gaussian spheres can also be generated according to pre-set rules, for example, parameters of Gaussian spheres that meet certain rules are generated, or Gaussian spheres that are completely identical are generated.

[0078] It should be noted that, in some cases, the three-dimensional Gaussian sphere can also be ellipsoidal. The covariance matrix of the ellipsoid is non-diagonal, and the lengths of its major and minor axes are determined by the eigenvalues ​​of the covariance matrix, and the direction is determined by the eigenvectors.

[0079] In step S12, at least one parameter optimization operation is performed on the multiple three-dimensional Gaussian spheres in the initial three-dimensional model to obtain an optimized three-dimensional model.

[0080] The parameter optimization operation includes: generating a two-dimensional image based on the multiple three-dimensional Gaussian spheres; and adjusting the parameters of the multiple three-dimensional Gaussian spheres according to the difference between the two-dimensional image and the real image to reduce the difference.

[0081] For the initial three-dimensional model, a two-dimensional image can be generated based on the generated multiple three-dimensional Gaussian spheres. Specifically, the three-dimensional Gaussian sphere can be projected onto the image plane through the camera posture to obtain a two-dimensional Gaussian sphere, and then the two-dimensional Gaussian sphere is rendered to obtain a two-dimensional image. The two-dimensional image can be regarded as obtained by photographing the three-dimensional Gaussian sphere. By taking the real image obtained by the real photograph as a reference, calculating the difference between the two-dimensional image and the real image, and adjusting the parameters of the three-dimensional Gaussian sphere with the goal of minimizing the difference, the three-dimensional model can be optimized to be optimized to a shape close to the real object in the real image. The optimization operation can be iteratively performed. Its specific implementation can refer to the possible implementation methods provided in the present disclosure, which will not be elaborated here.

[0082] Since the initial three-dimensional model is a generated three-dimensional Gaussian sphere, there is no need to provide an accurate camera pose in advance to estimate the initial three-dimensional model. Instead, a two-dimensional image is generated based on multiple three-dimensional Gaussian spheres, and the parameters of the multiple three-dimensional Gaussian spheres are adjusted according to the difference between the two-dimensional image and the real image. This improves the efficiency of building the three-dimensional model, and there is no failure in camera pose reconstruction, so the three-dimensional model is optimized.

[0083] In a possible implementation, generating a plurality of three-dimensional Gaussian spheres in a random manner includes: uniformly setting the position parameters within a unit sphere in a world coordinate system.

[0084] When the parameters of multiple three-dimensional Gaussian spheres are randomly initialized, the parameters of the three-dimensional Gaussian spheres can be randomly distributed and generated. The randomly generated three-dimensional Gaussian spheres here can be randomly generated for at least one parameter of the three-dimensional Gaussian spheres. Other parameters other than the randomly generated parameters can be generated according to a predetermined rule.

[0085] In one example, the position parameters may be uniformly set within a unit sphere in the world coordinate system to uniformly distribute the multiple three-dimensional Gaussian spheres. Uniformly distributed Gaussian spheres are suitable for situations where there are no obvious structures or features in the scene.

[0086] In order to facilitate a clearer understanding of the process of randomly generating multiple three-dimensional Gaussian spheres, the parameters of the three-dimensional Gaussian spheres are explained below through specific mathematical expressions. It should be noted that the specific mathematical expressions provided in the present disclosure are a possible implementation method of the embodiments of the present disclosure during specific implementation, and should not be understood as limiting the scope of protection of the embodiments of the present disclosure.

[0087] The expression of the three-dimensional Gaussian sphere G(x) can be found in formula (1).

[0088]

[0089] Where x represents the position of the three-dimensional Gaussian sphere in the world coordinate system, which is randomly generated uniformly within the unit sphere in the world coordinate system, e represents a natural constant, and T represents transpose. Σ represents the covariance matrix. The specific expression of the covariance matrix can be found in formula (2).

[0090] Σ=RSS T R T (2)

[0091] R is the rotation matrix and S is the scaling matrix.

[0092] In a possible implementation, the pre-set rule may be an image feature rule, and then a three-dimensional Gaussian sphere may be generated according to the image feature rule. In one example, key points may be extracted from an input image, and a three-dimensional Gaussian sphere may be generated at the three-dimensional spatial position corresponding to the key points, and a three-dimensional Gaussian sphere may be randomly generated at the three-dimensional spatial position other than the key points; in another example, the input image may be segmented, and then a Gaussian sphere may be randomly generated within each segmented area to capture the structure of different objects. By using the initial model generated by the image feature rule, the objects and details in the scene may be more accurately reflected, and the information provided by the image features may help the optimization process to more accurately adjust the parameters of the Gaussian sphere, thereby improving the reconstruction accuracy.

[0093] In a possible implementation, the pre-set rule may be the scene category of the image, and then a three-dimensional Gaussian sphere may be generated according to the scene category of the image. Specifically, the scene category of the image input by the user (such as indoor, outdoor, natural scenery, etc.) may be received, and then the position and shape of the Gaussian sphere may be generated according to the preset parameters of the three-dimensional Gaussian sphere corresponding to the category. By pre-establishing the association between the scene category and the parameters of the three-dimensional Gaussian sphere, the parameters of the three-dimensional Gaussian sphere that conforms to the scene category are pre-set for different scenes. Thus, by utilizing the prior knowledge of the scene category, an initial three-dimensional model that is more in line with the characteristics of the category may be generated, so that the initial three-dimensional model is closer to the real scene, and the reconstruction efficiency and accuracy are improved.

[0094] The pre-set rules can be selected and adjusted according to specific scenarios and needs, and this disclosure does not limit this.

[0095] In a possible implementation, generating a two-dimensional image based on the multiple three-dimensional Gaussian spheres includes: projecting the three-dimensional Gaussian spheres onto an image plane according to a camera posture to obtain a two-dimensional Gaussian sphere; and rendering the two-dimensional Gaussian spheres to obtain a two-dimensional image.

[0096] In 3D computer vision and robotics, the camera pose is the position and orientation of the camera in 3D space, usually consisting of a position vector (indicating the coordinates of the camera in 3D space) and a quaternion matrix (indicating the orientation of the camera, i.e. how it is oriented relative to the world coordinate system or a reference coordinate system). The position vector is usually a 3D vector, indicating the coordinate position of the camera in the world coordinate system; the quaternion matrix is ​​used to describe the rotation information of the camera, which describes how the camera coordinate system rotates relative to the world coordinate system or the reference coordinate system.

[0097] When the three-dimensional Gaussian sphere is projected onto the image plane according to the camera position, the position x of the Gaussian point in the world coordinate system can be multiplied by the projection transformation Σ′. The mathematical expression of Σ′ can be found in formula (3).

[0098] Σ′=JWΣW T J T (3)

[0099] Among them, J is the Jacobian matrix of the projection transformation, which reflects the intrinsic parameter information of the camera. The intrinsic parameter can be obtained according to the camera that takes the image, for example, it can be obtained according to the calibration calculation or the camera model; W is the pose information, which can be an optimizable parameter. The pose information is described in detail below.

[0100] In a possible implementation, the method further includes: initializing a camera pose; and the parameter optimization operation further includes: adjusting parameters of the camera pose according to a difference between the two-dimensional image and a real image to reduce the difference.

[0101] When initializing the camera pose, the camera pose may be randomly set; or the camera pose may be set to a preset pose.

[0102] The mathematical expression of the initialized posture information W can be found in formula (4).

[0103]

[0104] Among them, I represents the identity matrix and 0 represents the zero matrix.

[0105] The randomly set camera pose can be any position and angle, or the camera can be set at the origin of the coordinate system and facing the direction of a certain coordinate axis. Then, parameter optimization is performed iteratively to optimize the camera pose. The specific optimization process can be to calculate the loss function between the two-dimensional image and the real image, and then perform gradient backpropagation to adjust the camera pose. The loss function can be found in formula (6), which is not repeated here. In the embodiment of the present disclosure, by jointly optimizing the parameters of the three-dimensional Gaussian sphere and the pose of the camera, the quality of the constructed three-dimensional model and the accuracy of the pose are improved, so that the optimization method of the three-dimensional model is more robust, less prone to optimization failure, and more practical.

[0106] When performing the first round of parameter optimization, the 3D Gaussian sphere can be projected onto the image plane using the initialized pose to obtain a 2D Gaussian sphere. When performing parameter optimization again, the 3D Gaussian sphere can be projected onto the image plane using the pose after the previous round of optimization to obtain a 2D Gaussian sphere. This projection operation projects the 3D Gaussian sphere that affects the input image into a 2D Gaussian sphere, that is, the 3D Gaussian sphere within the camera's view is projected onto the image plane to obtain a 2D Gaussian sphere.

[0107] After projecting the 3D Gaussian sphere onto the 2D image plane, the 3D Gaussian sphere can be assigned to specific image pixels through differentiable rasterization technology.

[0108] When assigning pixels to 3D Gaussian spheres, it is necessary to consider the possibility that multiple 3D Gaussian spheres may be projected onto the same pixel. In this case, these Gaussian spheres need to be depth-sorted to determine which Gaussian sphere contributes most to the pixel. Depth sorting is usually based on the distance from the center point of the Gaussian sphere to the camera. The closer the Gaussian sphere is, the greater its contribution to the pixel.

[0109] After the depth sorting, the pixel value (RGB value) of each pixel can be calculated according to the parameters of the three-dimensional Gaussian sphere, the lighting conditions, and the camera parameters to obtain the pixel values ​​of the two-dimensional image.

[0110] The process of determining the pixel value C can be seen in formula (5).

[0111]

[0112] Among them, c i represents the color parameter of the i-th three-dimensional Gaussian ball learned in the previous iteration, α i Indicates the opacity parameter of the i-th three-dimensional Gaussian sphere learned in the previous iteration, α j Represents the opacity parameter of the j-th three-dimensional Gaussian sphere learned in the previous round. Both will be updated in each iteration. N is a positive integer.

[0113] The predicted pixel values ​​can form a two-dimensional image, which can be regarded as an image obtained by photographing the three-dimensional model. Then, according to the difference between the two-dimensional image and the real image, the parameters of multiple three-dimensional Gaussian balls are adjusted to reduce the difference, so that the three-dimensional model can be closer to the real object corresponding to the real image.

[0114] In one possible implementation, according to the difference between the two-dimensional image and the real image, the parameters of the multiple three-dimensional Gaussian spheres are adjusted to reduce the difference, including: obtaining a loss function according to the difference between the two-dimensional image and the real image; based on the loss function, adjusting the parameters of the three-dimensional Gaussian spheres by gradient descent to reduce the value of the loss function.

[0115] When determining the difference between the two-dimensional image and the real image, a loss function may be used to calculate the loss between the two-dimensional image and the real image. In one example, the loss function may be referred to in formula (6).

[0116]

[0117] in, Represents the real pixel RGB value, is the first-order moment, is the structural similarity loss, and λ is an adjustable weight parameter.

[0118] After determining the loss between the two-dimensional image and the real image based on the loss function, the gradient can be backpropagated through gradient descent to optimize the parameters of the three-dimensional Gaussian sphere and the camera pose.

[0119] In addition, the loss can be used to optimize the density distribution of the three-dimensional Gaussian sphere in space. In a possible implementation, after optimizing the three-dimensional model, the method further includes: adjusting the density distribution of the multiple three-dimensional Gaussian spheres in space according to the difference between the two-dimensional image and the real image.

[0120] Adjusting the density of the three-dimensional Gaussian balls can be done by increasing or decreasing the number of Gaussian balls in each part of the space. If the two-dimensional image is too blurry or not detailed enough in some areas relative to the real image, the detail level of the three-dimensional model in these areas can be improved by increasing the number of Gaussian balls corresponding to the area; conversely, if the Gaussian balls in some areas are too dense, resulting in a too sharp or unnatural rendering result, these areas can be smoothed by reducing the number of Gaussian balls corresponding to the area. Therefore, by adjusting the density of the three-dimensional Gaussian balls, the accuracy of the optimized three-dimensional model can be improved.

[0121] An application scenario of the embodiment of the present disclosure is described below. Figure 2 As shown, Figure 2 A flow chart of the three-dimensional model optimization process according to an embodiment of the present disclosure is shown. In this application scenario, a three-dimensional Gaussian sphere is generated uniformly within the unit sphere in the world coordinate system, and the three-dimensional Gaussian sphere is processed in two aspects: the operation direction and the gradient direction. In the operation direction, the three-dimensional Gaussian sphere is projected onto the image plane using the initialized camera pose and camera parameters (rotation and translation matrices) to obtain a two-dimensional Gaussian distribution; then the two-dimensional Gaussian distribution is rendered into a two-dimensional image using differentiable rasterization technology; in the gradient direction, the rendered image is compared with the input real image, and the difference between them, i.e., the loss function, is calculated. The parameters of the three-dimensional Gaussian sphere are updated according to the loss function, and the spatial density of the three-dimensional Gaussian sphere is dynamically adjusted to achieve the optimization of the three-dimensional model.

[0122] It can be understood that the above-mentioned various method embodiments mentioned in the present disclosure can be combined with each other to form a combined embodiment without violating the principle logic. Due to space limitations, the present disclosure will not repeat them. It can be understood by those skilled in the art that in the above-mentioned method of the specific implementation method, the specific execution order of each step should be determined according to its function and possible internal logic.

[0123] In addition, the present disclosure also provides a three-dimensional model optimization device, an electronic device, a computer-readable storage medium, and a program, all of which can be used to implement any three-dimensional model optimization method provided by the present disclosure. The corresponding technical solutions and descriptions are referred to the corresponding records in the method part and will not be repeated here.

[0124] Figure 3 A block diagram of a three-dimensional model optimization device according to an embodiment of the present disclosure is shown as follows: Figure 3 As shown, the device 20 comprises:

[0125] A generating module 21, used for generating a plurality of three-dimensional Gaussian spheres as an initial three-dimensional model;

[0126] An optimization module 22, configured to perform at least one parameter optimization operation on a plurality of three-dimensional Gaussian spheres in the initial three-dimensional model to obtain an optimized three-dimensional model;

[0127] The parameter optimization operation includes:

[0128] generating a two-dimensional image based on the plurality of three-dimensional Gaussian spheres;

[0129] According to the difference between the two-dimensional image and the real image, the parameters of the multiple three-dimensional Gaussian spheres are adjusted to reduce the difference.

[0130] In a possible implementation, the generation module is used to generate a plurality of three-dimensional Gaussian spheres in a random manner as an initial three-dimensional model.

[0131] In a possible implementation, the generating module is used to randomly initialize parameters of a plurality of three-dimensional Gaussian spheres, and the parameters include at least one of the following:

[0132] Position parameters, shape parameters, opacity parameters, and color parameters.

[0133] In a possible implementation, the generating module is used to uniformly set the position parameters within a unit sphere in a world coordinate system.

[0134] In a possible implementation manner, generating a two-dimensional image based on the multiple three-dimensional Gaussian spheres includes:

[0135] According to the camera posture, the three-dimensional Gaussian sphere is projected onto the image plane to obtain a two-dimensional Gaussian sphere;

[0136] The two-dimensional Gaussian sphere is rendered to obtain a two-dimensional image.

[0137] In a possible implementation manner, the device further includes:

[0138] Initialization module, used to initialize the camera pose;

[0139] The parameter optimization operation further includes:

[0140] According to the difference between the two-dimensional image and the real image, the parameters of the camera pose are adjusted to reduce the difference.

[0141] In a possible implementation, the initialization module is used to:

[0142] The position and posture of the camera are randomly set; or the position and posture of the camera are set to a preset position and posture.

[0143] In a possible implementation, adjusting the parameters of the plurality of three-dimensional Gaussian spheres according to the difference between the two-dimensional image and the real image to reduce the difference includes:

[0144] The loss function is obtained based on the difference between the two-dimensional image and the real image;

[0145] Based on the loss function, the parameters of the three-dimensional Gaussian sphere are adjusted by gradient descent to reduce the value of the loss function.

[0146] In a possible implementation manner, the parameter optimization operation further includes:

[0147] According to the difference between the two-dimensional image and the real image, the density distribution of the multiple three-dimensional Gaussian balls in space is adjusted.

[0148] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0149] The embodiment of the present disclosure also provides a computer-readable storage medium on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor. The computer-readable storage medium can be a volatile or non-volatile computer-readable storage medium.

[0150] An embodiment of the present disclosure further proposes an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to call the instructions stored in the memory to execute the above method.

[0151] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0152] The electronic device may be provided as a terminal, a server, or a device in other forms.

[0153] Figure 4 1 is a block diagram of an electronic device 1900 according to an embodiment of the present disclosure. For example, the electronic device 1900 may be provided as a server or a terminal device. Figure 4 , the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0154] The electronic device 1900 may also include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output (I / O) interface 1958. The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as a Microsoft Server operating system (Windows Server 2003). TM ), a graphical user interface operating system launched by Apple (Mac OSX TM ), a multi-user, multi-process computer operating system (Unix TM ), a free and open source Unix-like operating system (Linux TM ), an open source Unix-like operating system (FreeBSD TM ) or similar.

[0155] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.

[0156] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0157] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, (but not limited to) an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0158] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0159] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0160] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0161] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0162] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0163] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0164] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK) and the like.

[0165] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0166] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0167] If the technical solution of this application involves personal information, the product using the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using the technical solution of this application has obtained the individual's separate consent before processing the sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, clear and prominent signs are set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that he or she agrees to the collection of his or her personal information; or on the device that processes personal information, the personal information processing rules are notified by obvious signs / information, and the individual's authorization is obtained through pop-up information or by asking the individual to upload his or her personal information; among them, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

[0168] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A three-dimensional model optimization method, characterized in that: include: Generate multiple three-dimensional Gaussian spheres as initial three-dimensional models; For the multiple three-dimensional Gaussian balls in the initial three-dimensional model, perform at least one parameter optimization operation to obtain an optimized three-dimensional model; The parameter optimization operation includes: generating a two-dimensional image based on the plurality of three-dimensional Gaussian spheres; According to the difference between the two-dimensional image and the real image, the parameters of the multiple three-dimensional Gaussian spheres are adjusted to reduce the difference.

2. The method according to claim 1, characterized in that The generating of a plurality of three-dimensional Gaussian spheres as an initial three-dimensional model includes: Multiple three-dimensional Gaussian spheres are randomly generated as the initial three-dimensional model.

3. The method according to claim 2, characterized in that The method of generating a plurality of three-dimensional Gaussian spheres in a random manner comprises: The parameters of a plurality of three-dimensional Gaussian spheres are randomly initialized, wherein the parameters include at least one of the following: Position parameters, shape parameters, opacity parameters, and color parameters.

4. The method according to claim 2, characterized in that: The method of generating a plurality of three-dimensional Gaussian spheres in a random manner comprises: The position parameters are uniformly set within a unit sphere in the world coordinate system.

5. The method according to claim 1, characterized in that The generating a two-dimensional image based on the multiple three-dimensional Gaussian spheres comprises: According to the camera posture, the three-dimensional Gaussian sphere is projected onto the image plane to obtain a two-dimensional Gaussian sphere; The two-dimensional Gaussian sphere is rendered to obtain a two-dimensional image.

6. The method according to claim 5, characterized in that The method further comprises: Initialize the camera pose; The parameter optimization operation further includes: According to the difference between the two-dimensional image and the real image, the parameters of the camera pose are adjusted to reduce the difference.

7. The method according to claim 6, characterized in that The initialization of the camera pose includes: The position and posture of the camera are randomly set; or the position and posture of the camera are set to a preset position and posture.

8. The method according to claim 1, characterized in that The adjusting the parameters of the plurality of three-dimensional Gaussian spheres according to the difference between the two-dimensional image and the real image to reduce the difference comprises: The loss function is obtained based on the difference between the two-dimensional image and the real image; Based on the loss function, the parameters of the three-dimensional Gaussian sphere are adjusted by gradient descent to reduce the value of the loss function.

9. The method according to claim 1, characterized in that: The parameter optimization operation also includes: According to the difference between the two-dimensional image and the real image, the density distribution of the multiple three-dimensional Gaussian balls in space is adjusted.

10. A three-dimensional model optimization device, characterized in that: include: A generation module, used for generating multiple three-dimensional Gaussian spheres as an initial three-dimensional model; An optimization module, used for performing at least one parameter optimization operation on a plurality of three-dimensional Gaussian spheres in the initial three-dimensional model to obtain an optimized three-dimensional model; The parameter optimization operation includes: generating a two-dimensional image based on the plurality of three-dimensional Gaussian spheres; According to the difference between the two-dimensional image and the real image, the parameters of the multiple three-dimensional Gaussian spheres are adjusted to reduce the difference.

11. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method described in any one of claims 1 to 9.

12. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images