A depth map determination method and apparatus, electronic device, and storage medium

Abstract

This application provides a depth map determination method, apparatus, electronic device, and storage medium. The method includes: for each target ellipsoid, determining nearest neighbor ellipsoids in a first 3D model whose distance to the target ellipsoid is less than a preset distance threshold; determining multiple position vectors originating from the target ellipsoid and ending at each of its nearest neighbor ellipsoids; determining the covariance matrix corresponding to each target ellipsoid; based on the covariance matrix, determining two planar vectors corresponding to the target ellipsoid; determining the shape of the surface of the target scene at the target ellipsoid based on the plane formed by the two planar vectors; performing 3D reconstruction of the target scene based on the surface shape at each target ellipsoid to obtain a second 3D model of the target scene; and rendering the second 3D model of the target scene to obtain a smoothed depth map of the target scene. This can improve the accuracy of the 3D model depth map.

Description

Technical Field

[0001] This application relates to the field of computer graphics technology, and in particular to a method, apparatus, electronic device and storage medium for determining depth maps. Background Technology

[0002] Currently, 3D model reconstruction of scenes is typically achieved using 3DGS (3D Gaussian Splatting) technology. 3DGS renders an image containing a scene using ellipsoids. By continuously adjusting the position, number, and shape of each ellipsoid in the image, the point cloud formed by combining these ellipsoids more closely approximates a freeform shape in the image, thus reconstructing the 3D model of the scene. However, because the smallest rendering unit in 3DGS is an ellipsoid, the depth information of the entire ellipsoid is identical. Therefore, when projecting an ellipsoid onto a plane to obtain an ellipse, the depth information of the entire ellipse is also identical. Furthermore, when projecting the 3D model reconstructed from 3DGS to obtain an image of the scene, multiple ellipsoids are projected onto the same plane, resulting in an image containing multiple ellipses. While the depth value of a particular ellipse in the image may be the same, the actual depth values ​​of the pixels corresponding to the ellipse's position in the image may differ. Therefore, the depth map obtained from the 3D model reconstructed using 3DGS has relatively low accuracy. Summary of the Invention

[0003] The purpose of this application is to provide a depth map determination method, apparatus, electronic device, and storage medium to improve the accuracy of depth maps for 3D models. The specific technical solution is as follows:

[0004] This application provides a depth map determination method, the method comprising:

[0005] Obtain a first 3D model of the target scene based on 3DGS, wherein the first 3D model contains multiple ellipsoids;

[0006] For each target ellipsoid, other ellipsoids in the first three-dimensional model whose distance from the target ellipsoid is less than a preset distance threshold are determined as nearest neighbor ellipsoids. The target ellipsoid can be any number of ellipsoids in the first three-dimensional model.

[0007] For each target ellipsoid, a vector is determined that starts from the target ellipsoid and ends at each of its nearest neighbor ellipsoids, thus obtaining multiple position vectors corresponding to the target ellipsoid.

[0008] The covariance matrix corresponding to each target ellipsoid is determined based on the position vector corresponding to each target ellipsoid. The value of the i-th row and j-th column of the covariance matrix is ​​the covariance between the i-th and j-th dimensions of the position vector corresponding to each target ellipsoid.

[0009] For each target ellipsoid, based on the covariance matrix corresponding to the target ellipsoid, determine the eigenvectors corresponding to two target eigenvalues ​​as two planar vectors corresponding to the target ellipsoid. The two target eigenvalues ​​are the two eigenvalues ​​that are the first two in descending order of the eigenvalues ​​of the covariance matrix corresponding to the target ellipsoid.

[0010] For each target ellipsoid, the shape of the surface of the target scene at the target ellipsoid is determined based on the plane formed by the two plane vectors corresponding to the target ellipsoid.

[0011] Based on the shape of the surface of the target scene at each of the target ellipsoids, the target scene is reconstructed in three dimensions to obtain a second three-dimensional model of the target scene;

[0012] The second 3D model of the target scene is rendered to obtain a smooth depth map of the target scene.

[0013] In one possible embodiment, determining the shape of the surface of the target scene at the target ellipsoid based on the plane formed by the two planar vectors corresponding to the target ellipsoid includes:

[0014] The normal vector of the plane formed by the two plane vectors corresponding to the target ellipsoid is taken as the normal vector of the target ellipsoid; the normal vector is used to characterize the shape of the surface of the target scene at the target ellipsoid.

[0015] In one possible embodiment, the step of performing three-dimensional reconstruction of the target scene based on the shape of the surface of the target scene at each of the target ellipsoids to obtain a second three-dimensional model of the target scene includes:

[0016] Calculate the vector field based on the normal vector of each target ellipsoid and the position of each target ellipsoid;

[0017] Construct and solve the Poisson equation to determine the indicator function corresponding to the vector field;

[0018] Based on the indicator function corresponding to the vector field, the isosurfaces where each target ellipsoid is located are determined; the values ​​of the indicator functions corresponding to each point on the isosurfaces are equal;

[0019] Based on the isosurfaces where each target ellipsoid is located, a second three-dimensional model of the target scene is constructed.

[0020] In one possible embodiment, rendering the second 3D model of the target scene to obtain a smooth depth map of the target scene includes:

[0021] The second 3D model of the target scene is rendered to obtain the original depth map of the target scene;

[0022] The original depth map is filtered to obtain a smooth depth map of the target scene.

[0023] In one possible embodiment, the target scene is a pre-defined large-area scene; the method further includes:

[0024] Obtain the target color map rendered based on the first 3D model;

[0025] Obtain other depth maps and other color maps of other scenes rendered using non-3DGS technology; the other scenes are preset small-area scenes, and the smooth depth map, the target color map, the other depth maps, and the other color maps are rendered from the same angle;

[0026] The target color map and the other color maps are fused to obtain a fused image through the following methods:

[0027] For each pixel, if the depth value of the pixel in the smooth depth map is greater than the depth value of the pixel in other depth maps, then the pixel value of the pixel in the other color maps is set as the pixel value of the pixel in the blended image; if the depth value of the pixel in the smooth depth map is less than the depth value of the pixel in other depth maps, then the pixel value of the pixel in the target color map is set as the pixel value of the pixel in the blended image.

[0028] In one possible embodiment, determining the vectors originating from the target ellipsoid and ending at each of its nearest neighbor ellipsoids to obtain multiple position vectors corresponding to the target ellipsoid includes:

[0029] By determining vectors that start from the center point of the target ellipsoid and end at the center points of each of the target ellipsoid's nearest neighbor ellipsoids, multiple position vectors corresponding to the target ellipsoid are obtained.

[0030] The step of determining other ellipsoids in the first three-dimensional model whose distance from the target ellipsoid is less than a preset distance threshold as neighboring ellipsoids includes:

[0031] In the first three-dimensional model, the nearest neighbor search algorithm KDTree is used to determine other ellipsoids corresponding to the center point whose distance from the center point of the target ellipsoid is less than a preset distance threshold, and these are taken as the nearest neighbor ellipsoids.

[0032] This application embodiment also provides a depth map determination device, the device comprising:

[0033] The first 3D model acquisition module is used to acquire the first 3D model of the target scene based on 3DGS, and the first 3D model contains multiple ellipsoids.

[0034] The nearest neighbor ellipsoid determination module is used to determine, for each target ellipsoid, other ellipsoids in the first three-dimensional model whose distance from the target ellipsoid is less than a preset distance threshold, as nearest neighbor ellipsoids. The target ellipsoid can be any number of ellipsoids in the first three-dimensional model.

[0035] The position vector determination module is used to determine, for each target ellipsoid, a vector starting from the target ellipsoid and ending at each of its nearest neighbor ellipsoids, thereby obtaining multiple position vectors corresponding to the target ellipsoid.

[0036] The covariance matrix determination module is used to determine the covariance matrix corresponding to each target ellipsoid based on the position vector corresponding to each target ellipsoid. The value of the i-th row and j-th column of the covariance matrix is ​​the covariance between the i-th and j-th dimensions of the position vector corresponding to each target ellipsoid.

[0037] The planar vector determination module is used to determine, for each target ellipsoid, the eigenvectors corresponding to two target eigenvalues ​​based on the covariance matrix corresponding to the target ellipsoid, as two planar vectors corresponding to the target ellipsoid, wherein the two target eigenvalues ​​are the first two eigenvalues ​​in the order of the eigenvalues ​​of the covariance matrix corresponding to the target ellipsoid from largest to smallest;

[0038] The surface shape determination module is used to determine the shape of the surface of the target scene at the target ellipsoid based on the plane formed by the two plane vectors corresponding to the target ellipsoid for each target ellipsoid.

[0039] The second three-dimensional model acquisition module is used to perform three-dimensional reconstruction of the target scene based on the shape of the surface of the target scene at each of the target ellipsoids, and obtain the second three-dimensional model of the target scene;

[0040] The smooth depth map acquisition module is used to render the second three-dimensional model of the target scene to obtain a smooth depth map of the target scene.

[0041] In one possible embodiment, the surface shape determination module determines the shape of the surface of the target scene at the target ellipsoid based on the plane formed by two planar vectors corresponding to the target ellipsoid, including:

[0042] The normal vector of the plane formed by the two plane vectors corresponding to the target ellipsoid is taken as the normal vector of the target ellipsoid; the normal vector is used to characterize the shape of the surface of the target scene at the target ellipsoid.

[0043] In one possible embodiment, the second three-dimensional model obtaining module performs three-dimensional reconstruction of the target scene based on the shape of the surface of the target scene at each of the target ellipsoids to obtain a second three-dimensional model of the target scene, including:

[0044] Calculate the vector field based on the normal vector of each target ellipsoid and the position of each target ellipsoid;

[0045] Construct and solve the Poisson equation to determine the indicator function corresponding to the vector field;

[0046] Based on the indicator function corresponding to the vector field, the isosurfaces where each target ellipsoid is located are determined; the values ​​of the indicator functions corresponding to each point on the isosurfaces are equal;

[0047] Based on the isosurfaces where each target ellipsoid is located, a second three-dimensional model of the target scene is constructed.

[0048] In one possible embodiment, the smooth depth map obtaining module renders a second 3D model of the target scene to obtain a smooth depth map of the target scene, including:

[0049] The second 3D model of the target scene is rendered to obtain the original depth map of the target scene;

[0050] The original depth map is filtered to obtain a smooth depth map of the target scene.

[0051] In one possible embodiment, the target scene is a pre-defined large-area scene; the device further includes:

[0052] The target color map acquisition module is used to acquire the target color map rendered based on the first 3D model.

[0053] The other depth map acquisition module is used to acquire other depth maps and other color maps of other scenes rendered based on non-3DGS technology; the other scenes are preset small-area scenes, and the smooth depth map, the target color map, the other depth map and the other color map are rendered from the same angle;

[0054] The target color map and the other color maps are fused to obtain a fused image through the following methods:

[0055] For each pixel, if the depth value of the pixel in the smooth depth map is greater than the depth value of the pixel in other depth maps, then the pixel value of the pixel in the other color maps is set as the pixel value of the pixel in the blended image; if the depth value of the pixel in the smooth depth map is less than the depth value of the pixel in other depth maps, then the pixel value of the pixel in the target color map is set as the pixel value of the pixel in the blended image.

[0056] In one possible embodiment, the position vector determination module determines vectors originating from the target ellipsoid and ending at each of its nearest neighbor ellipsoids, thereby obtaining multiple position vectors corresponding to the target ellipsoid, including:

[0057] By determining vectors that start from the center point of the target ellipsoid and end at the center points of each of the target ellipsoid's nearest neighbor ellipsoids, multiple position vectors corresponding to the target ellipsoid are obtained.

[0058] The nearest neighbor ellipsoid determination module determines other ellipsoids in the first three-dimensional model whose distance from the target ellipsoid is less than a preset distance threshold, as nearest neighbor ellipsoids, including:

[0059] In the first three-dimensional model, the nearest neighbor search algorithm KDTree is used to determine other ellipsoids corresponding to the center point whose distance from the center point of the target ellipsoid is less than a preset distance threshold, and these are taken as the nearest neighbor ellipsoids.

[0060] This application also provides an electronic device, including:

[0061] Memory, used to store computer programs;

[0062] The processor, when executing a program stored in memory, implements any of the depth map determination methods described above.

[0063] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the depth map determination methods described above.

[0064] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to execute any of the depth map determination methods described above.

[0065] Beneficial effects of the embodiments in this application:

[0066] This application provides a depth map determination method, apparatus, electronic device, and storage medium, which can acquire a first three-dimensional model of a target scene based on 3DGS. The first three-dimensional model contains multiple ellipsoids. For each target ellipsoid, other ellipsoids in the first three-dimensional model whose distance from the target ellipsoid is less than a preset distance threshold are determined as nearest neighbor ellipsoids. The target ellipsoid can be any multiple ellipsoids in the first three-dimensional model. For each target ellipsoid, vectors starting from the target ellipsoid and ending at each of its nearest neighbor ellipsoids are determined to obtain multiple position vectors corresponding to the target ellipsoid. The covariance matrix corresponding to each target ellipsoid is determined based on the position vectors corresponding to each target ellipsoid, wherein the value in the i-th row and j-th column of the covariance matrix is: the value of the target ellipsoid. The covariance between the components of each position vector in the i-th and j-th dimensions; for each target ellipsoid, based on the covariance matrix corresponding to the target ellipsoid, determine the eigenvectors corresponding to two target eigenvalues, which serve as two planar vectors corresponding to the target ellipsoid. The two target eigenvalues ​​are the two largest eigenvalues ​​in the covariance matrix corresponding to the target ellipsoid, sorted from largest to smallest; for each target ellipsoid, determine the shape of the surface of the target scene at that target ellipsoid based on the plane formed by the two planar vectors corresponding to that target ellipsoid; based on the shape of the surface of the target scene at each target ellipsoid, perform 3D reconstruction of the target scene to obtain a second 3D model of the target scene; render the second 3D model of the target scene to obtain a smooth depth map of the target scene. It can be understood that since the multiple position vectors corresponding to the target ellipsoid are vectors originating from the target ellipsoid and ending at its nearest neighbor ellipsoids, the larger the eigenvalue of the covariance matrix corresponding to the target ellipsoid obtained from the position vectors, the sparser the distribution of the nearest neighbor ellipsoids is in the direction of the eigenvector corresponding to that eigenvalue. Furthermore, it can be understood that in the first 3D model, the ellipsoids are distributed along the surface of the target scene. Therefore, in the cross-section formed along the target scene surface, the distribution of ellipsoids is sparse, while in the direction perpendicular to this cross-section, the distribution is concentrated. Thus, the plane formed by the eigenvectors corresponding to two larger eigenvalues ​​can be considered as the cross-section of the target scene's surface at the target ellipsoid. The shape of the target scene's surface at the target ellipsoid can then be determined using the plane formed by the two plane vectors. Based on the shape of the target scene's surface at each target ellipsoid, a 3D reconstruction of the target scene can be performed, constructing the complete surface shape of the target scene and obtaining the second 3D model of the target scene. Since the second 3D model can represent the complete surface shape of the target scene, the depth information of each point on the target scene's surface shape can be obtained from the second 3D model. The depth information corresponding to points at different positions in the second 3D model is different, whereas in the first 3D model, even if two points are at different positions, if they are within the same ellipsoid, their depth information is the same.Obviously, the second 3D model has higher precision in depth information compared to the first 3D model. Therefore, the smooth depth map rendered from the second 3D model is more precise than the depth map rendered from the first 3D model (the depth map rendered from the 3D model constructed using 3DGS technology), thus improving the precision of the depth map of the 3D model.

[0067] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0068] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0069] Figure 1a A two-dimensional schematic diagram of Mesh rendering provided in an embodiment of this application;

[0070] Figure 1b A schematic diagram illustrating a process for generating images of a target scene from a new perspective using 3DGS, as provided in an embodiment of this application.

[0071] Figure 1c A schematic diagram of the 3DGS shape fitting process provided in an embodiment of this application;

[0072] Figure 1d This is a schematic diagram of ellipsoidal projection provided in an embodiment of this application;

[0073] Figure 2a A schematic diagram of an input image provided in an embodiment of this application;

[0074] Figure 2b A schematic diagram of a three-dimensional model provided in an embodiment of this application;

[0075] Figure 2c According to the embodiments provided in this application Figure 2b A schematic diagram of an RGB image obtained from the shown 3D model;

[0076] Figure 2d According to the embodiments provided in this application Figure 2b A schematic diagram of a depth map obtained from the shown 3D model;

[0077] Figure 3 A schematic flowchart of a depth map determination method provided in an embodiment of this application;

[0078] Figure 4A schematic diagram of a smooth depth map provided in an embodiment of this application;

[0079] Figure 5 A schematic diagram of a binary tree generated by the KDTree nearest neighbor search algorithm provided in this application embodiment;

[0080] Figure 6a A schematic diagram of an RGB image obtained by Mesh rendering as provided in an embodiment of this application;

[0081] Figure 6b A schematic diagram of a fused image provided in an embodiment of this application;

[0082] Figure 7 This is a schematic flowchart of an image fusion method based on a depth map determination method provided in the embodiments of this application;

[0083] Figure 8 A schematic diagram of the depth map determination device provided in the embodiments of this application;

[0084] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0085] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0086] To provide a clearer explanation of the depth map determination method, apparatus, electronic device, and storage medium provided in this application, the following will explain some terms that appear in this document:

[0087] Mesh (3D Mesh): A 3D mesh is composed of polygons formed by multiple adjacent points on the surface of an object; that is, it is composed of polygons formed by the cloud of adjacent points of the object. 3D meshes are typically composed of triangles, quadrilaterals, or other simple convex polygons. In computer graphics, they also include information such as material or texture. Because a 3D mesh describes a 3D object by representing the points on its surface, it is an explicit data method for describing 3D objects.

[0088] 3D Reconstruction: refers to the creation of a mathematical model of a three-dimensional object that is suitable for computer representation and processing. Commonly used 3D reconstruction methods include SFM (Structure from Motion).

[0089] SFM: It can take multiple pictures of a scene with a camera, calculate the relative position of the camera to the scene when the picture was taken, and obtain the point cloud of the scene. In the following text, the relative position of the camera to the scene when the picture was taken is called the camera position, and the point cloud of the scene is obtained.

[0090] 3DGS: Based on an image of a scene captured by an input camera from one viewpoint, it can generate images of that scene from other viewpoints. This process will be referred to as new-view image rendering below. 3DGS also describes 3D objects in a scene using point clouds; therefore, 3DGS is also a way to explicitly describe 3D objects.

[0091] NeRF (Neural Radiance Fields): NeRF can generate a 3D reconstruction model based on an input image of a scene. By rendering the 3D reconstruction model, an image of the scene from a perspective not captured by the camera can be obtained; that is, NeRF can achieve new perspective image rendering. Since NeRF does not directly describe the points on the surface of 3D objects in the scene, but rather uses functions to represent the relationships between points on the surface of 3D objects, NeRF is an implicit way of describing 3D objects.

[0092] KDTree (k-dimensional tree): A data structure for partitioning a k-dimensional data space. It is mainly used for searching key data in multi-dimensional space, such as range search and nearest neighbor search. Range search refers to finding all data in the dataset that are less than the target point given a target point and a preset distance threshold. Nearest neighbor search is the K-nearest neighbor search when K=1. K-nearest neighbor search refers to finding the K nearest data points to the target point given a target point and a positive integer K.

[0093] To more clearly illustrate the depth map determination method, apparatus, electronic device, and storage medium provided in this application, the following will provide illustrative examples of possible application scenarios for the depth map determination method, apparatus, electronic device, and storage medium provided in this application. It is understood that the following examples are only possible application scenarios for the depth map determination method, apparatus, electronic device, and storage medium provided in this application. In other possible embodiments, the depth map determination method, apparatus, electronic device, and storage medium provided in this application can also be applied to other possible application scenarios. The following examples do not impose any limitations on this.

[0094] Currently, new perspective image rendering can be achieved using NeRF. Although NeRF is low-cost, highly automated, and highly realistic, as mentioned earlier, it is an implicit way of describing 3D objects. Therefore, it is difficult for NeRF to work simultaneously with explicit 3D object description methods such as Mesh rendering. Furthermore, NeRF has poor controllability, is incompatible with mainstream editing software, and can basically only output scene color information, not depth segmentation information. Therefore, new perspective image rendering is currently usually achieved through Mesh rendering or 3DGS.

[0095] 3DGS can estimate the image of a target scene from any other viewpoint using an image captured from a limited camera viewpoint, thus achieving new perspective image rendering. Mesh rendering refers to the method of generating a mesh by 3D scanning an image of a target scene captured from a limited camera viewpoint, and then rendering the mesh to obtain an image of the target scene from any other viewpoint. Mesh rendering has strong compatibility; all existing 3D software supports mesh rendering. Furthermore, by setting the density of the point cloud, the output result can be made closer to the real object, meaning the output result is controllable and can output depth, segmentation, pose, and other information relatively well.

[0096] However, the aforementioned mesh rendering method requires expensive equipment, is time-consuming, and necessitates significant manual post-processing, making it costly and unable to be automated. Furthermore, since computers have a limited capacity for processing data, generating meshes for large scenes, such as parks or amusement parks, requires processing a massive amount of data. Therefore, mesh rendering is difficult to use for large-scale scenes. The following will use a 2D diagram as an example to illustrate the principles of mesh rendering and 3DGS.

[0097] Because it is mathematically difficult to accurately describe free-form shapes, and computers cannot process non-discrete data, mesh rendering approximates continuous shapes by using point clouds and connectivity to form polygons. (See [link to relevant documentation]). Figure 1a As shown, the Mesh rendering method has low realism and struggles to accurately depict freeform shapes.

[0098] 3DGS does not use polygons; instead, it uses a series of ellipsoids to fit freeform shapes. The process of 3DGS generating an image of a target scene from a new perspective can be described as follows: Figure 1b As shown. Figure 1bThe direction indicated by the black solid line is the direction of the original data flow, and the direction indicated by the blue solid line is the direction of the gradient flow. First, the point cloud of the target scene and the camera position are obtained through SFM. This point cloud is a Gaussian point cloud, and each point in the point cloud has an initial radius. These points are referred to as ellipsoids below. After initializing each ellipsoid, the shape of the target scene in the image is fitted using each ellipsoid; this is the shape fitting process.

[0099] The process of fitting a shape can be as follows: Figure 1c As shown, 3DGS clones ellipses to obtain multiple ellipses. By continuously adjusting the position, number, and shape of each ellipses in the image, it optimizes the connected components formed by these ellipses. When the ellipses are combined, the resulting point cloud more closely resembles... Figure 1c When the target shape is shown by the solid black line in the middle, 3DGS has completed the shape fitting process. (Comparison) Figure 1a as well as Figure 1c It is evident that describing free shapes using 3DGS is more accurate than describing free shapes using Mesh, and it also uses less data.

[0100] The shape fitting process involves not only fitting the shape of the target scene but also fitting its color. It's understandable that the density of the target scene affects its ability to absorb light; the higher the density, the stronger its light absorption, meaning light doesn't easily pass through, and consequently, the darker the color of the target scene. Therefore, the process of fitting the color of the target scene can be viewed as... Figure 1b The density correction process is shown in the diagram. After the shape fitting process is completed, a 3D model of the target scene can be built.

[0101] After establishing a 3D model of the target scene, the camera position can be set, and the 3D model of the target scene can be rendered using a differentiable renderer. This involves projecting the 3D model of the target scene to obtain an image of the target scene from the camera's perspective. Furthermore, the density of the fitted shape can be corrected using the gradient of the obtained image.

[0102] Since 3DGS establishes a 3D model of a target scene by fitting multiple ellipsoids, projecting the 3D model of the target scene is equivalent to projecting each ellipsoid within that 3D model. Therefore, the following example will focus on projecting one ellipsoid from the 3D model of the target scene. Figure 1d The illustration provides an example of how to project a 3D model of a target scene in 3D space onto a 2D plane in camera space. See [link to documentation]. Figure 1d Let the ellipsoidal pose be denoted as W, where W is defined by the ellipsoidal major axis vector (w) in the figure. 00 w 10w 20 ) T ellipsoid minor axis vector (w 01 w 11 w 21 ) T And the coordinates of the center point of the ellipsoid (u0, u1, u2) T Description, i.e. Let V be the position of the ellipse obtained by projecting the ellipsoid onto the plane. Then, the position of the ellipse on the plane can be calculated using the following formulas (1)-(3):

[0103]

[0104]

[0105] S = Major axis of ellipsoid / Minor axis of ellipsoid ... (3)

[0106] Where W is the translation and rotation matrix, i.e. the ellipsoidal pose; J is the Jcobian matrix, i.e. the partial derivative matrix; the parameter V" can be calculated by formula (2); the parameter S can be calculated by formula (3).

[0107] For example, suppose a series of images are input to 3DGS as follows: Figure 2a As shown, 3DGS can be based on Figure 2a Images 1-9 shown are used to construct a 3D model of the scene in the image. The constructed 3D model can be used as follows: Figure 2b As shown. Rendering the 3D model of this scene yields an RGB (Red, Green, Blue) image, which can be displayed as follows. Figure 2c As shown, the RGB image is only slightly different from images 1-9 captured by the camera. Rendering the 3D model of this scene yields a depth map as shown below. Figure 2d As shown, it is obvious that Figure 2d The depth map shown has low accuracy.

[0108] See Figure 2b The smallest rendering unit in 3DGS technology is an ellipsoid, and the depth information of the entire ellipsoid is identical. Therefore, when an ellipsoid is projected onto a plane to obtain an ellipse, the depth information of the entire ellipse is also identical. Furthermore, when projecting an image of a scene from a 3D model reconstructed using 3DGS technology, multiple ellipsoids are projected onto the same plane, resulting in an image containing multiple ellipses. While the depth value of a particular ellipse in the image may be the same, the actual depth values ​​of the pixels corresponding to the ellipse's position in the image may differ. Therefore, the depth map obtained from a 3D model reconstructed using 3DGS technology has relatively low accuracy.

[0109] Therefore, in order to improve the accuracy of the depth map of the 3D model, this application provides a depth map determination method, see [link to relevant documentation]. Figure 3 The methods include:

[0110] S301, Obtain the first 3D model of the target scene based on 3DGS.

[0111] The first three-dimensional model contains multiple ellipsoids.

[0112] S302, for each target ellipsoid, determine other ellipsoids in the first three-dimensional model whose distance from the target ellipsoid is less than a preset distance threshold, and designate them as neighboring ellipsoids.

[0113] The target ellipsoid can be any number of ellipsoids in the first three-dimensional model.

[0114] S303, for each target ellipsoid, determine the vectors starting from the target ellipsoid and ending at each of its nearest neighbor ellipsoids, thus obtaining multiple position vectors corresponding to the target ellipsoid.

[0115] S304, determine the covariance matrix of each target ellipsoid based on the position vector of each target ellipsoid.

[0116] The value in the i-th row and j-th column of the covariance matrix represents the covariance between the components of each position vector corresponding to the target ellipsoid in the i-th and j-th dimensions.

[0117] S305, for each target ellipsoid, based on the covariance matrix corresponding to the target ellipsoid, determine the eigenvectors corresponding to the two target eigenvalues, which are then used as the two planar vectors corresponding to the target ellipsoid.

[0118] Among them, the two target eigenvalues ​​are the two eigenvalues ​​that are the first two in descending order of the eigenvalues ​​of the covariance matrix corresponding to the target ellipsoid.

[0119] S306, For each target ellipsoid, determine the shape of the surface of the target scene at the target ellipsoid based on the plane formed by the two plane vectors corresponding to the target ellipsoid.

[0120] S307. Based on the shape of the surface of the target scene at each target ellipsoid, the target scene is reconstructed in three dimensions to obtain a second three-dimensional model of the target scene.

[0121] S308 renders the second 3D model of the target scene to obtain a smooth depth map of the target scene.

[0122] Using the embodiments of this application, a first three-dimensional model of a target scene based on 3DGS can be obtained. The first three-dimensional model contains multiple ellipsoids. For each target ellipsoid, other ellipsoids in the first three-dimensional model whose distance from the target ellipsoid is less than a preset distance threshold are determined as nearest neighbor ellipsoids. The target ellipsoid can be any multiple ellipsoids in the first three-dimensional model. For each target ellipsoid, vectors starting from the target ellipsoid and ending at each of its nearest neighbor ellipsoids are determined to obtain multiple position vectors corresponding to the target ellipsoid. The covariance matrix corresponding to each target ellipsoid is determined based on the position vectors corresponding to each target ellipsoid, wherein the value of the i-th row and j-th column of the covariance matrix is: the sum of the values ​​of each position vector corresponding to the target ellipsoid in the i-th dimension and the sum of the values ​​of each position vector ... The covariance between the components of the j-th dimension; for each target ellipsoid, based on the covariance matrix corresponding to the target ellipsoid, determine the eigenvectors corresponding to the two target eigenvalues, which are used as the two planar vectors corresponding to the target ellipsoid. The two target eigenvalues ​​are the two largest eigenvalues ​​in the covariance matrix corresponding to the target ellipsoid, sorted from largest to smallest. For each target ellipsoid, determine the shape of the surface of the target scene at that target ellipsoid based on the plane formed by the two planar vectors corresponding to that target ellipsoid. Based on the shape of the surface of the target scene at each target ellipsoid, perform 3D reconstruction of the target scene to obtain the second 3D model of the target scene. Render the second 3D model of the target scene to obtain a smooth depth map of the target scene. It can be understood that since the multiple position vectors corresponding to the target ellipsoid are vectors originating from the target ellipsoid and ending at its nearest neighbor ellipsoids, the larger the eigenvalue of the covariance matrix corresponding to the target ellipsoid obtained from the position vectors, the sparser the distribution of the nearest neighbor ellipsoids is in the direction of the eigenvector corresponding to that eigenvalue. Furthermore, it can be understood that in the first 3D model, the ellipsoids are distributed along the surface of the target scene. Therefore, in the cross-section formed along the target scene surface, the distribution of ellipsoids is sparse, while in the direction perpendicular to this cross-section, the distribution is concentrated. Thus, the plane formed by the eigenvectors corresponding to two larger eigenvalues ​​can be considered as the cross-section of the target scene's surface at the target ellipsoid. The shape of the target scene's surface at the target ellipsoid can then be determined using the plane formed by the two plane vectors. Based on the shape of the target scene's surface at each target ellipsoid, a 3D reconstruction of the target scene can be performed, constructing the complete surface shape of the target scene and obtaining the second 3D model of the target scene. Since the second 3D model can represent the complete surface shape of the target scene, the depth information of each point on the target scene's surface shape can be obtained from the second 3D model. The depth information corresponding to points at different positions in the second 3D model is different, whereas in the first 3D model, even if two points are at different positions, if they are within the same ellipsoid, their depth information is the same.Obviously, the second 3D model has higher precision in depth information compared to the first 3D model. Therefore, the smooth depth map rendered from the second 3D model is more precise than the depth map rendered from the first 3D model (the depth map rendered from the 3D model constructed using 3DGS technology), thus improving the precision of the depth map of the 3D model.

[0123] The following will provide exemplary descriptions of S301-S308 respectively:

[0124] In S301, the target scene refers to all objects included in the image captured by the camera, as well as the surrounding environment of the objects. The method for building the first 3D model of the target scene based on 3DGS can be as described above. Figure 1b , Figure 1c as well as Figure 1d The relevant explanations will not be repeated here. As mentioned above, the smallest rendering unit in 3DGS is an ellipsoid; therefore, the first 3D model contains multiple ellipsoids.

[0125] For example, if the target scenario is Figure 2a The scene shown in Images 1-9 in the image refers to the flowerpot and its surrounding environment (such as the platform on which the flowerpot is placed). The first 3D model of the target scene built based on 3DGS can be as follows: Figure 2b As shown. See also Figure 2b The first three-dimensional model contains multiple ellipsoids.

[0126] In S302, the target ellipsoid can refer to all ellipsoids in the first 3D model, or it can refer to a portion of the ellipsoids in the first 3D model. If the target ellipsoid is a portion of the ellipsoids in the first 3D model, the user can determine the region of interest in the first 3D model. All ellipsoids contained in the region of interest are the target ellipsoids. Therefore, steps S103-S108 can be performed only on the ellipsoids of the region of interest to improve the accuracy of the depth map of the 3D model of the region of interest.

[0127] The region of interest (ROI) refers to the area within the target scene where objects of interest to the user are located. For example, suppose the target scene is... Figure 2a In the scenes shown in Images 1-9, if the user is only interested in the flowerpots in the target scene, the area where the flowerpots are located is taken as the region of interest. If the user is interested in both the flowerpots and the platform on which they are placed, the area containing both the flowerpots and the platform is taken as the region of interest.

[0128] The distance between two ellipsoids refers to the distance between the endpoints of the two ellipsoids, where the endpoints refer to any points that can represent the position of the ellipsoid. Exemplarily, the endpoints can refer to the center point of the ellipsoid, the vertex of the long axis of the ellipsoid, the vertex of the short axis of the ellipsoid, or a point inside the ellipsoid. In one possible embodiment, in order to more accurately determine the position of the ellipsoid, the endpoints refer to the center point of the ellipsoid. In the following, taking the endpoints as the center point of the ellipsoid as an example, an exemplary description of the depth map determination method of the present application will be given.

[0129] If a certain target ellipsoid is denoted as S i , that is, the center point coordinates of this target ellipsoid are S i , the neighboring ellipsoid is denoted as S j , that is, the center point coordinates of the neighboring ellipsoid are S j , and the set composed of the neighboring ellipsoids S i of this target ellipsoid S j is denoted as N si , then N si = {││S j - S i ││ < r}. Where │S j - S i │ refers to the distance between this target ellipsoid S i and the neighboring ellipsoid S j , and r is a preset distance threshold, and r can be set according to the actual situation or past experience. Exemplarily, r can be set to 1mm, 2mm, 3mm, etc. In the following, the method for determining the neighboring ellipsoid of the target ellipsoid in the first three-dimensional model will be described and will not be elaborated here.

[0130] In S303, corresponding to the relevant description of the distance in the foregoing S302, the vector starting from a certain target ellipsoid and ending at the neighboring ellipsoids of this target ellipsoid refers to: the vector starting from the endpoint of this target ellipsoid and ending at the endpoints of the neighboring ellipsoids of this target ellipsoid.

[0131] Exemplarily, if the endpoint is the center point of the ellipsoid, and this target ellipsoid is denoted as S i , that is, the center point coordinates of this target ellipsoid are S i , and the neighboring ellipsoid of this target ellipsoid is denoted as S j , that is, the center point coordinates of the neighboring ellipsoid of this target ellipsoid are S j , then the position vector corresponding to this target ellipsoid S i can be expressed as If this target ellipsoid is S1, the multiple neighboring ellipsoids of this target ellipsoid are respectively: S 11 , S 12 , S 13 , ……, S1n Then the multiple position vectors corresponding to the target ellipsoid S1 are as follows:

[0132] In S304, if the covariance matrix is ​​denoted as C, and the position vectors corresponding to the target ellipsoid are three-dimensional vectors with three dimensions of x, y, and z, then the covariance matrix of a target ellipsoid can be calculated using the following formula (4):

[0133]

[0134] Where, cov(x,x) is the covariance between the x-dimensional and x-dimensional components of each position vector corresponding to the target ellipsoid; cov(x,y) is the covariance between the x-dimensional and y-dimensional components of each position vector corresponding to the target ellipsoid; cov(x,z) is the covariance between the x-dimensional and z-dimensional components of each position vector corresponding to the target ellipsoid; cov(y,x) is the covariance between the y-dimensional and x-dimensional components of each position vector corresponding to the target ellipsoid; and cov(y,y) is the covariance between the x-dimensional and z-dimensional components of each position vector corresponding to the target ellipsoid. The covariance between the position vectors in the y-dimensional and z-dimensional components; cov(y,z) is the covariance between the position vectors corresponding to the target ellipsoid in the y-dimensional and z-dimensional components; cov(z,x) is the covariance between the position vectors corresponding to the target ellipsoid in the z-dimensional and x-dimensional components; cov(z,y) is the covariance between the position vectors corresponding to the target ellipsoid in the z-dimensional and y-dimensional components; cov(z,z) is the covariance between the position vectors corresponding to the target ellipsoid in the z-dimensional and z-dimensional components.

[0135] In the example of formula (4) above, x is the first dimension, y is the second dimension, and z is the third dimension. That is, the value of the i-th row and j-th column in the covariance matrix is ​​the covariance between the components of the position vectors corresponding to the target ellipsoid in the i-th and j-th dimensions. For example, if i is 1 and j is 3, then the value of cov(x,z) in the 1-th row and 3-th column in the covariance matrix is ​​the covariance between the components of the position vectors corresponding to the target ellipsoid in the x-th and z-th dimensions.

[0136] If the multiple position vectors corresponding to the target ellipsoid S1 are respectively: The covariance between the x- and y-dimensional components of each position vector corresponding to the target ellipsoid can then be calculated using the following formula (5):

[0137]

[0138] Where cov(x,y) is the covariance between the components of each position vector corresponding to the target ellipsoid in the x and y dimensions, and x a The position vector corresponding to the target ellipsoid S1 In the x-dimensional component, y a The position vector corresponding to the target ellipsoid S1 The components in the y-dimension, Let x be the expectation of the components of each position vector corresponding to the target ellipsoid S1 in the x-axis. Let S be the expectation of the components of each position vector corresponding to the target ellipsoid S1 in the y-dimensional direction.

[0139] The calculation method for other elements in the covariance matrix C is the same as that in formula (5) above, the only difference being the dimension. The position vector corresponding to the target ellipsoid S1 in formula (5) above can be used. The component in a certain dimension is modified to the position vector corresponding to the target ellipsoid S1. The other elements in the covariance matrix C can be calculated by considering the components of the dimensions corresponding to the other elements.

[0140] In S305, after calculating the covariance matrix corresponding to each target ellipsoid according to the aforementioned formulas (4) and (5), the eigenvalues ​​of the covariance matrix corresponding to each target ellipsoid and the eigenvectors corresponding to the eigenvalues ​​can be calculated. The eigenvalues ​​that are the first two in descending order of size are taken as the two target eigenvalues. The eigenvectors corresponding to these two target eigenvalues ​​are taken as the two planar vectors corresponding to the target ellipsoid.

[0141] In S306, it can be understood that since the multiple position vectors corresponding to the target ellipsoid are vectors that start from the target ellipsoid and end at each of its nearest neighbor ellipsoids, the larger the eigenvalue of the covariance matrix corresponding to the target ellipsoid obtained from the position vectors corresponding to the target ellipsoid, the sparser the distribution of the nearest neighbor ellipsoids is in the direction of the eigenvector corresponding to that eigenvalue. Furthermore, it can be understood that the ellipsoids in the first 3D model are distributed along the surface of the target scene. Therefore, in the cross-section formed along the surface of the target scene, the distribution of ellipsoids is sparse, while in the direction perpendicular to the cross-section, the distribution of ellipsoids is concentrated. Therefore, the eigenvectors corresponding to the two larger eigenvalues ​​of the covariance matrix corresponding to the target ellipsoid, i.e., the plane formed by the two planar vectors, can be regarded as the cross-section of the surface of the target scene at the target ellipsoid. Thus, the shape of the surface of the target scene at the target ellipsoid can be determined by the plane formed by the two planar vectors.

[0142] For example, suppose the two planar vectors corresponding to a certain target ellipsoid are respectively Then the two planar vectors corresponding to the target ellipsoid can be stored: This is used to represent the shape of the surface of the target scene at the target ellipsoid.

[0143] In another possible embodiment, for each target ellipsoid, the normal vector of the plane formed by the two plane vectors corresponding to the target ellipsoid can be directly calculated based on the two plane vectors corresponding to the target ellipsoid, and the normal vector of the plane formed by the two plane vectors corresponding to the target ellipsoid can be used as the normal vector of the target ellipsoid. The normal vector can then be used to characterize the shape of the surface of the target scene at the target ellipsoid.

[0144] For example, suppose the two planar vectors corresponding to a certain target ellipsoid are respectively Then the normal vector corresponding to the target ellipsoid can be calculated. And store the normal vector corresponding to the target ellipsoid. This is used to represent the shape of the surface of the target scene at the target ellipsoid.

[0145] By using this embodiment, the shape of the surface of the target scene at the target ellipsoid can be directly represented by the normal vector of the plane formed by the two plane vectors corresponding to the target ellipsoid. Compared with the method of storing two plane vectors to store the shape of the surface of the target scene at the target ellipsoid, storing only the normal vector can save storage space, and the normal vector can more easily represent the shape of the surface of the target scene at the target ellipsoid.

[0146] In S307, the target scene can be reconstructed in 3D based on the surface shape at each target ellipsoid, thus constructing a complete surface shape and obtaining a second 3D model of the target scene. Since the second 3D model can represent the complete surface shape of the target scene, depth information of each point on the surface shape can be obtained from it, and the depth information corresponding to points at different locations in the second 3D model is different. The method of reconstructing the target scene in 3D based on the surface shape at each target ellipsoid will be explained below and will not be repeated here.

[0147] In S308, a virtual camera can be set up to simulate the imaging of a second 3D model from the perspective of the virtual camera, thereby rendering the second 3D model of the target scene and obtaining a smooth depth map of the target scene.

[0148] Furthermore, by simulating a radar detector, a second-dimensional model of the target scene can be rendered to obtain a smooth depth map of the target scene.

[0149] To improve the smoothness of the depth map obtained from rendering the second 3D model, in one possible embodiment, the second 3D model of the target scene can be rendered to obtain the original depth map of the target scene; the original depth map can then be filtered to obtain a smoothed depth map of the target scene.

[0150] The original depth map can be filtered using mean filtering. Specifically, any pixel in the original depth map is taken as the target pixel. A filtering template is formed using the neighboring pixels of the target pixel. The average pixel value of all pixels in the filtering template is calculated, and the pixel value of the target pixel is modified to this average value, thus achieving mean filtering and obtaining a smooth depth map of the target scene. The smooth depth map obtained based on the depth map determination method provided in this application can be used as follows: Figure 4 As shown.

[0151] The neighboring pixels around the target pixel can refer to the eight pixels surrounding the target pixel, in which case the target pixel itself is included in the filter template. Alternatively, the neighboring pixels around the target pixel can refer to the 24 pixels surrounding the target pixel, in which case the target pixel itself is included in the filter template.

[0152] By using this embodiment, the second 3D model of the target scene can be rendered to obtain the original depth map of the target scene. By filtering the original depth map, a smooth depth map of the target scene can be obtained, thereby improving the smoothness of the original depth map obtained from the rendering of the second 3D model.

[0153] The following will provide an exemplary description of how to determine the nearest neighbor ellipsoids of the target ellipsoid in the first three-dimensional model in the aforementioned S302.

[0154] In one possible embodiment, all ellipsoids in the first three-dimensional model except the target ellipsoid can be traversed, the distance between the endpoints of each ellipsoid except the target ellipsoid and the endpoints of the target ellipsoid can be determined, and other ellipsoids whose distance from the endpoints of the target ellipsoid is less than a preset distance threshold can be regarded as the nearest neighbor ellipsoids of the target ellipsoid.

[0155] Since the number of ellipses in the first 3D model is large, and all ellipses in the first 3D model need to be traversed for each target ellipsoid, the above method may result in a large amount of data to be processed and a large consumption of computing resources when determining the nearest neighbor ellipsoids of each target ellipsoid.

[0156] Therefore, in order to reduce the amount of data required to determine the nearest neighbor ellipsoids of the target ellipsoid and reduce the consumption of computing resources, in one possible embodiment, the KDTree nearest neighbor search algorithm can be used in the first 3D model to determine other ellipsoids corresponding to endpoints whose distance from the endpoint of the target ellipsoid is less than a preset distance threshold, and these are taken as nearest neighbor ellipsoids. As explained above, an endpoint refers to any point that can represent the position of an ellipsoid. The following will use the endpoint as the center point as an example to illustrate this embodiment.

[0157] If the endpoint is the center point, then in the first three-dimensional model, the nearest neighbor search algorithm KDTree can be used to determine other ellipsoids corresponding to the center point whose distance from the center point of the target ellipsoid is less than a preset distance threshold, and these other ellipsoids can be used as the nearest neighbor ellipsoids.

[0158] Specifically, the set of coordinates of the center points of all ellipsoids in the first 3D model is input into the KDTree algorithm as a dataset. This allows the KDTree algorithm to sort the coordinates in the dataset according to their components in different dimensions, generating a binary tree corresponding to the dataset. Figure 5 As shown. See also Figure 5 In a binary tree, taking coordinates 1 and 2 as an example, if coordinates 1 and 2 are directly connected, that is, coordinates 1 and 2 are located at the two ends of an arrow, it means that the difference between the components of coordinates 1 and 2 in a certain dimension is less than the difference between the components of coordinates 1 and other coordinates in that dimension.

[0159] When determining the nearest neighbor ellipsoids of a target ellipsoid, the center point of the target ellipsoid can be used as the target point, and input along with a preset distance threshold into the KDTree algorithm. This allows the KDTree algorithm to use the binary tree corresponding to the dataset to find all data points in the dataset whose distance to the target point is less than the preset distance threshold. Since each data point in the dataset corresponds to the center point coordinates of each ellipsoid in the first 3D model, finding all data points in the dataset whose distance to the target point is less than the preset distance threshold determines all other ellipsoids whose distance to the center point of the target ellipsoid is less than the preset distance threshold, thus identifying the nearest neighbor ellipsoids of the target ellipsoid.

[0160] For example, suppose the coordinates of the center points of all ellipsoids in the first 3D model are given as coordinate 1 - coordinate n. These n coordinates are input as a dataset into the KDTree algorithm. The binary tree generated by the KDTree algorithm can be as follows: Figure 5 As shown. If the coordinates of the center point of a target ellipsoid are coordinates 6, then the KDTree algorithm, when determining the nearest neighbor ellipsoids of the target ellipsoid, will... Figure 5 The binary tree shown starts at coordinate 6 and proceeds forward or backward. Figure 5Search in the direction of the middle arrow until all coordinates whose distance from coordinate 6 is less than the preset distance threshold are determined, and the nearest neighbor ellipsoids of the target ellipsoid can be determined.

[0161] For example, starting from coordinate 6, first check whether the distance between coordinates 3 and 6, and the distance between coordinates 6 and 9 are less than a preset distance threshold. Coordinates 3 and 9 can then be marked as visited to avoid multiple checks on the same coordinate. If the distances between coordinates 3 and 6, and between coordinates 6 and 9 are both less than the preset distance threshold, then check whether the distances between coordinates 1 and 6, 6 and 10, and 6 and 11 are less than the preset distance threshold. Coordinates 1, 10, and 11 can then be marked as visited.

[0162] If the distance between coordinates 1 and 6 is not less than a preset distance threshold, then coordinate 2, which is connected to coordinate 1 and has not been visited, will no longer be visited. In other words, it will no longer be checked whether the distance between coordinates 2 and 6 is less than the preset distance threshold. It can be understood that since the difference between the components of coordinates 1 and 2 in a certain dimension is less than the difference between the components of coordinate 1 and other coordinates in that dimension, coordinates 1 and 2 can be considered relatively close. Therefore, if the distance between coordinates 1 and 6 is not less than the preset distance threshold, then the distance between coordinates 2 and 6 can also be considered not less than the preset distance threshold. Therefore, if the distance between coordinates 1 and 6 is not less than the preset distance threshold, then coordinate 2, which is connected to coordinate 1 and has not been visited, will no longer be visited, and consequently, coordinates 4 and 5, which are connected to coordinate 2 and have not been visited, will also not be visited.

[0163] If the distance between coordinate 6 and coordinate 10 is less than the preset distance threshold, it is necessary to continue to check whether the distance between coordinate 6 and coordinate 10 that is connected to coordinate 10 and has not been visited is less than the preset distance threshold. This process continues until a coordinate is found whose distance to coordinate 6 is not less than the preset distance threshold. In this case, the coordinate connected to that coordinate and has not been visited will not be visited.

[0164] In this embodiment, the set of center point coordinates of all ellipsoids in the first 3D model is input as a dataset into the KDTree algorithm, enabling the algorithm to generate a binary tree corresponding to the dataset. Then, based on the binary tree, the nearest neighbor ellipsoids of each target ellipsoid are determined through a range search. In determining the nearest neighbor ellipsoids of each target ellipsoid, it is not necessary to traverse all ellipsoids in the first 3D model each time, thereby reducing the amount of data processed, lowering computational resource consumption, improving efficiency, reducing costs, and achieving a high degree of automation.

[0165] The method of determining the nearest neighbor ellipsoids of the target ellipsoid in the first three-dimensional model has been explained above. The following will provide an exemplary description of how to perform three-dimensional reconstruction of the target scene in S307 based on the shape of the surface of the target scene at each target ellipsoid.

[0166] Based on the normal vectors and positions of each target ellipsoid, a vector field is calculated; the Poisson equation is constructed and solved to determine the indicator function corresponding to the vector field; based on the indicator function corresponding to the vector field, the isosurfaces on which each target ellipsoid is located are determined; wherein, the values ​​of the indicator functions corresponding to each point on the isosurfaces are equal; based on the isosurfaces on which each target ellipsoid is located, a second three-dimensional model of the target scene is constructed.

[0167] The position of the target ellipsoid can be represented by the coordinates of its endpoints. The following explanation will use the endpoints as the center point. If the endpoints are the center point, the position of the target ellipsoid can be represented by the coordinates of its center point.

[0168] An octree space can be constructed based on the normal vector of the target ellipsoid and the coordinates of its center point, and the vector field can then be calculated. In this paper, the vector field is denoted as... Let the indicator function be denoted as The indicator function corresponding to the vector field can then be obtained by solving the following formula (6).

[0169]

[0170] Where Δ is the Laplace operator, For divergence operators, For a vector field, Let be the indicator function. Formula (6) above is the Poisson equation.

[0171] The indicator function can be used to indicate whether the center point of the target ellipsoid is located inside or outside the surface of the target scene. Based on the indicator function corresponding to the vector field, the value of the indicator function corresponding to the center point of each target ellipsoid can be determined. Center points with the same indicator function value are then grouped into isosurfaces, thus obtaining the isosurfaces where each target ellipsoid is located. A second 3D model of the target scene can then be constructed based on these isosurfaces.

[0172] Using this embodiment, a vector field can be calculated based on the normal vector of each target ellipsoid and the position of each target ellipsoid; the Poisson equation can be constructed and solved to determine the indicator function corresponding to the vector field; based on the indicator function corresponding to the vector field, the isosurface on which each target ellipsoid is located can be determined, and the value of the indicator function corresponding to each point on the isosurface is equal; based on the isosurface on which each target ellipsoid is located, a second three-dimensional model of the target scene can be constructed, so that the shape of the target scene is no longer determined by fitting each ellipsoid, but by determining the surface shape of the target scene through multiple points, thereby improving the accuracy of the depth map obtained from the second three-dimensional model.

[0173] As explained above regarding mesh rendering, it's difficult to create large scenes, such as outdoor scenes like parks or amusement parks. However, mesh rendering can be used to reconstruct 3D models of small scenes. For example, a small scene could be a cup photographed by a camera and its surrounding environment, or a flowerpot photographed by a camera and its surrounding environment. SDGS technology can reconstruct 3D models of large scenes. Therefore, based on the depth map determination method provided in this application, a second 3D model can be reconstructed from the first 3D model of the large scene constructed using 3DGS technology, resulting in a more accurate depth map of the large scene obtained from the second 3D model.

[0174] Based on the above explanation, if you need to obtain an image of a complete scene from a certain perspective, you can use 3DGS technology to reconstruct the 3D model of large scenes such as large buildings in the complete scene, and use Mesh rendering to reconstruct the 3D model of small scenes such as small decorations in the complete scene. Then, you can fuse the 3D model image obtained by 3DGS with the 3D model image obtained by Mesh rendering from the same perspective to obtain the image of the complete scene from that perspective.

[0175] To fuse images of two 3D models from the same viewpoint, high precision is required in the depth maps rendered from these two 3D models to determine their positions within that viewpoint, i.e., which part of the scene is visible from that viewpoint. Therefore, after obtaining a smoother depth map with higher precision based on steps S301-S308, this application also provides an image fusion method, which includes:

[0176] Obtain the target color map rendered based on the first 3D model.

[0177] The target color image and other color images mentioned in this article can be RGB images, HSV (Hue Saturation Value) images, CMYK (Cyan Magenta Yellow Black) images, etc. The following explanation will use RGB images as an example.

[0178] Obtain other depth maps and other color maps of other scenes rendered using non-3DGS technology; other scenes are preset small-area scenes, and the smooth depth map, target color map, other depth maps and other color maps are rendered from the same angle; the target scene is a preset large-area scene.

[0179] Non-3DGS technology can refer to the aforementioned Mesh rendering, or it can refer to any 3D model reconstruction method different from 3DGS technology. The following explanation will take Mesh rendering as an example of non-3DGS technology.

[0180] The preset large area refers to the area of ​​the target scene captured by the camera being greater than a first preset area threshold. The first preset area threshold can be set according to the actual situation. For example, the first preset area threshold can be set to 30 units, 50 units, 100 units, etc.

[0181] The preset large area refers to the area of ​​other scenes captured by the camera being smaller than the second preset area threshold. The second preset area threshold can be set according to the actual situation. For example, the second preset area threshold can be set to 10 units, 5 units, 4 units, etc.

[0182] The target color map and other color maps are fused to obtain a fused image in the following manner: for each pixel, if the depth value of the pixel in the smooth depth map is greater than the depth value of the pixel in other depth maps, then the pixel value of the pixel in other color maps is set as the pixel value of the pixel in the fused image; if the depth value of the pixel in the smooth depth map is less than the depth value of the pixel in other depth maps, then the pixel value of the pixel in the target color map is set as the pixel value of the pixel in the fused image.

[0183] Understandably, from the same viewpoint, if the depth value of object 1 in the depth map is greater than the depth value of object 2 in the depth map, then object 2 is closer to the camera, and the camera can only see object 2 from that viewpoint. Therefore, it is necessary to set the pixel value of the corresponding pixel in the color map of object 2 to the pixel value of the corresponding pixel in the fused image.

[0184] If we denote the other color patterns obtained through Mesh rendering as I... c Other depth maps obtained through mesh are denoted as D. c The target color map obtained by rendering the first 3D model is denoted as I. gs The smooth depth map obtained by rendering the second 3D model is denoted as D. gs Let the fused image be denoted as I. Then the pixel value of each pixel in the fused image I can be obtained by the following formula (7):

[0185]

[0186] Where I(x) is the pixel value of pixel x in the fused image I, I c (x) represents pixel x in other color maps I c The pixel value in I gs (x) represents the pixel x in the target color map I. gs The pixel value in D gs (x) represents the pixel x in the smooth depth map D. gs The depth value in D c (x) represents the pixel x in other depth maps D. c The depth value in the text.

[0187] If pixel x is in the smooth depth map D gs The depth value in the depth map is equal to the pixel x value in other depth maps D. c The depth value in the image allows you to map pixel x to other color maps. c The pixel value in the image is set to the pixel value of pixel x in the blended image I, or the pixel value of pixel x in the target color map I can be set to the pixel value of pixel x in the blended image I. gs The pixel value in D is set to the pixel value of pixel x in the fused image I, that is, when D gs (x)=D c (x), I(x)=I c (x), or, I(x) = I gs (x).

[0188] For example, the target color map obtained by rendering the first 3D model can be as follows: Figure 2c As shown, smooth depth map D gs It is possible Figure 4 As shown, other color maps obtained through Mesh rendering can be as follows: Figure 6a As shown, the fused image obtained based on the above image fusion method can be as follows: Figure 6b As shown.

[0189] Using this embodiment, a second 3D model can be obtained based on a first 3D model constructed using 3DGS technology. This results in a higher accuracy of the smooth depth map rendered from the second 3D model. Furthermore, the RGB image and smooth depth map rendered from the second 3D model can be fused with the RGB image and depth map rendered using Mesh technology. This fusion method is applicable to synthetic data generation, game and film production, and can also be used for the simulation of various power lines in substations. By improving the accuracy of the depth map obtained from the 3D model rendering, the practicality of the images rendered from the 3D model is enhanced.

[0190] The following will combine Figure 7 The flowchart and specific examples shown illustrate the image fusion method provided by this application based on the depth map determination method.

[0191] Multiple images and / or video clips captured by the camera are input into the SFM algorithm and the 3D scanning algorithm, respectively. The camera position and sparse point cloud output by the SFM algorithm are obtained, and the mesh and texture information of the target scene output by the 3D scanning algorithm are obtained.

[0192] The scenes in the images or videos input to the SFM algorithm and the 3D scanning algorithm can be different. For example, multiple images and / or video clips of the target scene captured by the camera can be input to the SFM algorithm, while multiple images and / or video clips of other scenes captured by the camera can be input to the 3D scanning algorithm. The multiple images of the target scene captured by the camera can be as follows: Figure 2a Images 1-9 are shown in the table.

[0193] The camera position and sparse point cloud are input into the 3DGS algorithm to obtain the first 3D model of the target scene constructed based on Gaussian point clouds using 3DGS technology. When rendering this first 3D model, the lighting conditions captured by the camera in the images and videos are restored using spherical harmonic functions to obtain the RGB image of the target scene from a certain viewpoint, i.e., the target color map, denoted as I1. For example, the first 3D model can be as follows: Figure 2b As shown, the target color map I1 can be as follows: Figure 2c As shown.

[0194] On the GPU (Graphics Processing Unit), raster rendering of the mesh and texture information of other scenes is performed using a shading algorithm to obtain the RGB image and depth map of other scenes from a certain viewpoint, which are the aforementioned other color maps and other depth maps. The other color map is denoted as I2, and the other depth map is denoted as D2. For example, the other color map can be as follows: Figure 6a As shown.

[0195] By executing steps S301-S308 above through the smoothing module, a smoothed depth map of the target scene is obtained, denoted as D1. The aforementioned S304 corresponds to... Figure 7 In the principal component analysis, S305-S306 correspond to Figure 7 Normal generation in the image. For example, smoothing the depth map D1 can be done as follows: Figure 4 As shown.

[0196] Identify the target color map I1, smoothed depth map D1, other color maps I2, and other depth maps D2 from the same viewpoint. This step corresponds to... Figure 7 The image is obtained by unifying the perspectives of the target color map I1 and other color maps I2 through a fusion module, and then performing the image fusion method described above. This fusion process yields a fused image, specifically through depth comparison and Poisson fusion. For example, the fused image can be as follows: Figure 6b As shown.

[0197] Corresponding to the aforementioned depth map determination method, this application also provides a depth map determination apparatus, such as... Figure 8 As shown, the device includes:

[0198] The first 3D model acquisition module 801 is used to acquire the first 3D model of the target scene based on 3DGS. The first 3D model contains multiple ellipsoids.

[0199] The nearest neighbor ellipsoid determination module 802 is used to determine, for each target ellipsoid, other ellipsoids in the first three-dimensional model whose distance from the target ellipsoid is less than a preset distance threshold, as nearest neighbor ellipsoids. The target ellipsoid can be any number of ellipsoids in the first three-dimensional model.

[0200] The position vector determination module 803 is used to determine, for each target ellipsoid, a vector starting from the target ellipsoid and ending at each of the target ellipsoid's nearest neighbor ellipsoids, thereby obtaining multiple position vectors corresponding to the target ellipsoid.

[0201] The covariance matrix determination module 804 is used to determine the covariance matrix corresponding to each target ellipsoid based on the position vector corresponding to each target ellipsoid. The value of the i-th row and j-th column of the covariance matrix is ​​the covariance between the i-th and j-th dimensions of the position vector corresponding to each target ellipsoid.

[0202] The plane vector determination module 805 is used to determine, for each target ellipsoid, the eigenvectors corresponding to two target eigenvalues ​​based on the covariance matrix of the target ellipsoid, as two plane vectors corresponding to the target ellipsoid. The two target eigenvalues ​​are the eigenvalues ​​that are the first two eigenvalues ​​in the eigenvalues ​​of the covariance matrix of the target ellipsoid in descending order.

[0203] The surface shape determination module 806 is used to determine the shape of the surface of the target scene at the target ellipsoid based on the plane formed by the two plane vectors corresponding to the target ellipsoid for each target ellipsoid.

[0204] The second three-dimensional model acquisition module 807 is used to perform three-dimensional reconstruction of the target scene based on the shape of the surface of the target scene at each target ellipsoid, and obtain the second three-dimensional model of the target scene.

[0205] The smooth depth map acquisition module 808 is used to render the second 3D model of the target scene to obtain a smooth depth map of the target scene.

[0206] In one possible embodiment, the surface shape determination module determines the shape of the surface of the target scene at the target ellipsoid based on the plane formed by two planar vectors corresponding to the target ellipsoid, including:

[0207] The normal vector of the plane formed by the two plane vectors corresponding to the target ellipsoid is taken as the normal vector of the target ellipsoid; the normal vector is used to characterize the shape of the surface of the target scene at the target ellipsoid.

[0208] In one possible embodiment, the second 3D model obtaining module performs 3D reconstruction of the target scene based on the shape of the surface of the target scene at each target ellipsoid, to obtain a second 3D model of the target scene, including:

[0209] Calculate the vector field based on the normal vector of each target ellipsoid and the position of each target ellipsoid;

[0210] Construct and solve the Poisson equation to determine the indicator function corresponding to the vector field;

[0211] Based on the indicator function corresponding to the vector field, determine the isosurface where each target ellipsoid is located; the value of the indicator function corresponding to each point on the isosurface is equal;

[0212] Based on the isosurfaces where each target ellipsoid is located, a second three-dimensional model of the target scene is constructed.

[0213] In one possible embodiment, the smooth depth map obtaining module renders a second 3D model of the target scene to obtain a smooth depth map of the target scene, including:

[0214] Render the second 3D model of the target scene to obtain the original depth map of the target scene;

[0215] The original depth map is filtered to obtain a smooth depth map of the target scene.

[0216] In one possible embodiment, the target scene is a pre-defined large-area scene; the device further includes:

[0217] The target color map acquisition module is used to acquire the target color map rendered based on the first 3D model;

[0218] The other depth map acquisition module is used to acquire other depth maps and other color maps of other scenes rendered based on non-3DGS technology; other scenes are preset small-area scenes, and the smooth depth map, target color map, other depth map and other color map are rendered from the same angle;

[0219] The target color map and other color maps are fused to obtain a fused image using the following methods:

[0220] For each pixel, if the depth value of the pixel in the smooth depth map is greater than the depth value of the pixel in other depth maps, then the pixel value of the pixel in other color maps is set as the pixel value of the pixel in the blended image; if the depth value of the pixel in the smooth depth map is less than the depth value of the pixel in other depth maps, then the pixel value of the pixel in the target color map is set as the pixel value of the pixel in the blended image.

[0221] In one possible embodiment, the position vector determination module determines vectors originating from the target ellipsoid and ending at each of its nearest neighbor ellipsoids, thereby obtaining multiple position vectors corresponding to the target ellipsoid, including:

[0222] By determining the vectors that start from the center point of the target ellipsoid and end at the center points of each of the target ellipsoid's nearest neighbor ellipsoids, multiple position vectors corresponding to the target ellipsoid are obtained.

[0223] The nearest ellipsoid determination module identifies other ellipsoids in the first 3D model whose distance from the target ellipsoid is less than a preset distance threshold as nearest neighbor ellipsoids, including:

[0224] In the first three-dimensional model, the nearest neighbor search algorithm KDTree is used to determine other ellipsoids corresponding to the center point whose distance from the center point of the target ellipsoid is less than a preset distance threshold, and these are taken as the nearest neighbor ellipsoids.

[0225] This application also provides an electronic device, such as... Figure 9 As shown, it includes:

[0226] Memory 901 is used to store computer programs;

[0227] When processor 902 executes a program stored in memory 901, it performs the following steps:

[0228] Obtain the first three-dimensional model of the target scene based on 3DGS. The first three-dimensional model contains multiple ellipsoids.

[0229] For each target ellipsoid, other ellipsoids in the first three-dimensional model whose distance from the target ellipsoid is less than a preset distance threshold are identified as nearest neighbor ellipsoids. The target ellipsoid can be any number of ellipsoids in the first three-dimensional model.

[0230] For each target ellipsoid, a vector is determined that starts from the target ellipsoid and ends at each of its nearest neighbor ellipsoids, thus obtaining multiple position vectors corresponding to the target ellipsoid.

[0231] The covariance matrix corresponding to each target ellipsoid is determined based on the position vector corresponding to each target ellipsoid. The value of the i-th row and j-th column of the covariance matrix is ​​the covariance between the i-th and j-th dimensions of the position vector corresponding to each target ellipsoid.

[0232] For each target ellipsoid, based on the covariance matrix corresponding to the target ellipsoid, determine the eigenvectors corresponding to the two target eigenvalues, which are used as the two planar vectors corresponding to the target ellipsoid. The two target eigenvalues ​​are the two eigenvalues ​​that are the first two eigenvalues ​​in the covariance matrix corresponding to the target ellipsoid, sorted from largest to smallest.

[0233] For each target ellipsoid, the shape of the surface of the target scene at the target ellipsoid is determined based on the plane formed by the two plane vectors corresponding to the target ellipsoid.

[0234] Based on the shape of the surface of the target scene at each target ellipsoid, the target scene is reconstructed in three dimensions to obtain a second three-dimensional model of the target scene;

[0235] Render the second 3D model of the target scene to obtain a smooth depth map of the target scene.

[0236] Furthermore, the aforementioned electronic device may also include a communication bus and / or a communication interface, with the processor 902, communication interface, and memory 901 communicating with each other via the communication bus.

[0237] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0238] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0239] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0240] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0241] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the depth map determination methods described above.

[0242] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the depth map determination methods described above.

[0243] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a solid-state drive (SSD), etc.

[0244] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0245] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, electronic devices, computer-readable storage media, and computer program products containing instructions are basically similar to the method embodiments, and therefore the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0246] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A depth map determination method characterized by, The method includes: Obtain a first 3D model of the target scene based on 3DGS, wherein the first 3D model contains multiple ellipsoids; For each target ellipsoid, other ellipsoids in the first three-dimensional model whose distance from the target ellipsoid is less than a preset distance threshold are determined as nearest neighbor ellipsoids. The target ellipsoid can be any number of ellipsoids in the first three-dimensional model. For each target ellipsoid, a vector is determined that starts from the target ellipsoid and ends at each of its nearest neighbor ellipsoids, thus obtaining multiple position vectors corresponding to the target ellipsoid. The covariance matrix corresponding to each target ellipsoid is determined based on the position vector corresponding to each target ellipsoid. The value of the i-th row and j-th column of the covariance matrix is ​​the covariance between the i-th and j-th dimensions of the position vector corresponding to each target ellipsoid. For each target ellipsoid, based on the covariance matrix corresponding to the target ellipsoid, determine the eigenvectors corresponding to two target eigenvalues ​​as two planar vectors corresponding to the target ellipsoid. The two target eigenvalues ​​are the two eigenvalues ​​that are the first two in descending order of the eigenvalues ​​of the covariance matrix corresponding to the target ellipsoid. For each target ellipsoid, the shape of the surface of the target scene at the target ellipsoid is determined based on the plane formed by the two plane vectors corresponding to the target ellipsoid. Based on the shape of the surface of the target scene at each of the target ellipsoids, the target scene is reconstructed in three dimensions to obtain a second three-dimensional model of the target scene; The second 3D model of the target scene is rendered to obtain a smooth depth map of the target scene.

2. The method according to claim 1, characterized in that, Determining the shape of the surface of the target scene at the target ellipsoid based on the plane formed by the two plane vectors corresponding to the target ellipsoid includes: The normal vector of the plane formed by the two plane vectors corresponding to the target ellipsoid is taken as the normal vector of the target ellipsoid; the normal vector is used to characterize the shape of the surface of the target scene at the target ellipsoid.

3. The method according to claim 2, characterized in that, The step of reconstructing the target scene in three dimensions based on the shape of the surface of the target scene at each of the target ellipsoids to obtain a second three-dimensional model of the target scene includes: Calculate the vector field based on the normal vector of each target ellipsoid and the position of each target ellipsoid; Construct and solve the Poisson equation to determine the indicator function corresponding to the vector field; Based on the indicator function corresponding to the vector field, the isosurfaces where each target ellipsoid is located are determined; the values ​​of the indicator functions corresponding to each point on the isosurfaces are equal; Based on the isosurfaces where each target ellipsoid is located, a second three-dimensional model of the target scene is constructed.

4. The method according to claim 1, characterized in that, The rendering of the second 3D model of the target scene to obtain a smooth depth map of the target scene includes: The second 3D model of the target scene is rendered to obtain the original depth map of the target scene; The original depth map is filtered to obtain a smooth depth map of the target scene.

5. The method according to claim 1, characterized in that, The target scene is a pre-defined large-area scene; the method further includes: Obtain the target color map rendered based on the first 3D model; Obtain other depth maps and other color maps of other scenes rendered using non-3DGS technology; the other scenes are preset small-area scenes, and the smooth depth map, the target color map, the other depth maps, and the other color maps are rendered from the same angle; The target color map and the other color maps are fused to obtain a fused image through the following methods: For each pixel, if the depth value of the pixel in the smooth depth map is greater than the depth value of the pixel in other depth maps, then the pixel value of the pixel in the other color maps is set as the pixel value of the pixel in the blended image; if the depth value of the pixel in the smooth depth map is less than the depth value of the pixel in other depth maps, then the pixel value of the pixel in the target color map is set as the pixel value of the pixel in the blended image.

6. The method according to claim 1, characterized in that, The step of determining vectors originating from the target ellipsoid and ending at each of its nearest neighbor ellipsoids to obtain multiple position vectors corresponding to the target ellipsoid includes: By determining vectors that start from the center point of the target ellipsoid and end at the center points of each of the target ellipsoid's nearest neighbor ellipsoids, multiple position vectors corresponding to the target ellipsoid are obtained. The step of determining other ellipsoids in the first three-dimensional model whose distance from the target ellipsoid is less than a preset distance threshold as neighboring ellipsoids includes: In the first three-dimensional model, the nearest neighbor search algorithm KDTree is used to determine other ellipsoids corresponding to the center point whose distance from the center point of the target ellipsoid is less than a preset distance threshold, and these are taken as the nearest neighbor ellipsoids.

7. A depth map determining device, characterized in that, The device includes: The first 3D model acquisition module is used to acquire the first 3D model of the target scene based on 3DGS, and the first 3D model contains multiple ellipsoids. The nearest neighbor ellipsoid determination module is used to determine, for each target ellipsoid, other ellipsoids in the first three-dimensional model whose distance from the target ellipsoid is less than a preset distance threshold, as nearest neighbor ellipsoids. The target ellipsoid can be any number of ellipsoids in the first three-dimensional model. The position vector determination module is used to determine, for each target ellipsoid, a vector starting from the target ellipsoid and ending at each of its nearest neighbor ellipsoids, thereby obtaining multiple position vectors corresponding to the target ellipsoid. The covariance matrix determination module is used to determine the covariance matrix corresponding to each target ellipsoid based on the position vector corresponding to each target ellipsoid. The value of the i-th row and j-th column of the covariance matrix is ​​the covariance between the i-th and j-th dimensions of the position vector corresponding to each target ellipsoid. The planar vector determination module is used to determine, for each target ellipsoid, the eigenvectors corresponding to two target eigenvalues ​​based on the covariance matrix corresponding to the target ellipsoid, as two planar vectors corresponding to the target ellipsoid, wherein the two target eigenvalues ​​are the first two eigenvalues ​​in the order of the eigenvalues ​​of the covariance matrix corresponding to the target ellipsoid from largest to smallest; The surface shape determination module is used to determine the shape of the surface of the target scene at the target ellipsoid based on the plane formed by the two plane vectors corresponding to the target ellipsoid for each target ellipsoid. The second three-dimensional model acquisition module is used to perform three-dimensional reconstruction of the target scene based on the shape of the surface of the target scene at each of the target ellipsoids, and obtain the second three-dimensional model of the target scene; The smooth depth map acquisition module is used to render the second three-dimensional model of the target scene to obtain a smooth depth map of the target scene.

8. The apparatus according to claim 7, characterized in that, The surface shape determination module determines the shape of the surface of the target scene at the target ellipsoid based on the plane formed by the two planar vectors corresponding to the target ellipsoid, including: The normal vector of the plane formed by the two plane vectors corresponding to the target ellipsoid is taken as the normal vector of the target ellipsoid; the normal vector is used to characterize the shape of the surface of the target scene at the target ellipsoid. The second 3D model acquisition module reconstructs the target scene in 3D based on the shape of the surface of the target scene at each of the target ellipsoids, to obtain a second 3D model of the target scene, including: Calculate the vector field based on the normal vector of each target ellipsoid and the position of each target ellipsoid; Construct and solve the Poisson equation to determine the indicator function corresponding to the vector field; Based on the indicator function corresponding to the vector field, the isosurfaces where each target ellipsoid is located are determined; the values ​​of the indicator functions corresponding to each point on the isosurfaces are equal; Based on the isosurfaces where each of the target ellipsoids is located, a second three-dimensional model of the target scene is constructed; The smooth depth map obtaining module renders the second 3D model of the target scene to obtain a smooth depth map of the target scene, including: The second 3D model of the target scene is rendered to obtain the original depth map of the target scene; The original depth map is filtered to obtain a smooth depth map of the target scene; The target scene is a pre-defined large-area scene; the device also includes: The target color map acquisition module is used to acquire the target color map rendered based on the first 3D model. The other depth map acquisition module is used to acquire other depth maps and other color maps of other scenes rendered based on non-3DGS technology; the other scenes are preset small-area scenes, and the smooth depth map, the target color map, the other depth map and the other color map are rendered from the same angle; The target color map and the other color maps are fused to obtain a fused image through the following methods: For each pixel, if the depth value of the pixel in the smooth depth map is greater than the depth value of the pixel in other depth maps, then the pixel value of the pixel in the other color maps is set as the pixel value of the pixel in the blended image; if the depth value of the pixel in the smooth depth map is less than the depth value of the pixel in other depth maps, then the pixel value of the pixel in the target color map is set as the pixel value of the pixel in the blended image. The position vector determination module determines vectors originating from the target ellipsoid and ending at each of its nearest neighbor ellipsoids, thus obtaining multiple position vectors corresponding to the target ellipsoid, including: By determining vectors that start from the center point of the target ellipsoid and end at the center points of each of the target ellipsoid's nearest neighbor ellipsoids, multiple position vectors corresponding to the target ellipsoid are obtained. The nearest neighbor ellipsoid determination module determines other ellipsoids in the first three-dimensional model whose distance from the target ellipsoid is less than a preset distance threshold, as nearest neighbor ellipsoids, including: In the first three-dimensional model, the nearest neighbor search algorithm KDTree is used to determine other ellipsoids corresponding to the center point whose distance from the center point of the target ellipsoid is less than a preset distance threshold, and these are taken as the nearest neighbor ellipsoids.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method of any one of claims 1-6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-6.

Images