Large-scale scene reconstruction method based on pixel perception Gaussian adaptive density control

Abstract

The embodiment of the invention discloses a large-scale scene reconstruction method based on pixel perception Gaussian adaptive density control, and relates to the technical field of smart cities. Calculating weighted average gradients of Gaussian points in different views based on coverage conditions of pixels in the views, and guiding cloning and splitting operations of the Gaussian points in an initial point cloud sparse region; gradient contributions of Gaussian points in different depth areas are adjusted through gradient field scaling factors, and the floating artifact phenomenon caused by viewpoint coverage range changes is avoided; and in combination with a distributed processing framework, performing partition optimization and unit combination on a large-scale scene, and finally generating a three-dimensional scene through Gaussian distribution after accumulation optimization. The method is suitable for the field of three-dimensional reconstruction of large-scale complex scenes, and is especially suitable for large-scale scenes needing high-precision and rapid reconstruction. The method can solve the problems of insufficient precision and low efficiency when an existing three-dimensional reconstruction technology is used for processing an initial point cloud sparse region in a large-scale complex scene.

Description

Technical Field

[0001] The present invention relates to the field of smart city technology, and in particular to a large-scale scene reconstruction method based on pixel-aware Gaussian adaptive density control. Background Art

[0002] With the widespread application of 3D reconstruction technology in urban planning, virtual reality, augmented reality, and other fields, the demand for high-precision reconstruction of large-scale, complex scenes is growing. Traditional methods face numerous challenges in reconstructing large-scale, complex scenes. Existing 3D reconstruction methods still suffer from low computational efficiency and high memory consumption when processing large-scale scenes. The growth mechanism is difficult to effectively trigger in sparse areas of the initial point cloud, resulting in loss of detail, blurring, and artifacts.

[0003] Therefore, how to solve the problem of insufficient reconstruction of 3DGS in the initial sparse point cloud area and the limitation of video memory in large-scale scene reconstruction, so as to reduce blur and needle artifacts to improve the accuracy of scene details and speed up reconstruction efficiency, has become a topic that needs further research. Summary of the Invention

[0004] An embodiment of the present invention provides a large-scale scene reconstruction method based on pixel-aware Gaussian adaptive density control, which can solve the problem of insufficient reconstruction of 3DGS in the initial sparse point cloud area, thereby reducing blur and needle artifacts to improve the accuracy of scene details, and at the same time improve the reconstruction efficiency of large-scale scenes.

[0005] To achieve the above objectives, the embodiments of the present invention adopt the following technical solutions:

[0006] In a first aspect, an embodiment of the present invention provides a method comprising:

[0007] S1. Collect the original data of the scene and process it to obtain a sparse point cloud of the corresponding scene and align and adjust the direction of the point cloud; wherein the original data is image data. In actual applications, a photo of the target scene is taken using a drone-mounted camera, etc., and used as the original data.

[0008] S2. Divide the scene into cells and assign point clouds; wherein, "partitioning" in this embodiment refers to dividing the entire scene into multiple areas in space, usually based on geographical distribution or projected camera position; a cell is the smallest independent optimization unit that can be obtained after partitioning, and is the basic unit of parallel optimization. After optimization, all cells are merged to obtain a complete scene. Theoretically, a partition includes at least one cell. The unit obtained by segmenting the scene data refers to a progressive scene partitioning strategy based on the photos of the scene, and the scene is divided into multiple cells

[0009] S3, performing distributed 3D reconstruction based on pixel-aware Gaussian adaptive density control for the point cloud of each cell;

[0010] S4. Merge the 3D Gaussian ellipsoid distributions of each cell and delete the overlapping parts to obtain a new perspective image of the scene.

[0011] Specifically, S1 includes: according to the image data set {I t |t=1,2,...,T} obtain the sparse point cloud P and determine each image I t The initial camera pose ξ t ; The Z axis of point cloud P is aligned by performing Manhattan world alignment so that the Z axis of point cloud P is perpendicular to the ground plane.

[0012] Specifically, S2 includes: S21. Dividing the scene into a plurality of cells based on the projection coordinates of cameras arranged in the scene, wherein the scene data is progressively partitioned, and the scene is divided into a plurality of cells using a progressive scene partitioning strategy. S22. For a cell, determining whether the camera view of an adjacent cell is selected by checking a visibility index, wherein the point cloud covered by the camera view of the selected adjacent cell is added to the cell.

[0013] Among them, S21 includes: dividing the scene into |V| / (m×n) views, where the boundary of the i-th cell is rectangle, Denote the length and width of the rectangle respectively, |V| represents the total number of views, and the ground plane of the scene is divided into m parts along one axis according to the projection position. Each of these m parts is subdivided into n smaller parts along the other axis. The original boundary of the i-th cell is expanded according to the ratio λ to obtain a size of The point cloud in the expanded i-th cell is {P i |i=1,2,3,…,m×n}. Specifically, the cell contains the view and point cloud of the corresponding camera assigned to it.

[0014] S22 includes: Select Greater than the predefined threshold T h The camera view of the neighboring cells of , where A proj Representing an image The projected area of ​​the jth unit in ( It can also be understood as the actual image data of the scene captured by the camera). is an image The area of ​​pixels in , cell j is adjacent to cell i, Represent the width and height of the image respectively, and t represents the number of the currently processed image in the sequence.

[0015] Specifically, S3 includes: S31, establishing a distributed processing subtask for each cell; S32, using 3D Gaussian primitives {G k (x)|k=1,...,K} describes the sparse point cloud of the cell, G k (x) represents a 3D Gaussian function, G k The color c, opacity α, center point u, scaling matrix s, and rotation matrix R parameters of (x) are optimized through continuous iteration. K represents the number of Gaussian basis points. The loss function used in the optimization process is: It represents the mean absolute error between the predicted value and the true value of the model on the data set. It is a loss function based on structural similarity, D-SSIM is the inverse measure of SSIM, and λ is the weight factor; S33, according to the number of pixels covered by the Gaussian point in the camera view, performs Gaussian adaptive density control on the sparse point cloud of the cell.

[0016] Among them, S33 includes: detecting whether the Gaussian point of the cell is split or cloned, and the detection method includes: When the gradient (to the left of the > sign) is greater than τ pos The split operation is performed when the gradient is less than τ pos When , Gaussian points perform cloning. M i is the number of viewpoints that Gaussian point i participates in one cycle, where viewpoint refers to the camera view, i.e., the viewpoint and position of the camera; is the normalized device coordinate, where “normalized device coordinate” is a mathematical coordinate system used to describe the position of points in the scene in computer graphics; τ pos Is the threshold for determining whether a point is split or cloned, L k is a measure of the difference between the rendered image and the real image at the kth viewpoint, and Respectively represent the gradient of Gaussian point i with respect to the x and y directions under the current viewpoint k, is the number of pixels covered by Gaussian point i at viewpoint k, and f(i,k) represents the scaling factor. Specifically, the camera views and Gaussian basis cells assigned to the cell are processed. During each optimization cycle, the density of Gaussian points is dynamically adjusted by calculating the gradient contribution of Gaussian points in each view.

[0017] Furthermore, the opacity value of the Gaussian primitive is adjusted after each cloning. The new opacity value The relationship with the original opacity α is:

[0018] In a second aspect, an embodiment of the present invention provides a device comprising:

[0019] The preprocessing module is used to collect the original data of the scene and process it to obtain the sparse point cloud of the corresponding scene and align and adjust the direction of the point cloud;

[0020] The scene division module is used to divide the scene into cells and allocate point clouds;

[0021] The reconstruction module is used to perform distributed 3D reconstruction based on pixel-aware Gaussian adaptive density control for the point cloud of each cell;

[0022] The merging module is used to merge the 3D Gaussian ellipsoid distributions of each cell and delete the overlapping parts to obtain a new perspective image of the scene.

[0023] Specifically, the scene division module is specifically configured to divide the scene into a plurality of cells according to the projection coordinates of the cameras arranged in the scene; for a cell, determine whether the camera view of the adjacent cell is selected by checking the visibility index, wherein the point cloud covered by the camera view of the selected adjacent cell is added to the cell;

[0024] The reconstruction module is further configured to perform Gaussian adaptive density control on the sparse point cloud of the cell according to the number of pixels covered by the Gaussian point in the camera view.

[0025] The large-scale scene reconstruction method based on pixel-aware Gaussian adaptive density control provided by an embodiment of the present invention calculates the weighted average gradient of Gaussian points in different views based on the coverage of pixels in the view, and guides the cloning and splitting operations of Gaussian points in the sparse areas of the initial point cloud; adjusts the gradient contribution of Gaussian points in different depth areas by the gradient field scaling factor to avoid floating artifacts caused by changes in viewpoint coverage; combines with a distributed processing framework to perform partition optimization and unit merging on large-scale scenes, and finally generates a three-dimensional scene by accumulating the optimized Gaussian distribution. The present invention is applicable to the field of three-dimensional reconstruction of large-scale complex scenes, and is particularly applicable to large-scale scenes that require high-precision and fast reconstruction. It can solve the problems of insufficient accuracy and low efficiency of existing three-dimensional reconstruction technologies when processing sparse areas of initial point clouds in large-scale complex scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0027] Figure 1 A schematic diagram of a large-scale scene 3D reconstruction framework provided by an embodiment of the present invention;

[0028] Figure 2 A distributed reconstruction flow chart based on an improved Gaussian adaptive density control method provided by an embodiment of the present invention;

[0029] Figure 3 A visual comparison diagram of scenes reconstructed by the present invention and the prior art provided in an embodiment of the present invention;

[0030] Figure 4 A comparison chart of indicators of various zones in a residential area scenario in a specific example provided by an embodiment of the present invention;

[0031] Figure 5 A diagram showing the speedup ratios achieved in various partitions by the solution of the present invention and the existing 3DGS provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0032] To enable those skilled in the art to better understand the technical solutions of the present invention, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. The embodiments of the present invention will be described in detail below, with examples of the embodiments illustrated in the accompanying drawings. Throughout, identical or similar reference numerals represent identical or similar elements or elements having identical or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and intended only to explain the present invention and are not to be construed as limiting the present invention. Those skilled in the art will appreciate that, unless otherwise stated, the singular forms "a," "an," "said," and "the" used herein may also include the plural forms. It should be further understood that the term "comprising" as used in the description of the present invention refers to the presence of the stated features, integers, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when an element is referred to as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or intervening elements may be present. Furthermore, "connected" or "coupled" as used herein may include wireless connections or couplings. The term "and / or" as used herein includes any and all combinations of one or more associated listed items. It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art, and will not be interpreted in an idealized or overly formal sense unless defined as such herein.

[0033] An embodiment of the present invention provides a large-scale scene reconstruction solution based on pixel-aware Gaussian adaptive density control, which is mainly used to achieve fast and high-quality three-dimensional reconstruction of large-scale complex scenes. The main body of the technical solution mainly includes the following four links: the first link is to use the point cloud reconstruction method of recovering structure from motion to generate a sparse point cloud and estimate the camera pose, and adjust the point cloud direction through Manhattan world alignment; the second link is to divide the scene into multiple cells through a progressive scene partitioning strategy, allocate point clouds and expand the boundaries, use visibility standards to select camera views of adjacent cells, and expand the point cloud set; the third link is to use distributed processing technology in each cell to adopt a pixel-aware Gaussian adaptive density control method to optimize 3D Gaussian basis element parameters, finely control point cloud growth, and improve reconstruction quality and efficiency; the fourth link is to merge the 3D Gaussian ellipsoid distributions of each cell, delete overlapping parts, synthesize realistic new perspective images, and achieve seamless splicing of the overall scene.

[0034] like Figure 1 As shown, the technical solution of this embodiment can be specifically implemented as follows:

[0035] Step 1: Reconstruct sparse point clouds of large-scale scenes;

[0036] Using the structure-from-motion method to recover the input image set {I t |t=1,2,...,T}, and generate a sparse point cloud P for each image I t Estimate the initial camera pose ξ t , adjust the Z axis of point cloud P to be perpendicular to the ground plane by performing Manhattan world alignment.

[0037] Step 2: Progressive partitioning of scene data. Divide the scene into multiple cells using a progressive scene partitioning strategy. The specific method is as follows:

[0038] Step 2.1: Region division based on camera projection coordinates. Divide the scene according to the camera position projected on the ground plane. First, divide the ground plane along one axis into m parts based on the projection position. Each part contains approximately |V| / m views, where |V| represents the total number of views. Then, each of these parts is further subdivided into n smaller parts along the other axis, each containing |V| / (m×n) views. Define the boundary of the i-th cell of the division as The original boundary is expanded by a certain ratio λ to obtain a larger rectangle with a size of The other areas are divided in the same way. The training view V and point cloud P after expanding the boundary of each cell are assigned to each cell, and the point cloud in the cell is defined as {P i |i=1,2,3,…,m×n}.

[0039] Step 2.2, Visibility-based camera selection and point cloud expansion, in order to improve the reconstruction quality, we introduce visibility-based camera selection and coverage-based point cloud expansion, adding more relevant camera data according to the visibility standard. Given a cell i, by checking the visibility To determine whether the camera view of the adjacent cell j meets the requirements for being selected, the calculation formula is as follows:

[0040]

[0041] A proj is an image The projected area of ​​the jth unit in , is an image The area of ​​pixels in pixels is calculated by dividing the width of the image by and height Multiply them together to get .

[0042] choose Greater than the predefined threshold T h The camera view of the i-th unit is added to the unit i. More related cameras are added to the camera set V of the i-th unit i Then, V i The point cloud covered by all views in P is added i In the process, the final point cloud in unit i is further expanded to the final point cloud set

[0043] Step 3: Distributed 3D reconstruction based on pixel-aware Gaussian adaptive density control;

[0044] Step 3.1, distributed processing. Based on the partition results obtained in step 2, the present invention uses distributed processing technology to synchronously carry out 3DGS reconstruction work for each partition of the large scene, and uses Python's multiprocessing module to achieve parallel optimization of the improved 3DGS algorithm. Figure 2 The complete process of distributed three-dimensional reconstruction of the present invention is demonstrated in detail. The scene data of large scene point cloud and camera view data are divided into different units through a progressive partitioning method. Each subtask corresponds to a unit, and a corresponding task information file is generated. These task files are distributed to each Device node by the Host. After receiving the task file, the Device node will independently call the 3DGS reconstruction technology based on pixel-aware Gaussian adaptive density control to perform distributed processing on the assigned unit data. This method reduces the transmission volume of large-scale data to a certain extent. At the same time, it comprehensively covers all tasks to be processed, eliminates the occurrence of task omissions, and thus ensures the integrity of the task.

[0045] Step 3.2, Gaussian element parameter optimization, using 3D Gaussian element {G k (x)|k=1,...,K} further describes the sparse point cloud of the scene, and each 3D Gaussian G k The color c, opacity α, center point u, scaling matrix s and rotation matrix R parameters of (x) are optimized through continuous iteration, and the Gaussian primitives are optimized using stochastic gradient descent technology, which is based on continuous rendering of the image.

[0046] The loss function is defined as a combination of the reconstruction error and the regularization term:

[0047]

[0048] It represents the mean absolute error between the predicted value and the true value of the model on the data set. It is a loss function based on structural similarity. D-SSIM is the inverse measure of SSIM, which is used to measure the structural dissimilarity between images. λ is the weight factor.

[0049] Step 3.3: Gaussian adaptive density control based on pixel perception;

[0050] Consider the number of pixels covered by the Gaussian point in each view and use these pixel numbers as weights to calculate the weighted average gradient, assigning different weights to the gradient contribution of each viewpoint. The improved equation for determining whether a Gaussian point is split or cloned is given by the following formula:

[0051]

[0052] M i is the number of viewpoints that Gaussian point i participates in one cycle, is the normalized device coordinate, τ pos Is the threshold for determining whether a point is split or cloned, L k is a measure of the difference between the rendered image and the real image at the kth viewpoint, and Respectively represent the gradient of Gaussian point i with respect to the x and y directions under the current viewpoint k, is the number of pixels covered by Gaussian point i under viewpoint k.

[0053] The floating objects generated during the rendering process of the point cloud near the camera occupy a large screen space, and their normalized device coordinates have a significant gradient, resulting in an increasing number of floating objects during the point cloud growth process. To solve this problem, the present invention performs gradient field scaling to solve this problem. Specifically, a circular area is defined to determine the scaled scene size, where the radius is calculated by the following formula:

[0054]

[0055] In the training set, N represents the number of viewpoints, C i and C j represents the coordinates of the camera of the i-th and j-th viewpoints in the world coordinate system, and κ is the gradient field scaling coefficient.

[0056] Then calculate the scaling factor of the NDC coordinate gradient of each Gaussian i at the kth viewpoint. The calculation formula of the scaling factor f(i,k) can be expressed as:

[0057]

[0058] is the z coordinate of Gaussian i in the camera coordinate system at the kth viewpoint, indicating the depth of Gaussian i from the viewpoint, γ depth is a manually set hyperparameter. R represents the radius of the scene. The clip() function is used to constrain a value to a specific range. For example, clip(x,a,b) will constrain the input x to the interval [a,b]. If x is less than a, it outputs a; if x is greater than b, it outputs b; otherwise, it outputs the value of x itself.

[0059] The final calculation equation that ultimately determines whether a Gaussian point is split or cloned is given by the following equation:

[0060]

[0061] In the Gaussian density control process, when a Gaussian primitive is cloned, its opacity value is also retained, which will cause the influence of the cloned Gaussian primitive on the final rendered color to be over-weighted. To solve this problem, in the improved adaptive density control module of the large-scale scene reconstruction architecture of the present invention, a correction method is introduced for the opacity of the newly generated Gaussian primitives, that is, the opacity value of the Gaussian primitive is adjusted after cloning to eliminate this deviation. The new opacity value The relationship with the original opacity α is:

[0062]

[0063] Step 4: Unit merging and new perspective synthesis;

[0064] After independently optimizing all cells, the best 3D Gaussian ellipsoid distributions obtained for each training partition are merged. To avoid ghosting between overlapping parts of different training views during the merging process, all 3D Gaussian ellipsoids outside the original region of each optimized cell are deleted before merging these cells. The 3D Gaussian ellipsoids in these non-overlapping cells are then merged. Finally, a realistic new viewpoint image is synthesized using these merged 3D Gaussian ellipsoid distributions. This stage achieves seamless stitching from local cells to the entire scene.

[0065] The beneficial effects of the technical solution of this embodiment are exemplified below by specific examples:

[0066] The large-scale scene 3D reconstruction method proposed in this invention is trained and evaluated on a server equipped with 8 GeForce RTX 4090 GPUs. The server configuration used in the experiment has a graphics card with 24GB video memory and a CPU of Core(TM) i9-14900K@3.20GHz, Windows 11 system, 128GB memory, based on Python 3.10.6, Pytorch deep learning framework, and CUDA 11.8 acceleration library for parallel acceleration. Visibility threshold T h The threshold τ is 0.25 to determine whether the Gaussian point is split or cloned. pos The value of 0.0002 indicates that each cell in the proposed method was optimized for 60,000 iterations. Densification started at iteration 500 and ended at iteration 30,000, with an interval of 200 iterations. All other settings were the same as for 3DGS. To ensure the validity of the experiment, 3DGS was optimized for 500,000 iterations.

[0067] In terms of datasets, this embodiment conducts experiments on the public datasets Mill-19 and UrbanScene3D datasets to verify the effectiveness of the present invention.

[0068] This embodiment compares the present invention with existing three-dimensional reconstruction methods, including 3DGS and VSGS reconstruction methods.

[0069] like Figure 3As shown, when processing high-frequency areas such as grass and trees, the 3DGS method uses a small number of large Gaussian primitives, the gradient accumulation change is low, and it is difficult to trigger the growth mechanism, resulting in insufficient details, blurring and blocky artifacts in the image. The VSGS method adopts a spatially aware visibility standard when segmenting data, taking into account the camera's visibility to specific areas, reducing boundary artifacts between units and floating artifacts in the air, and improving the reconstruction quality of high-frequency areas. However, due to the use of the Gaussian adaptive density control module of 3DGS, the image still has blurring and stripe artifacts. The present invention adopts a large-scale scene three-dimensional reconstruction method based on pixel-aware Gaussian adaptive density control, adds pixel-aware gradients and gradient field scaling strategies on the basis of 3DGS, and effectively reduces the floating point phenomenon near the camera. This method promotes point cloud growth in sparse areas of the initial point cloud, improves reconstruction details and accuracy, solves the problems of large-area blurring and blocky artifacts, and has better reconstruction effects than 3DGS and VSGS, and the new perspective image is close to the real image.

[0070] In the 3D reconstruction of residential areas, the comparison results of the partition indicators of the present invention and the VSGS reconstruction method are as follows: Figure 4 As shown in the figure. Among the eight partitions compared with the VSGS method, the PSNR index of six partitions is better than that of the VSGS method, and the SSIM and LPIPS indexes of all partitions are better than those of the VSGS method. In addition, the average PSNR, SSIM, and LPIPS of all partitions of the present invention are 25.61, 0.889, and 0.175, respectively, while the average PSNR, SSIM, and LPIPS of all partitions of the VSGS method are 25.19, 0.860, and 0.221, respectively. The average values ​​of all evaluation indicators of the present invention are superior to those of the VSGS method.

[0071] In order to demonstrate the effectiveness of the present invention, the PSNR, SSIM and LPIPS indicators of each method are shown in Table 1, Table 2 and Table 3 respectively.

[0072] Table 1

[0073]

[0074] The average PSNR values ​​across all datasets show an overall average improvement of 1.38% compared to the VSGS method, and an average improvement of 2.98% compared to the 3DGS method. The proposed method synthesized virtual viewpoint images from five different scenes in a large-scale scene dataset that were closer to real images than those synthesized by other methods.

[0075] Table 2

[0076]

[0077]

[0078] The SSIM numerical results of the virtual viewpoint images synthesized by the present invention outperformed those of other methods across four datasets. The average SSIM values ​​for all datasets showed an average improvement of 2.69% over the VSGS method and 3.59% over 3DGS. These numerical results demonstrate that the brightness, contrast, and structure of the virtual viewpoint images synthesized by the present invention for five different scenes in a large-scale scene dataset are closer to the real images than those produced by other methods.

[0079] Table 3

[0080]

[0081] The proposed method achieved optimal IPIPS values ​​across all five datasets. The average IPIPS value across all datasets was superior to that of other methods. Specifically, compared to the VSGS method, the proposed method achieved an overall average reduction of 11.49% in IPIPS values, and compared to the 3DGS method, the proposed method achieved an average reduction of 17.2%. These results demonstrate that the novel perspective images generated by the proposed large-scale reconstruction method exhibit high accuracy in human subjective perception.

[0082] The reconstruction speed of the present invention is compared with that of the 3DGS reconstruction method in different scenes and different partitions.

[0083] The performance of the method of the present invention is evaluated by calculating the speedup ratio, which is obtained by dividing the total time of 3DGS training by the time of each partition of the method of the present invention, reflecting the performance improvement of the present invention over the 3DGS reconstruction method.

[0084] The present invention achieves acceleration in each scene and each partition, such as Figure 4 As shown in the figure, the average speedup ratio of the 3D reconstruction of the campus scene is 5.67, with a maximum value of 6.52 and a minimum value of 3.96. The average speedup ratio of the 3D reconstruction of the residential area scene is 4.79, with a maximum value of 5.27 and a minimum value of 5.11. The average speedup ratio of the 3D reconstruction of the Science and Art Museum scene is 5.18, with a maximum value of 6.52 and a minimum value of 3.96. The average speedup ratio of the 3D reconstruction of the ruins scene is 5.25, with a maximum value of 5.68 and a minimum value of 4.56. The average speedup ratio of the 3D reconstruction of the building scene is 5.18, with a maximum value of 6.14 and a minimum value of 4.46.

[0085] Through comprehensive analysis of the acceleration ratios of five scenes, the average acceleration ratio achieved by the present invention in all scenes is 5.21. The research results prove that the present invention is highly efficient in three-dimensional reconstruction of large-scale scenes.

[0086] In general, the technical solution of this embodiment has the following advantages over the existing 3DGS method:

[0087] 1. By calculating the gradient by weighted pixel coverage, the problem of insufficient reconstruction of 3DGS in the initial sparse point cloud area is solved, which significantly reduces blur and needle artifacts and improves the accurate representation of scene details.

[0088] 2. Combining progressive partitioning with distributed processing strategies, the present invention can efficiently handle large-scale 3D reconstruction tasks. Through distributed computing, it achieves an average speedup ratio of 5.21 times in multiple large scene data sets, significantly improving reconstruction efficiency.

[0089] 3. The above experimental results show that the proposed method outperforms existing methods (such as 3DGS and VSGS) in key indicators such as PSNR, SSIM and LPIPS, and the generated virtual viewpoint images are closer to real scenes in terms of brightness, contrast, structural similarity and human perception similarity.

[0090] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited to this. Any changes or replacements that can be easily thought of by any technician familiar with this technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A large-scale scene reconstruction method based on pixel-aware Gaussian adaptive density control, characterized in that: include: S1. Collect the original data of the scene and process it to obtain a sparse point cloud of the corresponding scene and align and adjust the direction of the point cloud; S2, divide the scene into cells and assign point clouds; S3, performing distributed 3D reconstruction based on pixel-aware Gaussian adaptive density control for the point cloud of each cell; S4. Merge the 3D Gaussian ellipsoid distributions of each cell and delete the overlapping parts to obtain a new perspective image of the scene.

2. The method according to claim 1, characterized in that S1 includes: According to the image data set of the collected scene {I t |t=1,2,...,T} obtain the sparse point cloud P and determine each image I t The initial camera pose ξ t ; The Z axis of the point cloud P is aligned by performing a Manhattan world alignment so that the Z axis of the point cloud P is perpendicular to the ground plane.

3. The method according to claim 1, characterized in that S2 include: S21, dividing the scene into a plurality of cells according to the projection coordinates of the cameras arranged in the scene; S22. For a cell, determine whether the camera view of the adjacent cell is selected by checking the visibility index, wherein the point cloud covered by the camera view of the selected adjacent cell is added to this cell.

4. The method according to claim 3, characterized in that S21 includes: The scene is divided into |V| / (m×n) views, where the boundary of the i-th cell is rectangle, Denote the length and width of the rectangle respectively, |V| represents the total number of views, and the ground plane of the scene is divided into m parts along one axis according to the projection position. Each of these m parts is subdivided into n smaller parts along the other axis; The original boundary of the i-th cell is expanded according to the ratio λ, and a size of The point cloud in the expanded i-th cell is {P i |i=1,2,3,…,m×n}.

5. The method according to claim 4, characterized in that S22 includes: choose Greater than the predefined threshold T h The camera view of the neighboring cells of , where A proj Representing an image The projected area of ​​the jth unit in , is an image The area of ​​pixels in the cell j is adjacent to the cell i, W t j 、 Represent the width and height of the image respectively, and t represents the number of the currently processed image in the sequence.

6. The method according to claim 1, characterized in that S3 include: S31. Establish distributed processing subtasks for each cell; S32, through 3D Gaussian basis element {G k (x)|k=1,...,K} describes the sparse point cloud of the cell, G k (x) represents a 3D Gaussian function, G k The color c, opacity α, center point u, scaling matrix s, and rotation matrix R parameters of (x) are optimized through continuous iteration. K represents the number of Gaussian basis points. The loss function used in the optimization process is: It represents the mean absolute error between the predicted value and the true value of the model on the data set. It is a loss function based on structural similarity, D-SSIM is the inverse measure of SSIM, and λ is the weight factor; S33. Perform Gaussian adaptive density control on the Gaussian primitives of the cell according to the number of pixels covered by the Gaussian primitives in the camera view.

7. The method according to claim 6, characterized in that S33 includes: Detect whether the Gaussian points of a cell are split or cloned. The detection methods include: M i is the number of viewpoints that Gaussian point i participates in one cycle, is the normalized device coordinate, τ pos Is the threshold for determining whether a point is split or cloned, L k is a measure of the difference between the rendered image and the real image at the kth viewpoint, and Respectively represent the gradient of Gaussian point i with respect to the x and y directions under the current viewpoint k, is the number of pixels covered by Gaussian point i under viewpoint k, and f(i,k) represents the scaling factor.

8. The method according to claim 7, characterized in that Also includes: Adjust the opacity value of the Gaussian primitive after each clone is completed. The new opacity value The relationship with the original opacity α is:

9. A large-scale scene reconstruction method based on pixel-aware Gaussian adaptive density control, characterized in that: include: The preprocessing module is used to collect the original data of the scene and process it to obtain the sparse point cloud of the corresponding scene and align and adjust the direction of the point cloud; The scene division module is used to divide the scene into cells and allocate point clouds; The reconstruction module is used to perform distributed 3D reconstruction based on pixel-aware Gaussian adaptive density control for the point cloud of each cell; The merging module is used to merge the 3D Gaussian ellipsoid distributions of each cell and delete the overlapping parts to obtain a new perspective image of the scene.

10. The method according to claim 1, characterized in that The scene division module is specifically configured to divide the scene into a plurality of cells according to the projection coordinates of the cameras arranged in the scene; for a cell, determining whether the camera view of an adjacent cell is selected by checking the visibility index, wherein the point cloud covered by the camera view of the selected adjacent cell is added to the cell; The reconstruction module is further configured to perform Gaussian adaptive density control on the sparse point cloud of the cell according to the number of pixels covered by the Gaussian point in the camera view.

Images