Large-scale scene reconstruction method and system based on 3D Gaussian algorithm, electronic equipment and computer readable storage medium

Abstract

The invention provides a large-scale scene reconstruction method based on a 3D Gaussian algorithm. The method comprises the following steps: S1, obtaining a first common-view original image set; s2, generating a first sparse point cloud set by adopting an SfM algorithm, and segmenting the large scene into a plurality of first grid units; s3, expanding the boundary of the first grid unit to obtain a plurality of second grid units; s4, adding a new camera in the original camera set to obtain an optimized camera set; s5, optimizing each second sparse point cloud subset to obtain a corresponding third sparse point cloud subset; s6, performing 3D Gaussian point cloud distribution conversion, rasterization and appearance modeling optimization processing on each third sparse point cloud subset by adopting a 3D Gaussian sputtering model to obtain a plurality of primary rendering image sub-blocks; s7, constructing a comprehensive loss function to iterate each primary rendering image sub-block to obtain a plurality of advanced rendering image sub-blocks; and S8, reconstructing by using the plurality of advanced rendering image sub-blocks to obtain dense sub-scenes, and seamlessly merging the dense sub-scenes to obtain a reconstructed large scene.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and deep learning, and specifically relates to a large scene reconstruction method, system, electronic device and computer-readable storage medium based on a 3D Gaussian algorithm. Background Art

[0002] Traditional 3D reconstruction methods mainly use point clouds, meshes and other methods to achieve 3D scene reconstruction. Although the rendering speed is fast, it is slightly insufficient in presenting scene details. Fully implicit 3D reconstruction methods have excellent scene representation capabilities, but they take a lot of time for training and rendering. In order to take into account the advantages of both, deep learning technology has made significant progress in scene reconstruction recently, especially 3D Gaussian Splatting (3DGS), which is particularly outstanding in terms of the clarity of reconstruction results and fast rendering speed. 3DGS technology uses Gaussian functions and sparse point cloud data to model the scene, converts 3D information into the form of Gaussian functions to reconstruct large scene models and achieve real-time rendering. At present, traditional 3DG works well in small scenes and object-centric scenes, but encounters difficulties in reconstruction in large scenes. First, the number of 3D Gaussians is limited by a given video memory, and the rich details of a large scene require a large number of 3D Gaussians. Directly applying 3DGS to large scenes will result in low-quality reconstruction or insufficient memory. Therefore, how to use 3DGS to perform high-quality 3D reconstruction of large scenes has become one of the research hotspots. Summary of the invention

[0003] In order to solve the above technical problems, the first object of the present invention is to provide a large scene reconstruction method based on a 3D Gaussian algorithm, the method comprising:

[0004] S1 acquires a plurality of original images covering a large target scene area, and selects original images satisfying a common-view condition from the plurality of original images to form a first common-view original image set; the first common-view original image set includes a plurality of original images from different perspectives;

[0005] S2 generates a first sparse point cloud set based on the first common view original image set using the SfM algorithm, and calculates the point cloud boundary of the first sparse point cloud set; then divides the target large scene area into a plurality of first grid units according to the point cloud boundary and the position of the original camera set projected on the plane by the first common view original image set;

[0006] S3: expanding the boundary of the first grid unit to obtain a plurality of second grid units; each of the second grid units contains a second common view original image subset and a second sparse point cloud subset;

[0007] S4 adds a new camera to the original camera set corresponding to each second grid unit to obtain an optimized camera set corresponding to each second grid unit;

[0008] S5 optimizes each second sparse point cloud subset in turn according to the optimized camera set to obtain a corresponding third sparse point cloud subset;

[0009] S6 adopts a 3D Gaussian sputtering model to perform 3D Gaussian point cloud distribution conversion, rasterization and appearance modeling optimization processing on each third sparse point cloud subset in turn to obtain multiple primary rendering image sub-blocks;

[0010] S7 constructs a comprehensive loss function, and iteratively updates each primary rendered image sub-block in turn based on the comprehensive loss function until the loss function tends to be stable, thereby obtaining multiple advanced rendered image sub-blocks;

[0011] S8 uses the geometric information of multiple high-level rendered image sub-blocks to reconstruct dense sub-scenes, and seamlessly merges the dense sub-scenes to obtain the reconstructed target large scene.

[0012] Specifically, the 3D Gaussian sputtering model in step S6 includes a Gaussian point cloud distribution conversion unit, a differentiable Gaussian rasterization unit and a decoupled appearance modeling unit.

[0013] Specifically, step S6 further includes:

[0014] S61: using a Gaussian point cloud distribution conversion unit to sequentially convert the third sparse point cloud subset corresponding to each second grid unit into a group of primary 3D Gaussian point cloud distributions; and then convert each group of primary 3D Gaussian point cloud distributions into each primary 3D Gaussian image sub-block;

[0015] S62 projects and renders each primary 3D Gaussian image sub-block onto a 2D image plane corresponding to each camera pose using a differentiable Gaussian rasterization unit to obtain a plurality of primary predicted image sub-blocks;

[0016] S63 generates multiple primary transformation maps based on the multiple primary prediction image sub-blocks using a decoupled appearance modeling unit, and optimizes the appearance of each corresponding primary prediction image sub-block in turn according to each primary transformation map to obtain multiple primary rendering image sub-blocks.

[0017] Specifically, step S7 further includes:

[0018] S71 constructs a comprehensive loss function, wherein the comprehensive loss function includes a first loss function and a second loss function; wherein the first loss function includes a first difference loss function and a D-SSIM loss function, and the second loss function includes a second difference loss function;

[0019] S72, based on the multiple primary rendered image sub-blocks, iteratively updating the parameters of each primary 3D Gaussian image sub-block by using the first loss function until the first loss function converges, to obtain multiple advanced 3D Gaussian image sub-blocks;

[0020] S73 uses a differentiable Gaussian rasterization unit to render each high-level 3D Gaussian image sub-block onto a 2D image plane in sequence to obtain a plurality of high-level prediction image sub-blocks;

[0021] S74 generates multiple corresponding high-level transformation maps based on multiple high-level prediction image sub-blocks using a decoupled appearance modeling unit, and performs iterative appearance optimization on each high-level prediction image sub-block in turn according to each high-level transformation map and the second loss function until the second loss function converges to obtain multiple high-level rendering image sub-blocks.

[0022] Specifically, the first loss function in step S7 is:

[0023]

[0024] in, is a first difference loss function, used to calculate the absolute difference in pixel values ​​between the predicted image sub-block and the original image; is the D-SSIM loss function, which is used to calculate the structural similarity difference between the predicted image sub-block and the original image; Weights of the D-SSIM loss function

[0025] Specifically, the parameters of the 3D Gaussian point cloud distribution include a center point, a covariance matrix, color information, and opacity.

[0026] Specifically, the common viewing conditions in step S1 include: the satellite zenith angle of each original image is less than 30°, the angle between every two original images is less than 5°, and the shadow area of ​​each original image accounts for less than 5%.

[0027] The second object of the present invention is to provide a large scene reconstruction system based on a 3D Gaussian algorithm, the system comprising:

[0028] An image acquisition module is configured to acquire a plurality of original images covering a target large scene area, and select original images satisfying a common viewing condition from the plurality of original images to form a first common viewing original image set;

[0029] A sparse point cloud generation module is configured to generate a first sparse point cloud set based on the first common view original image set by using the SfM algorithm, and calculate a point cloud boundary of the first sparse point cloud set;

[0030] The large scene segmentation module further segments the target large scene area into a plurality of first grid units based on the point cloud boundary and the original camera position projected on the plane;

[0031] A boundary expansion module is configured to expand the boundary of the initial first grid unit to obtain a plurality of second grid units; each of the second grid units contains a second common view original image subset and a second sparse point cloud subset;

[0032] A camera optimization module is configured to add a new camera to an original camera set corresponding to each second grid unit based on an airspace perception visibility standard to obtain an optimized camera set corresponding to each second grid unit;

[0033] a sparse point cloud optimization module, configured to optimize the second sparse point cloud subset in each second grid unit in turn according to the optimized camera set to obtain a corresponding third sparse point cloud subset;

[0034] A 3D Gaussian transformation module is configured to use a 3D Gaussian sputtering model to sequentially perform 3D Gaussian point cloud distribution transformation, rasterization, and appearance modeling optimization processing on the third sparse point cloud subset corresponding to each second grid unit to obtain a plurality of primary rendering image sub-blocks;

[0035] The rendering image optimization module constructs a comprehensive loss function and iteratively updates each primary rendering image sub-block in turn based on the comprehensive loss function until the loss function tends to be stable, thereby obtaining multiple advanced rendering image sub-blocks;

[0036] The merging and reconstruction module is configured to reconstruct a dense sub-scene using geometric information of multiple high-level rendered image sub-blocks, and seamlessly merge the dense sub-scenes to obtain a reconstructed target large scene.

[0037] The third object of the present invention is to provide an electronic device, which includes: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the method described above.

[0038] A fourth object of the present invention is to provide a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the method described above.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] (1) The present invention distributes the original image and sparse point cloud in a large scene to different grid units, and sequentially performs 3D Gaussian point cloud distribution conversion, rasterization processing and appearance optimization on each independent grid unit to achieve reconstruction in a large scene and improve the reconstruction rate;

[0041] (2) The expansion of the boundaries of the grid cells in the present invention helps to include more visual information, thereby obtaining better results in local optimization and obtaining a more accurate 3D structure, providing richer geometric details for the subsequent 3D Gaussian point cloud distribution conversion and retaining more scene details;

[0042] (3) The present invention introduces decoupled appearance modeling to independently optimize the appearance of each predicted image sub-block, so that the final generated rendered image sub-block is more visually realistic while maintaining structural consistency; and introduces a comprehensive loss function for iterative optimization to reduce rendering errors. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 This is a technical flow chart of a large scene reconstruction method based on a 3D Gaussian algorithm in an embodiment of the present invention;

[0045] Figure 2 A flow technology framework diagram of a large scene reconstruction method based on a 3D Gaussian algorithm in an embodiment of the present invention;

[0046] Figure 3 Schematic diagram of the structure of a large scene reconstruction system based on a 3D Gaussian algorithm in an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present invention.

[0048] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0049] See also Figure 1 and Figure 2 , Figure 1This is a flow chart of a large scene reconstruction method based on a 3D Gaussian algorithm in an embodiment of the present invention; Figure 1 This is a technical flow chart of a large scene reconstruction method based on a 3D Gaussian algorithm in an embodiment of the present invention; the method comprises the following steps:

[0050] S1 acquires a plurality of original images covering a large target scene area, and selects original images satisfying a common-view condition from the plurality of original images to form a first common-view original image set; the first common-view original image set includes a plurality of original images from different perspectives;

[0051] In the embodiment of the present invention, the target large scene area is a large scene in the real world, including roads, buildings, ruins, woodlands, elevated roads and bridges, etc. The above scene data records multiple original images taken from different positions and angles in the large scene. These images include not only the front view, but also the side, top and other non-traditional view, which can capture the overall view and complex geometric structure of the scene. The common view conditions for shooting the original images include: the satellite zenith angle of each original image is less than 30°, the angle between each two original images is less than 5°, and the shadow area of ​​each original image accounts for less than 5%. 80 original images from different perspectives are obtained to form the first common view original image set.

[0052] S2 generates a first sparse point cloud set based on the first common view original image set using the SfM algorithm, and calculates the point cloud boundary of the first sparse point cloud set; then divides the target large scene area into a plurality of first grid units according to the point cloud boundary and the position of the original camera set projected on the plane by the first common view original image set;

[0053] S3 expands the boundary of the first grid unit to obtain a plurality of second grid units; each of the second grid units contains a second common view original image subset and a second sparse point cloud subset.

[0054] In an embodiment of the present invention, a first sparse point cloud set is first estimated and generated from a first common view original image set using a Structure from Motion (SfM) algorithm. The target large scene area is segmented according to the point cloud boundary of the first sparse point cloud set and the original camera position set projected on the plane by the first common view original image set, and the ground plane is divided into m parts along the x-axis, and then the m parts are divided into n segments along the y-axis in turn, each segment contains approximately 80 / (m×n) original images of different perspectives, and a segment is regarded as a first grid unit, and each first grid unit contains a similar number of original images of different perspectives. The boundary of each first grid unit is expanded by 10% to ensure that the original images between adjacent units overlap, and multiple second grid units are obtained. Each second grid unit contains a second common-view original image subset and a second sparse point cloud subset; each second common-view original image subset is composed of original images of different perspectives in each first grid unit and original images within an extended 10% range; the size of each original image is W×H, W is 1920, and H is 1080.

[0055] S4 adds a new camera to the original camera set corresponding to each second grid unit to obtain an optimized camera set corresponding to each second grid unit;

[0056] S5 optimizes each second sparse point cloud subset in turn according to the optimized camera set to obtain a corresponding third sparse point cloud subset.

[0057] In the embodiment of the present invention, for each second grid unit q i , construct an axis-aligned bounding box (AABB) based on its corresponding second sparse point cloud subset. The height direction of the axis-aligned bounding box extends to the distance between the highest point of the target large scene area and the ground plane, covering all possible airspaces in the second grid unit; project the bounding box to the candidate camera E i On the image plane, a convex hull region is formed, and the area of ​​the convex hull region is calculated. ; The visibility ratio refers to the ratio of the visible area of ​​a camera to a specific grid cell to the total area of ​​the camera image. The spatial perception calculation can cover the complete spatial area within the cell, avoiding the omission of important perspectives due to sparse surface points, thereby improving the reconstruction quality and reducing suspension artifacts. The calculation formula is: ,in The camera E is to be selected i The total image area of ​​the candidate camera E i If the visibility ratio of the current grid unit exceeds the predefined threshold, i.e., 25%, it will be added to the optimized camera set V of the second grid unit. i. And according to the boundary expansion of 10% of the grid cells, the newly added cameras are initially allocated. Traverse the remaining unoptimized original camera set, evaluate the visibility ratio one by one according to the above steps, and add the cameras that meet the predefined threshold, i.e. 25%, to V i Finally, the optimized camera set V for each unit i It contains two types of cameras: the original camera set within the 10% extended boundary and the newly added cameras filtered by spatial awareness visibility. The point cloud observed by the newly added camera (even if it is outside the second grid cell of the current unit) is added to the second sparse point cloud subset of the second grid cell to obtain the third sparse point cloud subset corresponding to the second grid cell.

[0058] S6 adopts a 3D Gaussian sputtering model to perform 3D Gaussian point cloud distribution conversion, rasterization and appearance modeling optimization processing on each third sparse point cloud subset in turn to obtain multiple primary rendering image sub-blocks.

[0059] In an embodiment of the present invention, the 3D Gaussian sputtering model includes a Gaussian point cloud distribution conversion unit, a differentiable Gaussian rasterization unit and a decoupled appearance modeling unit.

[0060] In this embodiment of the present invention, step S6 further includes:

[0061] S61 uses the Gaussian point cloud distribution conversion unit to convert each second grid unit q i The corresponding third sparse point cloud subset is converted into a set of primary 3D Gaussian point cloud distribution G i ; Then distribute each group of primary 3D Gaussian point clouds G i Transformed into each primary 3D Gaussian image sub-block.

[0062] In the embodiment of the present invention, each third sparse point cloud subset is sequentially mapped to the 3D Gaussian point cloud distribution space to generate multiple groups of primary 3D Gaussian point cloud distribution G i The parameters of the primary 3D Gaussian point cloud distribution include the center point coordinate μ, the covariance matrix ∑, the color value Y, and the opacity A. Primary 3D Gaussian point cloud In three-dimensional space, it is represented by the three-dimensional covariance matrix ∑ and the center point coordinates μ:

[0063]

[0064] Where x is the coordinate of a point in three-dimensional space; the three-dimensional covariance matrix ,in is the rotation matrix and S is the scaling matrix.

[0065] S62 uses a differentiable Gaussian rasterization unit to project and render each primary 3D Gaussian image sub-block onto a 2D image plane corresponding to each camera posture to obtain multiple primary predicted image sub-blocks.

[0066] In the embodiment of the present invention, each primary 3D Gaussian image sub-block is projected onto the 2D image plane corresponding to each camera pose, and the 2D Gaussian point cloud corresponding to the different camera poses is obtained; specifically, for a camera pose N, the projection transformation matrix from the world coordinate to the camera coordinate is M, the primary 3D Gaussian point cloud distribution corresponding to the primary 3D Gaussian image sub-block is projected onto the 2D image plane corresponding to the camera pose N, and the coordinates under the projected camera coordinates are calculated. And the covariance matrix for:

[0067]

[0068]

[0069] Where J is the Jacobian matrix of the projection transformation, which is used to convert the 3D vector into a 2D vector, and then realize the projection transformation from the world coordinate system to the camera coordinate system. The purpose of this transformation is to transform the primary 3D Gaussian image sub-block in the world coordinate system to the 2D image plane space in the camera coordinate system. The 2D Gaussian point cloud is then subjected to differentiable Gaussian rasterization. In the differentiable rasterization process, the 2D image plane is divided into multiple pixel areas. Each pixel area is responsible for processing the 2D Gaussian point cloud projected into its range, and spatial hashing or grid indexing is used to quickly determine the subset of the 2D Gaussian point cloud corresponding to each pixel in the pixel area. In each pixel area, the depth value of the Gaussian in the camera coordinate system is sorted from far to near to ensure the correct order of subsequent mixing; for each pixel, Alpha blending is used to cover the color value and opacity of all 2D Gaussian point clouds of the pixel in depth order to obtain the corresponding primary predicted image sub-block of size σW×βH. The feature value f of each pixel in the primary prediction image sub-block w,h Calculated by AlphaBlending, the calculation formula is:

[0070]

[0071] in, ; is the subset of the 2D Gaussian point cloud corresponding to the pixel; for The eigenvalue of the i-th 2D Gaussian point cloud in; for The opacity of the i-th 2D Gaussian point cloud in ; is the coordinate of a point in three-dimensional space; for The coordinates of the center point of the i-th 2D Gaussian point cloud in .

[0072] The differentiable Gaussian rasterization method can quickly render 3D Gaussian distributions onto a 2D image plane, sorting the Gaussians according to depth information to efficiently mix them. This process involves calculating the contribution of each Gaussian to the pixel color and accumulating these contributions to form the final pixel feature value. Differentiable Gaussian rasterization combines efficient rendering and neural network optimization capabilities, allowing high performance in complex image tasks.

[0073] S63 generates multiple primary transformation maps based on the multiple primary prediction image sub-blocks using a decoupled appearance modeling unit, and optimizes the appearance of each corresponding primary prediction image sub-block in turn according to each primary transformation map to obtain multiple primary rendering image sub-blocks.

[0074] In an embodiment of the present invention, the appearance of the primary predicted image sub-block is optimized by decoupling the appearance modeling unit, with the aim of eliminating appearance inconsistencies (such as brightness differences, color casts) caused by lighting changes, camera exposure differences, etc., thereby generating visually consistent primary rendered image sub-blocks.

[0075] First, each primary prediction image sub-block is Downsampling to a low resolution to obtain a downsampled image; the size of the downsampled image is and an appearance embedding with a length of t is introduced , and then embed the appearance The downsampled image of 3 channels is spliced ​​pixel by pixel to obtain the embedded image of 3+t channels. ;

[0076]

[0077] Then, each embedded image Input convolutional neural network CNN, CNN network gradually Upsample and generate Multiple primary transform maps with the same resolution Finally, use For each corresponding primary prediction image sub-block Perform appearance optimization to obtain multiple primary rendered image sub-blocks .

[0078] S7 constructs a comprehensive loss function, and iteratively updates each primary rendered image sub-block in turn based on the comprehensive loss function until the loss function tends to be stable, thereby obtaining multiple advanced rendered image sub-blocks.

[0079] In an embodiment of the present invention, step S7 further includes:

[0080] S71 constructs a comprehensive loss function, wherein the comprehensive loss function includes a first loss function and a second loss function; wherein the first loss function includes a first difference loss function and a D-SSIM loss function, and the second loss function includes a second difference loss function;

[0081] S72, based on the multiple primary rendered image sub-blocks, iteratively updating the parameters of each primary 3D Gaussian image sub-block by using the first loss function until the first loss function converges, to obtain multiple advanced 3D Gaussian image sub-blocks;

[0082] S73 uses a differentiable Gaussian rasterization unit to render each high-level 3D Gaussian image sub-block onto a 2D image plane in sequence to obtain a plurality of high-level prediction image sub-blocks;

[0083] S74 generates multiple corresponding high-level transformation maps based on multiple high-level prediction image sub-blocks using a decoupled appearance modeling unit, and performs iterative appearance optimization on each high-level prediction image sub-block in turn according to each high-level transformation map and the second loss function until the second loss function converges to obtain multiple high-level rendering image sub-blocks.

[0084] In this embodiment of the present invention, the first loss function in step S7 is:

[0085]

[0086] in, is a first difference loss function, used to calculate the absolute difference in pixel values ​​between the predicted image sub-block and the original image; is the D-SSIM loss function, which is used to calculate the structural similarity difference between the predicted image sub-block and the original image; is the weight of the D-SSIM loss function. The calculation formula is:

[0087]

[0088] in is the total number of predicted image sub-blocks, is the i-th predicted image sub-block, is the corresponding original image. Ensure that the color and brightness of the predicted image sub-block are aligned with the original image.

[0089] The calculation formula is:

[0090]

[0091] The structural similarity index (SSIM) is used to measure the consistency between the predicted image and the real image in texture, contrast, and structure to avoid loss of geometric details.

[0092] In this embodiment of the present invention, the second loss function in step S7 includes a second difference loss function: is a second difference loss function, which is used to calculate the absolute difference in pixel values ​​between the rendered image sub-block and the predicted image sub-block. The calculation formula is:

[0093]

[0094] in is the total number of primary prediction image sub-blocks, is the i-th predicted image sub-block, is the corresponding rendered image sub-block. Constraining the consistency between the rendered image sub-block and the predicted image sub-block prevents deviation during the optimization process.

[0095] In the embodiment of the present invention, a comprehensive loss function is constructed to balance pixel accuracy, structural consistency and rendering stability, so as to avoid a single loss dominating the optimization direction. The strategy of optimizing geometric parameters first and then optimizing appearance parameters is adopted to prevent parameter coupling.

[0096] S8 uses the geometric information of multiple high-level rendered image sub-blocks to reconstruct dense sub-scenes, and seamlessly merges the dense sub-scenes to obtain the reconstructed target large scene.

[0097] In an embodiment of the present invention, dense point clouds are extracted from the geometric information of each high-level rendered image sub-block in turn to construct a local dense sub-scene; each dense sub-scene corresponds to a triangular mesh; and the dense sub-scenes are aligned, smoothed and optimized in turn to generate a complete reconstructed target large scene.

[0098] See also Figure 3 , Figure 3 : is a structural schematic diagram of a large scene reconstruction system based on a 3D Gaussian algorithm in an embodiment of the present invention; an embodiment of the present invention also provides a large scene reconstruction system 100 based on a 3D Gaussian algorithm, the system comprising:

[0099] The image acquisition module 101 is configured to acquire a plurality of original images covering a target large scene area, and select original images satisfying a common viewing condition from the plurality of original images to form a first common viewing original image set;

[0100] The sparse point cloud generation module 102 is configured to generate a first sparse point cloud set based on the first common view original image set by using the SfM algorithm, and calculate the point cloud boundary of the first sparse point cloud set;

[0101] The large scene segmentation module 103 further segments the target large scene area into a plurality of first grid units based on the point cloud boundary and the original camera position projected on the plane;

[0102] The boundary extension module 104 is configured to extend the boundary of the initial first grid unit to obtain a plurality of second grid units; each of the second grid units contains a second common view original image subset and a second sparse point cloud subset;

[0103] The camera optimization module 105 is configured to add a new camera to the original camera set corresponding to each second grid unit based on the airspace perception visibility standard to obtain an optimized camera set corresponding to each second grid unit;

[0104] The sparse point cloud optimization module 106 is configured to optimize the second sparse point cloud subset in each second grid unit in turn according to the optimized camera set to obtain a corresponding third sparse point cloud subset;

[0105] The 3D Gaussian conversion module 107 is configured to use a 3D Gaussian sputtering model to sequentially perform 3D Gaussian point cloud distribution conversion, rasterization, and appearance modeling optimization processing on the third sparse point cloud subset corresponding to each second grid unit to obtain a plurality of primary rendering image sub-blocks;

[0106] The rendering image optimization module 108 constructs a comprehensive loss function, and iteratively updates each primary rendering image sub-block in turn based on the comprehensive loss function until the loss function tends to be stable, thereby obtaining multiple advanced rendering image sub-blocks;

[0107] The merging and reconstruction module 109 is configured to reconstruct a dense sub-scene using geometric information of multiple high-level rendered image sub-blocks, and seamlessly merge the dense sub-scenes to obtain a reconstructed target large scene.

[0108] An embodiment of the present invention also provides an electronic device, which includes: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the method described above.

[0109] An embodiment of the present invention further provides a computer-readable storage medium on which executable instructions are stored. When the instructions are executed by a processor, the processor executes the method described above.

[0110] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A large scene reconstruction method based on 3D Gaussian algorithm, characterized in that: The method comprises: S1 acquires a plurality of original images covering a large target scene area, and selects original images satisfying a common-view condition from the plurality of original images to form a first common-view original image set; the first common-view original image set includes a plurality of original images from different perspectives; S2 generates a first sparse point cloud set based on the first common view original image set using the SfM algorithm, and calculates the point cloud boundary of the first sparse point cloud set; then divides the target large scene area into a plurality of first grid units according to the point cloud boundary and the position of the original camera set projected on the plane by the first common view original image set; S3: expanding the boundary of the first grid unit to obtain a plurality of second grid units; each of the second grid units contains a second common view original image subset and a second sparse point cloud subset; S4 adds a new camera to the original camera set corresponding to each second grid unit to obtain an optimized camera set corresponding to each second grid unit; S5 optimizes each second sparse point cloud subset in turn according to the optimized camera set to obtain a corresponding third sparse point cloud subset; S6 adopts a 3D Gaussian sputtering model to perform 3D Gaussian point cloud distribution conversion, rasterization and appearance modeling optimization processing on each third sparse point cloud subset in turn to obtain multiple primary rendering image sub-blocks; S7 constructs a comprehensive loss function, and iteratively updates each primary rendered image sub-block in turn based on the comprehensive loss function until the loss function tends to be stable, thereby obtaining multiple advanced rendered image sub-blocks; S8 uses the geometric information of multiple high-level rendered image sub-blocks to reconstruct dense sub-scenes, and seamlessly merges the dense sub-scenes to obtain the reconstructed target large scene.

2. The method according to claim 1, characterized in that The 3D Gaussian sputtering model in step S6 includes a Gaussian point cloud distribution conversion unit, a differentiable Gaussian rasterization unit and a decoupled appearance modeling unit.

3. The method according to claim 2, characterized in that Step S6 further comprises: S61: using a Gaussian point cloud distribution conversion unit to sequentially convert the third sparse point cloud subset corresponding to each second grid unit into a group of primary 3D Gaussian point cloud distributions; and then convert each group of primary 3D Gaussian point cloud distributions into each primary 3D Gaussian image sub-block; S62 projects and renders each primary 3D Gaussian image sub-block onto a 2D image plane corresponding to each camera pose using a differentiable Gaussian rasterization unit to obtain a plurality of primary predicted image sub-blocks; S63 generates multiple primary transformation maps based on the multiple primary prediction image sub-blocks using a decoupled appearance modeling unit, and optimizes the appearance of each corresponding primary prediction image sub-block in turn according to each primary transformation map to obtain multiple primary rendering image sub-blocks.

4. The method according to claim 3, characterized in that Step S7 further comprises: S71 constructs a comprehensive loss function, wherein the comprehensive loss function includes a first loss function and a second loss function; wherein the first loss function includes a first difference loss function and a D-SSIM loss function, and the second loss function includes a second difference loss function; S72, based on the multiple primary rendered image sub-blocks, iteratively updating the parameters of each primary 3D Gaussian image sub-block by using the first loss function until the first loss function converges, to obtain multiple advanced 3D Gaussian image sub-blocks; S73 uses a differentiable Gaussian rasterization unit to render each high-level 3D Gaussian image sub-block onto a 2D image plane in sequence to obtain a plurality of high-level prediction image sub-blocks; S74 generates multiple corresponding high-level transformation maps based on multiple high-level prediction image sub-blocks using a decoupled appearance modeling unit, and performs iterative appearance optimization on each high-level prediction image sub-block in turn according to each high-level transformation map and the second loss function until the second loss function converges to obtain multiple high-level rendering image sub-blocks.

5. The method according to claim 4, characterized in that The first loss function in step S7 is: in, is a first difference loss function, used to calculate the absolute difference in pixel values ​​between the predicted image sub-block and the original image; is the D-SSIM loss function, which is used to calculate the structural similarity difference between the predicted image sub-block and the original image; is the weight of the D-SSIM loss function.

6. The method according to claim 4, characterized in that The parameters of the 3D Gaussian point cloud distribution include a center point, a covariance matrix, color information, and opacity.

7. The method according to claim 1, characterized in that The common viewing conditions in step S1 include: the satellite zenith angle of each original image is less than 30°, the angle between every two original images is less than 5°, and the shadow area of ​​each original image accounts for less than 5%.

8. A large scene reconstruction system based on 3D Gaussian algorithm, characterized in that: include: An image acquisition module is configured to acquire a plurality of original images covering a target large scene area, and select original images satisfying a common viewing condition from the plurality of original images to form a first common viewing original image set; A sparse point cloud generation module is configured to generate a first sparse point cloud set based on the first common view original image set by using the SfM algorithm, and calculate a point cloud boundary of the first sparse point cloud set; The large scene segmentation module further segments the target large scene area into a plurality of first grid units based on the point cloud boundary and the original camera position projected on the plane; A boundary expansion module is configured to expand the boundary of the initial first grid unit to obtain a plurality of second grid units; each of the second grid units contains a second common view original image subset and a second sparse point cloud subset; A camera optimization module is configured to add a new camera to an original camera set corresponding to each second grid unit based on an airspace perception visibility standard to obtain an optimized camera set corresponding to each second grid unit; a sparse point cloud optimization module, configured to optimize the second sparse point cloud subset in each second grid unit in turn according to the optimized camera set to obtain a corresponding third sparse point cloud subset; A 3D Gaussian transformation module is configured to use a 3D Gaussian sputtering model to sequentially perform 3D Gaussian point cloud distribution transformation, rasterization, and appearance modeling optimization processing on the third sparse point cloud subset corresponding to each second grid unit to obtain a plurality of primary rendering image sub-blocks; The rendering image optimization module constructs a comprehensive loss function and iteratively updates each primary rendering image sub-block in turn based on the comprehensive loss function until the loss function tends to be stable, thereby obtaining multiple advanced rendering image sub-blocks; The merging and reconstruction module is configured to reconstruct a dense sub-scene using geometric information of multiple high-level rendered image sub-blocks, and seamlessly merge the dense sub-scenes to obtain a reconstructed target large scene.

9. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors are caused to perform the method according to claims 1-7.

10. A computer-readable storage medium, characterized in that: Executable instructions are stored thereon, and when the instructions are executed by a processor, the processor is caused to perform the method according to claims 1-7.

Images