Three-dimensional scene reconstruction method and device based on point cloud augmentation, equipment and medium

Abstract

The application relates to the technical field of computers, in particular to a three-dimensional scene reconstruction method and device based on point cloud augmentation, equipment and medium, wherein the method comprises the following steps: acquiring a preliminary point cloud of a target scene and a scene complexity; determining the required preliminary point cloud density of each region in the target scene according to the scene complexity, and extracting an attention region in each perspective image in combination with the preliminary point cloud density; generating an augmented point cloud according to the attention region in each perspective image, and obtaining a target point cloud by weighting the preliminary point cloud and the augmented point cloud; and reconstructing a three-dimensional representation of the target scene by using the target point cloud. Thus, the problems of limited point cloud representation capability and insufficient sparse perspective coverage in the related art, which lead to poor three-dimensional reconstruction quality, are solved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, equipment and medium for reconstructing a three-dimensional scene based on point cloud augmentation. Background Art

[0002] With the development of computer vision, the demand for accurate and fast 3D reconstruction continues to increase. The goal of 3D scene reconstruction is to convert the captured scene view into a form that can be processed by computers. This is the basis for computers to understand the complex environment of the real world.

[0003] 3DGS (3D Gaussian Splatting) is a major breakthrough in 3D reconstruction and 3D scene representation. By representing the scene as a set of learnable 3D Gaussian kernels, it offers intuitive and highly editable representations. Unlike traditional explicit representation methods, the position, size, shape, and number of Gaussian kernels are unrestricted, offering flexible representation and high resolution. For scene reconstruction, the Gaussian kernel parameters are iteratively optimized under the supervision of multi-view images, combined with adaptive density control to enhance representation capabilities and accurately represent the scene. Leveraging a CUDA-based differentiable rasterization rendering pipeline, it achieves fast reconstruction and near-real-time rendering while maintaining effective reconstruction results.

[0004] However, in practical applications, it is often difficult to obtain a sufficient number of perspective images due to various limitations such as shooting position and shooting angle. Scene reconstruction based on sparse angles can significantly improve the optimization rate and reduce the difficulty of the data acquisition stage. However, in the case of sparse angles, the scene cannot be fully covered by the perspective, so the reconstruction problem is a serious pathological problem. The incompleteness of scene information will cause the reconstruction process to converge to a local optimum, generating a reasonable interpretation of the scene under limited training perspectives, but a variety of artifacts and distorted reconstructed areas under the test perspective, which reduces the performance of 3DGS scene representation. In particular, when the SfM (Structure from Motion) method is used to generate the initial point cloud, since only well-matched feature points are selected, a large amount of potentially useful information is ignored, which seriously affects the reconstruction effect, and there are also problems such as accuracy dependence and scale matching error. Summary of the Invention

[0005] The present application provides a three-dimensional scene reconstruction method, apparatus, device and medium based on point cloud augmentation to solve the problems in related technologies such as limited point cloud representation capability and insufficient sparse perspective coverage, resulting in poor quality of three-dimensional reconstruction.

[0006] The first aspect of the present application provides a three-dimensional scene reconstruction method based on point cloud augmentation, comprising the following steps: obtaining a preliminary point cloud and a scene complexity level of a target scene; determining the preliminary point cloud density required for each area in the target scene according to the scene complexity, and extracting the attention area in each perspective image in combination with the preliminary point cloud density; generating an augmented point cloud based on the attention area in each perspective image, and obtaining a target point cloud by weighting the preliminary point cloud and the augmented point cloud; and reconstructing a three-dimensional representation of the target scene using the target point cloud.

[0007] Optionally, extracting the attention area in each perspective image includes: dividing each perspective image into dense areas and sparse areas according to the preliminary point cloud density; performing a two-dimensional Fourier transform on the sparse area, and defining the area with a high-frequency component higher than a preset threshold as a complex structure but sparse point cloud area as the attention area.

[0008] Optionally, an augmented point cloud is generated based on the attention area in each perspective image, including: in the attention area in multiple multi-perspective images, two-dimensional feature point pairs corresponding to the multi-perspective views are obtained by first extracting and then matching, or directly using a feature matcher without extraction; the two-dimensional feature points corresponding to the two view trajectories are used as optional options for point cloud generation and mapped to the three-dimensional space together with the multi-view trajectory to generate an augmented point cloud.

[0009] Optionally, the two-view trajectory is a two-dimensional feature point appearing in two different viewing angle images, and the multi-view trajectory is a two-dimensional feature point appearing in three or more viewing angle images.

[0010] Optionally, the two-dimensional feature points corresponding to the two view trajectories are used as options for point cloud generation and mapped to the three-dimensional space together with the multi-view trajectory to generate an augmented point cloud, including: obtaining the two-dimensional coordinates of the two-dimensional feature points in images with different viewing angles; calculating the actual position and position error and other parameters of the two-dimensional feature points in the three-dimensional space based on the two-dimensional coordinates; and generating an augmented point cloud based on the actual position and position error.

[0011] Optionally, reconstructing a three-dimensional scene of a target scene using a target point cloud includes: reconstructing a three-dimensional representation of the target scene using an efficient training mechanism and the target point cloud, wherein the efficient training mechanism is obtained by adjusting the 3DGS training mechanism, and the adjustment parameters of the 3DGS training mechanism include: reducing the number of iterations, delaying the introduction of adaptive density control, reducing the frequency of use of adaptive density control, and removing the reset opacity operation.

[0012] Optionally, before obtaining the target point cloud based on the weighted preliminary point cloud and the augmented point cloud, the method further includes: obtaining three-dimensional feature points in the preliminary point cloud and the augmented point cloud; calculating the retention probability of each three-dimensional feature point based on the error level of the three-dimensional feature point, determining the target point cloud data based on the retention probability, and eliminating part of the target point cloud data, wherein the retention probability is inversely proportional to the error level.

[0013] The second aspect of the present application provides a three-dimensional scene reconstruction device based on point cloud augmentation, including: an acquisition module for acquiring a preliminary point cloud and a scene complexity level of a target scene; a determination module for determining the preliminary point cloud density required for each area in the target scene according to the scene complexity, and extracting the attention area in each perspective image in combination with the preliminary point cloud density; a processing module for generating an augmented point cloud based on the attention area in each perspective image, and obtaining a target point cloud by weighting the preliminary point cloud and the augmented point cloud; and a reconstruction module for reconstructing a three-dimensional representation of the target scene using the target point cloud.

[0014] Optionally, the processing module is further used to divide each perspective image into dense areas and sparse areas according to the point cloud density; perform a two-dimensional Fourier transform on the sparse area, and define the area with a high-frequency component higher than a preset threshold as a complex structure but sparse point cloud area as the attention area.

[0015] Optionally, the processing module is further used to obtain two-dimensional feature point pairs corresponding to multi-view views in the attention area of ​​multiple multi-view images by first extracting and then matching, or directly using a feature matcher without extraction; the two-dimensional feature points corresponding to the two view trajectories are used as options for point cloud generation and mapped to the three-dimensional space together with the multi-view trajectory to generate an augmented point cloud.

[0016] Optionally, the two-view trajectory is a two-dimensional feature point appearing in two different viewing angle images, and the multi-view trajectory is a two-dimensional feature point appearing in three or more viewing angle images.

[0017] Optionally, the processing module is further used to obtain the two-dimensional coordinates of the two-dimensional feature points in images of different perspectives; calculate the actual position and error and other parameters of the two-dimensional feature points in three-dimensional space based on the two-dimensional coordinates; and generate an augmented point cloud based on the actual position and error and other parameters.

[0018] Optionally, the reconstruction module is further used to reconstruct a three-dimensional representation of the target scene using an efficient training mechanism and a target point cloud, wherein the efficient training mechanism is obtained by adjusting the 3DGS training mechanism, and the adjustment parameters of the 3DGS training mechanism include: reducing the number of iterations, introducing adaptive density control, reducing the frequency of use of adaptive density control, and removing the reset opacity operation.

[0019] Optionally, the three-dimensional scene reconstruction device based on point cloud augmentation also includes: a filtering module, which is used to obtain three-dimensional feature points in the preliminary point cloud and the augmented point cloud before obtaining the target point cloud by weighting the preliminary point cloud and the augmented point cloud; calculate the retention probability of each three-dimensional feature point according to the error level of the three-dimensional feature point, determine the target point cloud data based on the retention probability, and eliminate part of the target point cloud data, wherein the retention probability is inversely proportional to the error level.

[0020] The third aspect of the present application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and runnable on the processor, wherein the processor executes the program to implement a three-dimensional scene reconstruction method based on point cloud augmentation as described in the above embodiment.

[0021] The fourth aspect of the present application provides a computer-readable storage medium having a computer program or instruction stored thereon. When the computer program or instruction is executed, it is used to implement the three-dimensional scene reconstruction method based on point cloud augmentation as in the above embodiment.

[0022] Therefore, this application has at least the following beneficial effects:

[0023] The embodiments of the present application can obtain a preliminary point cloud of the target scene and the scene complexity level, and use the scene complexity to determine the point cloud density required for each area. This can focus on improving the reconstruction effect of areas with complex structures but sparse point clouds, effectively balancing the overall density distribution of the point cloud, and by generating an augmented point cloud in the attention area, it can fully exploit the perspective information within the scene, especially in areas with sparse point cloud distribution, significantly increasing the number of initial points, making the reconstructed 3D scene more accurate and detailed. This solves the problems of limited point cloud representation capabilities and insufficient sparse perspective coverage in related technologies, resulting in poor 3D reconstruction quality.

[0024] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0026] Figure 1 This is a flowchart of a three-dimensional scene reconstruction method based on point cloud augmentation provided in an embodiment of the present application;

[0027] Figure 2 This is an example diagram of 3D scene reconstruction based on point cloud augmentation according to one embodiment of the present application;

[0028] Figure 3 A block diagram of a three-dimensional scene reconstruction device based on point cloud augmentation according to an embodiment of the present application;

[0029] Figure 4 Schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0030] The following describes in detail embodiments of the present application. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0031] The following describes the three-dimensional scene reconstruction method, device, electronic device and storage medium based on point cloud augmentation according to the embodiment of the present application with reference to the accompanying drawings. In response to the problems mentioned in the above background technology, the present application provides a three-dimensional scene reconstruction method based on point cloud augmentation. In this method, by obtaining the preliminary point cloud and scene complexity level of the target scene, the complexity of the scene is used to determine the point cloud density required for each area, focusing on improving the reconstruction effect of areas with complex structures but sparse point clouds, effectively balancing the overall density distribution of the point cloud, and by generating augmented point clouds in the attention area, it is possible to fully exploit the perspective information in the scene, especially in areas with sparse point cloud distribution, greatly increasing the number of initial points, and making the reconstructed three-dimensional scene more accurate and detailed. As a result, the problems of limited point cloud representation capabilities and insufficient sparse perspective coverage in related technologies, which lead to poor quality of three-dimensional reconstruction, are solved.

[0032] Specifically, Figure 1 A flowchart of a three-dimensional scene reconstruction method based on point cloud augmentation provided in an embodiment of the present application.

[0033] like Figure 1 As shown, the 3D scene reconstruction method based on point cloud augmentation includes the following steps:

[0034] In step S101 , a preliminary point cloud and scene complexity level of a target scene are obtained.

[0035] It is understandable that a point cloud represents a scene as a collection of points, each of which contains position, color, and error information. In the embodiment of this application, the preliminary point cloud is sparsely distributed and cannot meet the requirements of 3D reconstruction of complex scenes. As input, through feature extraction and matching, the point-to-point correspondence of multiple views is obtained. Then, through triangulation, bundle adjustment and other steps, the two-dimensional feature points corresponding to the multi-view trajectory are mapped back to a three-dimensional point in space to obtain a preliminary point cloud. And get the estimated camera pose for each view

[0036] Secondly, the scene complexity of the target scene in the embodiments of this application directly affects which areas require more attention and higher point cloud density, which is crucial for the selection of subsequent point cloud augmentation strategies. The scene complexity level can be determined by analyzing the structural characteristics within the scene, such as texture richness, geometric complexity, and other factors. It can also be determined through frequency domain feature analysis, for example, areas with higher high-frequency components correspond to higher complexity.

[0037] In step S102, the preliminary point cloud density required for each area in the target scene is determined according to the scene complexity, and the attention area in each perspective image is extracted in combination with the preliminary point cloud density.

[0038] Among them, there is a certain connection between the scene complexity and the density distribution required by the area. Therefore, the embodiment of the present application can determine the density distribution of the point cloud required for each area according to the scene complexity level, which provides a good guide for the subsequent division of the attention area and the construction of the augmented point cloud.

[0039] In one embodiment of the present application, the attention area in each perspective image is extracted, including: dividing each perspective image into dense areas and sparse areas according to the preliminary point cloud density; performing a two-dimensional Fourier transform on the sparse area, and defining the area with a high-frequency component higher than a preset threshold as a complex structure but sparse point cloud area as the attention area.

[0040] The embodiment of the present application can be guided by the complexity of the reconstruction scene, and the initial point cloud The scene area is divided based on the point cloud density distribution characteristics. Specifically, it is divided into dense areas and sparse areas according to the complexity of the structure and the sparsity of the point cloud. The reconstruction effect of the block with complex structure but sparse point cloud is the worst, and it is set as the attention area M i , which is selected as the implementation area for subsequent point cloud augmentation. It can be solved by the following steps:

[0041] Step 1: Based on the point cloud density required for each area in the target scene, various clustering algorithms are used Get the dense blocks in the view image, and then take the difference to get the sparse blocks:

[0042] Step 2: Perform two-dimensional Fourier transform on the sparse blocks one by one Using the threshold function The blocks with high-frequency components in the frequency domain higher than the set threshold t are retained, and the final attention area is obtained through inverse Fourier transform:

[0043] For different views, perform steps 1 and 2 in sequence to obtain a set of per-view attention regions.

[0044] In step S103, an augmented point cloud is generated according to the attention area in each perspective image, and a target point cloud is obtained by weighting the preliminary point cloud and the augmented point cloud.

[0045] The embodiment of the present application can determine the initial point cloud density distribution required for the scene area by scene complexity, divide the scene area based on the density distribution characteristics of the preliminary point cloud, construct the scene-adaptive augmented point cloud, and finally weightedly combine them into a high-quality target point cloud. Therefore, based on the corresponding relationship between scene complexity and the required point cloud distribution density, the embodiment of the present application focuses on the tolerance and quantity requirements of 3DGS initialization for point cloud accuracy. Based on this, the algorithm generates augmented point cloud This improves scene coverage and the effect of sparse-angle three-dimensional scene reconstruction, thereby achieving the purpose of using sparse-angle shooting to increase the reconstruction rate and reduce the difficulty of the data acquisition stage.

[0046] In one embodiment of the present application, an augmented point cloud is generated based on the attention area in each perspective image, including: in the attention area in multiple multi-perspective images, two-dimensional feature point pairs corresponding to the multi-perspective views are obtained by first extracting and then matching, or directly using a feature matcher without extraction; the two-dimensional feature points corresponding to the two view trajectories are used as optional options for point cloud generation and mapped to the three-dimensional space together with the multi-view trajectory to generate an augmented point cloud.

[0047] The multiple multi-view images can be in either order or disorder, without any specific limitation. A two-view trajectory is a two-dimensional feature point that appears in two different view images, and a multi-view trajectory is a two-dimensional feature point that appears in three or more view images.

[0048] It is understandable that within the attention region, based on the existing camera pose Feature extraction and matching are re-performed, and the point cloud is reconstructed. During this process, improved point cloud generation solutions with relaxed conditions are introduced, such as using a pre-trained feature-free matcher, relaxing the required number of viewpoints for local trajectories, and increasing the triangulation error threshold. These solutions reduce the accuracy requirements for the point cloud to a certain extent, allowing the full exploitation of viewpoint information within the attention region during the initialization phase.

[0049] During the actual execution process, the embodiment of the present application can obtain the two-dimensional coordinates of the two-dimensional feature points in images with different perspectives; calculate the actual position and error and other parameters of the two-dimensional feature points in the three-dimensional space based on the two-dimensional coordinates; and generate an augmented point cloud based on the actual position and error and other parameters.

[0050] Specifically, the embodiment of the present application can use n disordered multi-view two-dimensional images in the scene Using feature extraction and matching as input, we obtain point-to-point correspondences across multiple views. Subsequently, through triangulation and bundle adjustment, the 2D feature points corresponding to the two view trajectories are used as options for point cloud generation. These points are then mapped back to 3D points in 3D space, fully exploiting underutilized viewpoint information in these areas. Together with the newly generated 3D points from all multi-view trajectories, they form an augmented point cloud.

[0051] In one embodiment of the present application, the target point cloud is obtained by weighting the preliminary point cloud and the augmented point cloud.

[0052] Previously, it also included: obtaining three-dimensional feature points in the preliminary point cloud and the augmented point cloud; calculating the retention probability of each three-dimensional feature point according to the error level of the three-dimensional feature point; determining the point cloud data that needs to be eliminated based on the retention probability, wherein the retention probability is inversely proportional to the error level, that is, the higher the error level, the lower the retention probability, the fewer points left in the point cloud, which means that the weight contribution of this point cloud to the final point cloud is lower. The embodiment of the present application can filter part of the target point cloud data to better support the 3DGS reconstruction task.

[0053] In the actual implementation process, the embodiment of the present application can reproject each point in the point cloud back to the image domain of each view. If the projection point x i Located in image I i Within the boundary of , calculate the projection point x i And the corresponding two-dimensional feature truth point The error distance can be Euclidean distance, Manhattan distance, Chebyshev distance, etc. The error level ε is defined as the sum of the error distances under different views:

[0054]

[0055] Where π() is the projection function that maps the scene's 3D points to the image domain, and ρ() is the threshold function used to filter out unreasonable observations and eliminate the influence of outliers. The error level ε is the key threshold used by the algorithm to screen and determine the weight of the point cloud.

[0056] Furthermore, according to formula (1), the augmented point clouds are calculated The error level ε a and the initial point cloud The error level ε c Obviously, The probability of retention will be higher than Low, has a lower weight coefficient when combined.

[0057] Specifically, the embodiment of the present application can compare the error levels ε of the three-dimensional feature points in the two groups of point clouds. c and ε aThe point cloud weight function used to model the initial point filtering is:

[0058] p c (ε c )=θ c C(ε c ) (2)

[0059] p a (ε a )=θ a C(ε a ) (3)

[0060] Among them, pc and pa represent point clouds The probability of an inlier being retained, the lower the point cloud weight, the higher the probability of a point being removed. θ c ,θ a is the corresponding normalization factor, normalizing the value to [0, 1]. C is the probability function of retaining 3D feature points, which is negatively correlated with the error level and can be expressed in various forms, including but not limited to inverse proportional relationships, exponential relationships, Gaussian function relationships, and so on.

[0061] Finally, high-quality target point cloud It is composed of a weighted combination of the preliminary point cloud and the augmented point cloud:

[0062]

[0063] for c represents a probability, Represents a collection, Indicates that Each element in is extracted according to probability c to obtain a new subset.

[0064] In step S104 , a three-dimensional representation of the target scene is reconstructed using the target point cloud.

[0065] In one embodiment of the present application, a three-dimensional representation of a target scene is reconstructed using a target point cloud, including: reconstructing a three-dimensional representation of the target scene using an efficient training mechanism and a target point cloud, wherein the efficient training mechanism is obtained by adjusting the 3DGS training mechanism, and the adjustment parameters of the 3DGS training mechanism include: reducing the number of iterations, introducing adaptive density control, reducing the frequency of use of adaptive density control, and removing the reset opacity operation.

[0066] It is understandable that the embodiment of the present application designs a 3DGS parameter training mechanism in the scene reconstruction stage, which can better adapt to the reconstruction task of high-quality point cloud (target point cloud) in the case of sparse angles. The adjusted training mechanism includes:

[0067] First, the three-dimensional scene of the target scene is reconstructed using a high-quality point cloud. Since the high-quality point cloud provides more accurate initial conditions, the overall number of iterations can be reduced during the actual reconstruction process. Second, considering that high-quality point clouds require more iterations to converge, the embodiment of the present application delays the introduction of the ADC (Adaptive Density Control) mechanism. The ADC is responsible for periodically splitting, cloning or cropping the Gaussian kernel, so that the number of Gaussian kernels changes. Therefore, by delaying the application of the ADC, it is ensured that the existing Gaussian kernel representation has fully converged before proceeding to the next step, and then a new Gaussian kernel is added to improve the scene representation capability. Third, since the Gaussian kernel after initialization of the high-quality point cloud is relatively accurate, the embodiment of the present application can reduce the need for frequent use of the ADC. Fourth, during sparse angle reconstruction, the practice of periodically cropping the low-opacity Gaussian kernel may cause the number of Gaussians to be excessively reduced, especially after resetting the opacity, thereby affecting the success rate of reconstruction. Therefore, the embodiment of the present application can remove the opacity reset operation to ensure the integrity and accuracy of the reconstruction.

[0068] The following is a detailed introduction to the use of a target training mechanism and a target point cloud to reconstruct a three-dimensional scene of a target scene in an embodiment of the present application with reference to specific examples. The 3DGS of the embodiment of the present application can represent the scene as a three-dimensional Gaussian kernel with several learnable parameters, wherein the three-dimensional Gaussian kernel is characterized by mean, covariance, spherical harmonic function coefficients, and opacity.

[0069] The embodiment of the present application can be given a previously generated high-quality point cloud 3DGS models each point as a 3D Gaussian kernel, obtaining a preliminary representation of the scene:

[0070]

[0071] During the subsequent optimization training process, the embodiment of the present application can randomly extract one of the perspectives and splash the three-dimensional Gaussian within the image boundary into the image domain, reducing the dimension into a two-dimensional Gaussian with a clear center and blurred boundaries;

[0072] Query the color c of each 2D Gaussian at each pixel. Calculate the opacity α′ at each pixel:

[0073]

[0074] where ∑ 2D , μ ′ They are the three-dimensional covariance ∑ of the Gaussian kernel, the projection of the three-dimensional mean μ in the image domain, and x ′ are pixel coordinates.

[0075] According to the depth value at this perspective, Two-dimensional Gaussians are sorted in sequence;

[0076] Calculate the color value at the pixel coordinate through α-rendering, which is the predicted value of 3DGS:

[0077]

[0078] Substitute the predicted value and the true value into the loss function By updating the trainable parameters through error backpropagation and calling ADC according to the training strategy, 3DGS can train a nearly converged scene representation based on the initial point cloud, and ultimately obtain a realistic three-dimensional scene.

[0079] In actual implementation, the embodiments of the present application can reduce the overall number of iterations for sparse angle reconstruction, thereby significantly shortening training time while maintaining similar reconstruction results. By increasing the periodic interval for implementing ADC, 3DGS can reasonably integrate the newly generated Gaussian kernels into the scene during the interval, reducing the number of Gaussians in the final scene representation and effectively preventing overfitting of complex scene regions. Removing the opacity reset operation avoids an excessive reduction in the number of Gaussians during the sparse angle reconstruction process.

[0080] Furthermore, the trained and reconstructed 3D scene can be visualized using professional software. By inputting the camera pose of the perspective of interest, 3DGS can present the corresponding rendering results in real time.

[0081] The following combination Figure 2 The method for 3D scene reconstruction based on point cloud augmentation in the embodiment of the present application is described in detail, including an initialization phase and a reconstruction phase. The initialization phase determines the density distribution of the initial point cloud required for the scene area based on the scene complexity, divides the scene area based on the density distribution characteristics of the preliminary point cloud, constructs a scene-adapted augmented point cloud, and finally weights and combines them into a high-quality target point cloud, significantly increasing the number of initial points and balancing their density distribution. The reconstruction phase performs a 3D reconstruction of the target scene based on the weighted target point cloud, better adapting to the task of reconstructing high-quality point clouds in sparse angles. While maintaining sufficient resolution and accuracy, it greatly improves computational efficiency and reduces memory overhead.

[0082] During the actual execution process, the 3DGS algorithm for scene reconstruction in the embodiment of the present application requires scene camera pose parameters and camera internal participating point clouds as initialization inputs. Among them, the camera internal parameters (focal length, principal point, pixel size, etc.) and camera pose (perspective transformation matrix) are used as the main basis for rendering pipeline direction selection and transformation. Point cloud represents the scene as a collection of a series of points, each point containing position, color and error information. After the 3D Gaussian reconstruction phase begins, these points are modeled into a series of three-dimensional Gaussian ellipsoids and characterized by several properties such as mean, covariance, spherical harmonic function coefficients, and opacity to form a preliminary interpretation of the scene, and are subsequently optimized through gradient descent and density control, including the following aspects:

[0083] 1) Preliminary point cloud and camera pose estimation

[0084] For n views, after feature extraction and matching, a set of estimated initial camera poses can be obtained The input of the point cloud reconstruction stage includes n scene views and its associated camera pose estimate COLMAP uses a triangulation method with two different views. Generate preliminary point cloud

[0085]

[0086] After completing the new image registration, perform bundle adjustment to improve the camera pose P i and the coordinate parameters of the three-dimensional point X, thereby minimizing the reprojection error and filtering out observation data with large errors:

[0087]

[0088] Among them, the 3D point must be visible in at least 3 views to be considered as a valid point. Updated to And the new three-dimensional point X * Add to Point Cloud Re-triangulation is performed to track the 2D feature points that were not successfully triangulated before the new image registration to prevent missing valid information.

[0089] Then the next image registration is started, and the bundle adjustment and triangulation measurement are performed alternately until all images are registered. After the global bundle adjustment, the optimal camera pose parameter set is finally obtained. and preliminary point cloud Calculate the average error level of each point in the point cloud as the error level of the entire preliminary point cloud

[0090]

[0091] Among them, the error distance uses the square of the Euclidean distance.

[0092] 2) Delineation of attention areas

[0093] Considering that DetectorfreeSfM utilizes an extraction-free feature matcher, it can significantly improve the matching of feature points in weakly textured areas. Therefore, when dividing the attention region, we retain regions with simple structures and sparse point clouds, and use the feature matcher to extract more structural and geometric information within these regions, thereby achieving better reconstruction results.

[0094] Based on preliminary point cloud Projection to View I i In the density distribution of the point cloud, the embodiment of the present application can use DBSCAN (Density-Based Spatial Clustering of Applications with Noise, density-based spatial clustering algorithm) to divide the point cloud dense area, and then use the difference set to obtain the attention area M under this perspective. i :

[0095]

[0096] Among them, ∈ and μ are two parameters of the DBSCAN method. ∈ is set to 1% of the image resolution, representing the radius of the search area around the starting point; μ is set to 15, representing the minimum number of samples for considering the starting point and surrounding points as a cluster. For each view, use formula (11) to specify the set of attention areas

[0097] 3) Define augmented point cloud

[0098] In the attention area, DetectorfreeSfM’s extraction-free feature matcher is used to directly obtain the multi-view point-to-point correspondence, especially in the weak texture area, which can obtain two-dimensional feature points several times that of COLMAP. Furthermore, in the point cloud reconstruction stage, the camera pose estimated by COLMAP can be used to obtain the corresponding point-to-point relationship between the two views. Relax the number of views for the local track, keeping the two-view track as an option for point cloud generation:

[0099]

[0100]

[0101] The three-dimensional point X ′* Add to Point Cloud Similar to the previous steps of initial point cloud reconstruction, the augmented point cloud is constructed Calculate the error level of the entire augmented point cloud

[0102]

[0103] 4) Define high-quality target point cloud

[0104] High-quality point cloud for 3DGS initialization in the embodiment of this application The above preliminary point cloud With augmented point cloud The point cloud weight function is related to the overall error level of the point cloud, as shown in Equations (2) and (3). The retention probability function is defined as being inversely proportional to the error level, that is:

[0105]

[0106] Among them, β is a factor that controls the influence of the error level on the retention probability and is generally set to 1. Because the range of C(ε) is (0, 1), the normalization factor α in equations (2) and (3) can be set to 1.

[0107] High-quality target point cloud Calculated by the following formula:

[0108]

[0109] 5) 3DGS reconstruction parameter optimization

[0110] As an example of a possible implementation, the optimization training process described in the embodiment of the present application specifically adopts the following implementation method:

[0111] The iteration cycle is set to 5000 times. The training method in each iteration cycle is consistent with that described in the above embodiment. As an example, the loss function is expressed as a weighted combination of L1 loss and SSIM loss:

[0112]

[0113] In the experiment, RGB Set to 0.2. Set the rendering value obtained by 3DGS training at a certain viewing angle to be I render , the corresponding truth value is I gt , then the L1 loss is defined as:

[0114]

[0115] The SSIM loss is defined as:

[0116]

[0117] The Adam optimizer was used and the opacity reset operation was removed.

[0118] High-quality point clouds contain a wealth of scene structure and detail information. Therefore, ADC is not introduced during the first 2000 iterations. 3DGS optimizes based on the initial point cloud to obtain a fully converged preliminary representation of the scene.

[0119] In the subsequent 2500 iterations, ADC is introduced to enhance the representation capability of 3DGS. The period interval of implementing ADC is set to 200 to ensure that the algorithm can effectively integrate the newly generated Gaussian kernel into the scene representation within the interval.

[0120] In the last 500 iterations, the reconstruction results are fine-tuned without increasing the number of Gaussian kernels.

[0121] The 3D scene reconstruction method based on point cloud augmentation proposed in the embodiment of this application obtains the preliminary point cloud and scene complexity level of the target scene, and uses the scene complexity to determine the point cloud density required for each area. This method focuses on improving the reconstruction effect of areas with complex structures but sparse point clouds, effectively balancing the overall density distribution of the point cloud. Moreover, by generating augmented point clouds in the attention area, it can fully exploit the perspective information within the scene, especially in areas with sparse point cloud distribution, significantly increasing the number of initial points, making the reconstructed 3D scene more accurate and detailed. This solves the problems of limited point cloud representation capabilities and insufficient sparse perspective coverage in related technologies, resulting in poor quality of 3D reconstruction.

[0122] Next, a three-dimensional scene reconstruction device based on point cloud augmentation proposed in an embodiment of the present application will be described with reference to the accompanying drawings.

[0123] Figure 3 3D scene reconstruction device based on point cloud augmentation according to an embodiment of the present application is shown in FIG.

[0124] like Figure 3 As shown, the three-dimensional scene reconstruction device 10 based on point cloud augmentation includes: an acquisition module 100, a determination module 200, a processing module 300 and a reconstruction module 400.

[0125] Among them, the acquisition module 100 is used to obtain the preliminary point cloud and scene complexity level of the target scene; the determination module 200 is used to determine the preliminary point cloud density required for each area in the target scene according to the scene complexity, and extract the attention area in each perspective image in combination with the preliminary point cloud density; the processing module 300 is used to generate an augmented point cloud based on the attention area in each perspective image, and obtain the target point cloud according to the weighted preliminary point cloud and the augmented point cloud; the reconstruction module 400 is used to use the target point cloud to reconstruct the three-dimensional scene of the target scene.

[0126] In one embodiment of the present application, the processing module 300 is further used to divide each perspective image into dense areas and sparse areas based on the preliminary point cloud density distribution; perform a two-dimensional Fourier transform on the sparse areas, and define the areas with high-frequency components higher than a preset threshold as areas with complex structures but sparse point clouds as attention areas.

[0127] In one embodiment of the present application, the processing module 300 is further used to obtain two-dimensional feature point pairs corresponding to multi-view views in the attention area of ​​multiple multi-view images by first extracting and then matching, or directly using a feature matcher without extraction, to extract the two-dimensional feature points of the preliminary point cloud in the multi-view images; the two-dimensional feature points corresponding to the two view trajectories are used as optional options for point cloud generation and mapped to the three-dimensional space together with the multi-view trajectory to generate an augmented point cloud.

[0128] In one embodiment of the present application, the two-view trajectory is a two-dimensional feature point appearing in two different perspective images, and the multi-view trajectory is a two-dimensional feature point appearing in three or more perspective images.

[0129] In one embodiment of the present application, the processing module 300 is further used to obtain the two-dimensional coordinates of the two-dimensional feature points in images of different perspectives; calculate the actual position and error and other parameters of the two-dimensional feature points in the three-dimensional space based on the two-dimensional coordinates; and generate an augmented point cloud based on the actual position and error and other parameters.

[0130] In one embodiment of the present application, the reconstruction module 400 is further used to reconstruct a three-dimensional scene of a target scene using an efficient training mechanism and a target point cloud, wherein the efficient training mechanism is obtained by adjusting the 3DGS training mechanism, and the adjustment parameters of the 3DGS training mechanism include: reducing the number of iterations, introducing adaptive density control, reducing the frequency of use of adaptive density control, and removing the reset opacity operation.

[0131] In one embodiment of the present application, the three-dimensional scene reconstruction device 10 based on point cloud augmentation also includes: a filtering module, which is used to obtain three-dimensional feature points in the preliminary point cloud and the augmented point cloud before obtaining the target point cloud by weighting the preliminary point cloud and the augmented point cloud; calculate the retention probability of each three-dimensional feature point according to the error level of the three-dimensional feature point, determine the target point cloud data based on the retention probability, and eliminate part of the target point cloud data, wherein the retention probability is inversely proportional to the error level.

[0132] It should be noted that the aforementioned explanation of the embodiment of the three-dimensional scene reconstruction method based on point cloud augmentation is also applicable to the three-dimensional scene reconstruction device based on point cloud augmentation in this embodiment, and will not be repeated here.

[0133] The 3D scene reconstruction device based on point cloud augmentation proposed in the embodiment of the present application obtains the preliminary point cloud and scene complexity level of the target scene, and uses the scene complexity to determine the point cloud density required for each area. This improves the reconstruction effect of areas with complex structures but sparse point clouds, effectively balances the overall density distribution of the point cloud, and by generating augmented point clouds in the attention area, it can fully exploit the perspective information within the scene, especially in areas with sparse point cloud distribution, significantly increasing the number of initial points, making the reconstructed 3D scene more accurate and detailed. This solves the problems of limited point cloud representation capabilities and insufficient sparse perspective coverage in related technologies, resulting in poor quality 3D reconstruction.

[0134] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0135] Memory 401 , processor 402 , and computer programs stored in the memory 401 and executable on the processor 402 .

[0136] When the processor 402 executes the program, the three-dimensional scene reconstruction method based on point cloud augmentation provided in the above embodiment is implemented.

[0137] Furthermore, the electronic device further includes:

[0138] The communication interface 403 is used for communication between the memory 401 and the processor 402 .

[0139] The memory 401 is used to store computer programs that can be run on the processor 402 .

[0140] The memory 401 may include a high-speed RAM (Random Access Memory) memory, and may also include a non-volatile memory, such as at least one disk memory.

[0141] If the memory 401, the processor 402, and the communication interface 403 are implemented independently, the communication interface 403, the memory 401, and the processor 402 can be connected to each other via a bus and communicate with each other. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4Only one bus or only one type of bus can exist, however.

[0142] Optionally, in a specific implementation, if the memory 401, the processor 402 and the communication interface 403 are integrated on a chip, the memory 401, the processor 402 and the communication interface 403 can complete the communication among each other through an internal interface.

[0143] The processor 402 can be a CPU (Central Processing Unit, central processor) or an ASIC (Application Specific Integrated Circuit, specific integrated circuit) or one or more integrated circuits configured to implement one or more embodiments of the present application.

[0144] The embodiments of the present application also provide a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the point cloud augmentation based three-dimensional scene reconstruction method as above.

[0145] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example" or "some examples" means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms is not necessarily directed to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the present specification and the features of the different embodiments or examples without contradiction.

[0146] In addition, the terms "first", "second" are only used for description purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "N" is at least two, for example, two, three, etc., unless otherwise explicitly and specifically limited.

[0147] Any process or method described in a flowchart or otherwise described herein can be understood as representing code modules, segments, or portions of code that include one or more executable instructions for implementing the specified logical functions or steps, and the various embodiments of the application include alternative implementations of the described processes or methods, in which the order of steps can be changed, including the use of simultaneous steps or otherwise, and in which certain steps can be performed concurrently, or in reverse order, depending on the functions involved, which should be apparent to those skilled in the art.

[0148] It should be understood that portions of the present application can be implemented in hardware, software, firmware, or combinations thereof. In the above embodiments, the steps or methods can be implemented in software or firmware that is stored in memory and executed by a suitable instruction execution system. As with the hardware, the software or firmware can be implemented using any of a number of programming languages, such as, for example, Java, C, C++, C#, Visual Basic, or the like, and as with the hardware, any combination of the above can be used. In another embodiment, the steps or methods can be implemented using a combination of software and hardware.

[0149] Those of skill in the art will appreciate that the steps or methods carried out in the above-described embodiments can be carried out by program instructions stored in a computer readable storage medium and executed by a data processing apparatus, where the data processing apparatus can be a general purpose computer or a dedicated computer. The steps or methods carried out in the above-described embodiments can also be implemented using a combination of software and hardware.

[0150] While the above has shown and described in detail certain embodiments of the application, it is to be understood that the above-described embodiments are only examples of the application and that alterations, modifications, replacements, and variations of the described embodiments can become apparent to those of ordinary skill in the art without departing from the scope of the application.

Claims

1. A three-dimensional scene reconstruction method based on point cloud augmentation, characterized in that: The following steps are involved: Obtain preliminary point cloud and scene complexity level of the target scene; Determine the preliminary point cloud density required for each area in the target scene according to the scene complexity, and extract the attention area in each perspective image based on the preliminary point cloud density; generating an augmented point cloud according to the attention area in each perspective image, and obtaining a target point cloud by weighting the preliminary point cloud and the augmented point cloud; reconstructing a three-dimensional representation of the target scene using the target point cloud; Extracting the attention area from each perspective image includes: dividing each perspective image into a dense area and a sparse area according to the preliminary point cloud density; performing a two-dimensional Fourier transform on the sparse area, and defining an area with a high-frequency component higher than a preset threshold as a complex structure but sparse point cloud area as the attention area; The method of generating an augmented point cloud based on the attention area in each perspective image includes: obtaining two-dimensional feature point pairs corresponding to the multi-perspective views in the attention area in multiple multi-perspective images by first extracting and then matching, or directly using a feature matcher without extraction; mapping the two-dimensional feature points corresponding to the two-view trajectories as optional options for point cloud generation together with the multi-view trajectory to the three-dimensional space to generate the augmented point cloud, wherein the two-view trajectory is the two-dimensional feature points appearing in two different perspective images, and the multi-view trajectory is the two-dimensional feature points appearing in three or more perspective images.

2. The 3D scene reconstruction method based on point cloud augmentation according to claim 1, characterized in that: The method of using the two-dimensional feature points corresponding to the two view trajectories as an option for generating a point cloud and mapping them together with the multi-view trajectories to a three-dimensional space to generate the augmented point cloud includes: Obtaining the two-dimensional coordinates of the two-dimensional feature points in images of different viewing angles; Calculating the actual position and position error of the two-dimensional feature point in the three-dimensional space according to the two-dimensional coordinates; The augmented point cloud is generated based on the actual position and the position error.

3. The 3D scene reconstruction method based on point cloud augmentation according to claim 1, characterized in that: The reconstructing a three-dimensional representation of the target scene using the target point cloud includes: A three-dimensional representation of the target scene is reconstructed using an efficient training mechanism and the target point cloud, wherein the efficient training mechanism is obtained by adjusting the 3DGS training mechanism, and the adjustment parameters of the 3DGS training mechanism include: reducing the number of iterations, delaying the introduction of adaptive density control, reducing the frequency of use of adaptive density control, and removing the reset opacity operation.

4. The 3D scene reconstruction method based on point cloud augmentation according to claim 1, characterized in that: Before obtaining the target point cloud based on the weighted preliminary point cloud and the augmented point cloud, it also includes: Acquire three-dimensional feature points in the preliminary point cloud and the augmented point cloud; The retention probability of each three-dimensional feature point is calculated according to the error level of the three-dimensional feature point, the target point cloud data is determined based on the retention probability, and part of the target point cloud data is eliminated, wherein the retention probability is inversely proportional to the error level.

5. A 3D scene reconstruction device based on point cloud augmentation, characterized in that: include: The acquisition module is used to obtain the preliminary point cloud and scene complexity level of the target scene; A determination module determines the preliminary point cloud density required for each area in the target scene based on the scene complexity, and extracts the attention area in each perspective image based on the preliminary point cloud density; a processing module, configured to generate an augmented point cloud according to the attention area in each perspective image, and obtain a target point cloud by weighting the preliminary point cloud and the augmented point cloud; a reconstruction module, configured to reconstruct a three-dimensional representation of the target scene using the target point cloud; The determination module is further configured to: divide the images of each perspective into dense areas and sparse areas according to the preliminary point cloud density; perform a two-dimensional Fourier transform on the sparse areas, and define areas with high-frequency components above a preset threshold as areas with complex structures but sparse point clouds as the attention areas; The processing module is further used to: obtain two-dimensional feature point pairs corresponding to multi-view views in the attention area of ​​multiple multi-view images by first extracting and then matching, or directly using a feature matcher without extraction; map the two-dimensional feature points corresponding to the two-view trajectories as optional options for point cloud generation together with the multi-view trajectory to the three-dimensional space to generate the augmented point cloud, wherein the two-view trajectory is the two-dimensional feature points appearing in two different view images, and the multi-view trajectory is the two-dimensional feature points appearing in three or more view images.

6. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the three-dimensional scene reconstruction method based on point cloud augmentation according to any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed, the three-dimensional scene reconstruction method based on point cloud augmentation according to any one of claims 1 to 4 is implemented.

Images