Three-dimensional scene reconstruction method and device based on point cloud augmentation, equipment and medium

Abstract

The invention relates to the technical field of computers, in particular to a three-dimensional scene reconstruction method and device based on point cloud augmentation, equipment and a medium, and the method comprises the steps: obtaining a preliminary point cloud and scene complexity of a target scene; determining an initial point cloud density required by each region in the target scene according to the scene complexity, and extracting an attention region in each view image in combination with the initial point cloud density; generating an augmented point cloud according to the attention area in each view angle image, and weighting according to the initial point cloud and the augmented point cloud to obtain a target point cloud; a three-dimensional representation of the target scene is reconstructed using the target point cloud. Therefore, the problems of poor three-dimensional reconstruction quality and the like caused by limited point cloud characterization capability and insufficient sparse view angle coverage in related technologies are solved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, equipment and medium for reconstructing a three-dimensional scene based on point cloud augmentation. Background Art

[0002] With the development of computer vision, the demand for accurate and fast 3D reconstruction continues to increase. The goal of 3D scene reconstruction is to convert the captured scene views into a form that can be processed by computers. This is the basis for computers to understand the complex environment of the real world.

[0003] 3DGS (3D Gaussian Splatting) is a major breakthrough in 3D reconstruction and 3D scene representation. By representing the scene as a number of learnable 3D Gaussian kernels, it has the advantages of being intuitive and highly editable. Unlike traditional explicit representation methods, the position, size, shape, and number of Gaussian kernels are not restricted, and it has the advantages of flexible representation and high resolution. In scene reconstruction, the parameters of the Gaussian kernel are optimized and iterated under the supervision of multi-view images, combined with adaptive density control to enhance the representation ability, and then accurately represent the scene; using the CUDA-based differentiable rasterization rendering pipeline, fast reconstruction and near real-time rendering are achieved while ensuring the reconstruction effect.

[0004] However, in practical applications, it is often difficult to obtain a sufficient number of view images due to various limitations such as shooting position and shooting angle. Scene reconstruction based on sparse angles can significantly improve the optimization rate and reduce the difficulty of the data acquisition stage. However, in the case of sparse angles, the scene cannot be fully covered by the view, so the reconstruction problem is a serious pathological problem. The incompleteness of scene information will cause the reconstruction process to converge to the local optimum, generating a reasonable interpretation of the scene under limited training perspectives, but various artifacts and distorted reconstruction areas under the test perspective, reducing the performance of 3DGS scene representation. In particular, when the SfM (Structure from Motion) method is used to generate the initial point cloud, since only well-matched feature points are selected, a large amount of potentially useful information is ignored, which seriously affects the reconstruction effect, and there are also problems such as accuracy dependence and scale matching error. Summary of the invention

[0005] The present application provides a three-dimensional scene reconstruction method, device, equipment and medium based on point cloud augmentation to solve the problems in the related art such as limited point cloud representation capability and insufficient sparse perspective coverage, resulting in poor quality of three-dimensional reconstruction.

[0006] The first aspect of the present application provides a three-dimensional scene reconstruction method based on point cloud augmentation, comprising the following steps: obtaining a preliminary point cloud and a scene complexity level of a target scene; determining the preliminary point cloud density required for each area in the target scene according to the scene complexity, and extracting the attention area in each perspective image in combination with the preliminary point cloud density; generating an augmented point cloud based on the attention area in each perspective image, and obtaining a target point cloud by weighting the preliminary point cloud and the augmented point cloud; and reconstructing a three-dimensional representation of the target scene using the target point cloud.

[0007] Optionally, extracting the attention area in each perspective image includes: dividing each perspective image into dense areas and sparse areas according to the preliminary point cloud density; performing a two-dimensional Fourier transform on the sparse area, and defining the area with a high-frequency component higher than a preset threshold as a complex structure but sparse point cloud area as the attention area.

[0008] Optionally, an augmented point cloud is generated based on the attention area in each perspective image, including: in the attention area in multiple multi-perspective images, two-dimensional feature point pairs corresponding to the multi-perspective views are obtained by first extracting and then matching, or directly using a non-extraction feature matcher; the two-dimensional feature points corresponding to the two view trajectories are used as options for point cloud generation and mapped to the three-dimensional space together with the multi-view trajectory to generate an augmented point cloud.

[0009] Optionally, the two-view trajectory is a two-dimensional feature point appearing in two different viewing angle images, and the multi-view trajectory is a two-dimensional feature point appearing in three or more viewing angle images.

[0010] Optionally, the two-dimensional feature points corresponding to the two view trajectories are used as options for point cloud generation and mapped to the three-dimensional space together with the multi-view trajectory to generate an augmented point cloud, including: obtaining the two-dimensional coordinates of the two-dimensional feature points in different viewing angle images; calculating the actual position and position error and other parameters of the two-dimensional feature points in the three-dimensional space according to the two-dimensional coordinates; and generating an augmented point cloud based on the actual position and position error.

[0011] Optionally, a three-dimensional scene of a target scene is reconstructed using a target point cloud, including: using an efficient training mechanism and a target point cloud to reconstruct a three-dimensional representation of the target scene, wherein the efficient training mechanism is adjusted according to a 3DGS training mechanism, and adjustment parameters of the 3DGS training mechanism include: reducing the number of iterations, delaying the introduction of adaptive density control, reducing the frequency of use of adaptive density control, and removing the reset opacity operation.

[0012] Optionally, before obtaining the target point cloud based on the weighted preliminary point cloud and the augmented point cloud, the method further includes: obtaining three-dimensional feature points in the preliminary point cloud and the augmented point cloud; calculating the retention probability of each three-dimensional feature point based on the error level of the three-dimensional feature point, determining the target point cloud data based on the retention probability, and eliminating part of the target point cloud data, wherein the retention probability is inversely proportional to the error level.

[0013] The second aspect of the present application provides a three-dimensional scene reconstruction device based on point cloud augmentation, including: an acquisition module, used to acquire a preliminary point cloud and a scene complexity level of a target scene; a determination module, used to determine the preliminary point cloud density required for each area in the target scene according to the scene complexity, and extract the attention area in each perspective image in combination with the preliminary point cloud density; a processing module, used to generate an augmented point cloud based on the attention area in each perspective image, and obtain a target point cloud by weighting the preliminary point cloud and the augmented point cloud; a reconstruction module, used to reconstruct a three-dimensional representation of the target scene using the target point cloud.

[0014] Optionally, the processing module is further used to divide each perspective image into dense areas and sparse areas according to the point cloud density; perform a two-dimensional Fourier transform on the sparse area, and define the area with a high-frequency component higher than a preset threshold as a complex structure but sparse point cloud area as an attention area.

[0015] Optionally, the processing module is further used to obtain two-dimensional feature point pairs corresponding to multi-view views in the attention area of ​​multiple multi-view images by first extracting and then matching, or directly using a non-extraction feature matcher; the two-dimensional feature points corresponding to the two view trajectories are used as options for point cloud generation and mapped to the three-dimensional space together with the multi-view trajectories to generate an augmented point cloud.

[0016] Optionally, the two-view trajectory is a two-dimensional feature point appearing in two different viewing angle images, and the multi-view trajectory is a two-dimensional feature point appearing in three or more viewing angle images.

[0017] Optionally, the processing module is further used to obtain the two-dimensional coordinates of the two-dimensional feature points in images with different viewing angles; calculate the actual position and error and other parameters of the two-dimensional feature points in three-dimensional space based on the two-dimensional coordinates; and generate an augmented point cloud based on the actual position and error and other parameters.

[0018] Optionally, the reconstruction module is further used to reconstruct a three-dimensional representation of the target scene using an efficient training mechanism and a target point cloud, wherein the efficient training mechanism is adjusted according to the 3DGS training mechanism, and the adjustment parameters of the 3DGS training mechanism include: reducing the number of iterations, introducing adaptive density control, reducing the frequency of use of adaptive density control, and removing the reset opacity operation.

[0019] Optionally, the three-dimensional scene reconstruction device based on point cloud augmentation also includes: a filtering module, which is used to obtain three-dimensional feature points in the preliminary point cloud and the augmented point cloud before obtaining the target point cloud by weighting the preliminary point cloud and the augmented point cloud; calculate the retention probability of each three-dimensional feature point according to the error level of the three-dimensional feature point, determine the target point cloud data based on the retention probability, and eliminate part of the target point cloud data, wherein the retention probability is inversely proportional to the error level.

[0020] The third aspect of the present application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement a three-dimensional scene reconstruction method based on point cloud augmentation as in the above-mentioned embodiment.

[0021] The fourth aspect of the present application provides a computer-readable storage medium having a computer program or instruction stored thereon. When the computer program or instruction is executed, it is used to implement the three-dimensional scene reconstruction method based on point cloud augmentation as in the above-mentioned embodiment.

[0022] Therefore, this application has at least the following beneficial effects:

[0023] The embodiment of the present application can obtain the preliminary point cloud and scene complexity level of the target scene, and use the complexity of the scene to determine the point cloud density required for each area, focusing on improving the reconstruction effect of areas with complex structures but sparse point clouds, effectively balancing the overall density distribution of the point cloud, and by generating augmented point clouds in the attention area, it can fully mine the perspective information in the scene, especially in areas with sparse point cloud distribution, greatly increasing the number of initial points, making the reconstructed three-dimensional scene more accurate and detailed. This solves the problems of limited point cloud representation capabilities and insufficient sparse perspective coverage in related technologies, resulting in poor quality of three-dimensional reconstruction.

[0024] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0026] Figure 1 A flowchart of a three-dimensional scene reconstruction method based on point cloud augmentation provided according to an embodiment of the present application;

[0027] Figure 2 This is an example diagram of three-dimensional scene reconstruction based on point cloud augmentation according to an embodiment of the present application;

[0028] Figure 3 A block diagram of a three-dimensional scene reconstruction device based on point cloud augmentation provided according to an embodiment of the present application;

[0029] Figure 4 Schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0030] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0031] The following describes the three-dimensional scene reconstruction method, device, electronic device and storage medium based on point cloud augmentation according to the embodiments of the present application with reference to the accompanying drawings. In response to the problems mentioned in the above background technology, the present application provides a three-dimensional scene reconstruction method based on point cloud augmentation. In this method, by obtaining the preliminary point cloud and scene complexity level of the target scene, the complexity of the scene is used to determine the point cloud density required for each area, and the reconstruction effect of the area with complex structure but sparse point cloud is improved, and the overall density distribution of the point cloud is effectively balanced. By generating an augmented point cloud in the attention area, the perspective information in the scene can be fully mined, especially in the area with sparse point cloud distribution, which greatly increases the number of initial points and makes the reconstructed three-dimensional scene more accurate and detailed. As a result, the problems of limited point cloud representation capability and insufficient sparse perspective coverage in the related technology, resulting in poor quality of three-dimensional reconstruction, are solved.

[0032] Specifically, Figure 1 A schematic flowchart of a three-dimensional scene reconstruction method based on point cloud augmentation provided in an embodiment of the present application.

[0033] like Figure 1 As shown, the 3D scene reconstruction method based on point cloud augmentation includes the following steps:

[0034] In step S101 , a preliminary point cloud and scene complexity level of a target scene are obtained.

[0035] It is understandable that a point cloud represents a scene as a collection of points, each of which contains position, color, and error information. In the embodiment of the present application, the preliminary point cloud is sparsely distributed and cannot meet the requirements of 3D reconstruction of complex scenes. As input, through feature extraction and matching, the correspondence between multi-view points is obtained. Then, through triangulation, bundle adjustment and other steps, the two-dimensional feature points corresponding to the multi-view trajectory are mapped back to a three-dimensional point in space to obtain a preliminary point cloud. And get the estimated camera pose for each view

[0036] Secondly, the scene complexity of the target scene in the embodiment of the present application directly affects which areas need more attention and higher point cloud density, which is crucial for the selection of subsequent point cloud augmentation strategies. The scene complexity level can be determined by analyzing the structural characteristics within the scene, such as factors such as texture richness and geometric shape complexity. It can also be determined by frequency domain feature analysis, for example, the higher the high-frequency component, the higher the corresponding complexity.

[0037] In step S102, the preliminary point cloud density required for each area in the target scene is determined according to the scene complexity, and the attention area in each perspective image is extracted in combination with the preliminary point cloud density.

[0038] Among them, there is a certain connection between the scene complexity and the density distribution required by the area. Therefore, the embodiment of the present application can determine the density distribution of the point cloud required for each area according to the scene complexity level, which provides a good guidance for the subsequent division of the attention area and the construction of the augmented point cloud.

[0039] In one embodiment of the present application, the attention area in each perspective image is extracted, including: dividing each perspective image into a dense area and a sparse area according to a preliminary point cloud density; performing a two-dimensional Fourier transform on the sparse area, and defining an area with a high-frequency component higher than a preset threshold as a complex structure but sparse point cloud area as the attention area.

[0040] The embodiment of the present application can be guided by the complexity of the reconstructed scene, and the initial point cloud The scene area is divided according to the point cloud density distribution characteristics. Specifically, it is divided into dense areas and sparse areas according to the complexity of the structure and the sparsity of the point cloud. The reconstruction effect of the block with complex structure but sparse point cloud is the worst, and it is set as the attention area M. i , selected as the implementation area for subsequent point cloud augmentation. It can be solved by the following steps:

[0041] Step 1: Based on the point cloud density required for each area in the target scene, various clustering algorithms are used Get the dense blocks in the perspective image, and then take the difference to get the sparse blocks:

[0042] Step 2: Perform two-dimensional Fourier transform on the sparse blocks one by one Using the Threshold Function The blocks with high-frequency components in the frequency domain higher than the set threshold t are retained, and the final attention area is obtained through inverse Fourier transform:

[0043] For different views, perform steps 1 and 2 in sequence to obtain a set of per-view attention regions.

[0044] In step S103, an augmented point cloud is generated according to the attention area in each perspective image, and a target point cloud is obtained by weighting the preliminary point cloud and the augmented point cloud.

[0045] The embodiment of the present application can determine the initial point cloud density distribution required for the scene area by scene complexity, divide the scene area based on the density distribution characteristics of the initial point cloud, construct a scene-adaptive augmented point cloud, and finally weightedly combine them into a high-quality target point cloud. Therefore, based on the correspondence between scene complexity and the required point cloud distribution density, the embodiment of the present application focuses on the tolerance and quantity requirements of 3DGS initialization for point cloud accuracy. Based on this, the algorithm generates an augmented point cloud This improves scene coverage and the effect of sparse-angle three-dimensional scene reconstruction, thereby achieving the purpose of using sparse-angle shooting to increase the reconstruction rate and reduce the difficulty of the data acquisition stage.

[0046] In one embodiment of the present application, an augmented point cloud is generated based on the attention area in each perspective image, including: in the attention area in multiple multi-perspective images, two-dimensional feature point pairs corresponding to the multi-perspective views are obtained by first extracting and then matching, or directly using a non-extraction feature matcher; the two-dimensional feature points corresponding to the two view trajectories are used as options for point cloud generation and mapped to the three-dimensional space together with the multi-view trajectory to generate an augmented point cloud.

[0047] The multiple multi-view images can be disordered or ordered, without specific limitation. The two-view trajectory is that the two-dimensional feature point appears in two different view images, and the multi-view trajectory is that the two-dimensional feature point appears in three or more view images.

[0048] It is understandable that within the attention area, based on the existing camera pose Re-execute feature extraction and matching, and reconstruct the point cloud. In this process, introduce some improved point cloud generation solutions with lenient conditions, such as using pre-trained feature-free matchers, relaxing the number of viewpoints required for local trajectories, increasing the triangulation error threshold, etc. These solutions reduce the requirements for point cloud accuracy to a certain extent, and can fully exploit the viewpoint information in the attention area during the initialization stage.

[0049] During the actual execution process, the embodiments of the present application can obtain the two-dimensional coordinates of the two-dimensional feature points in images with different viewing angles; calculate the actual position and error and other parameters of the two-dimensional feature points in the three-dimensional space based on the two-dimensional coordinates; and generate an augmented point cloud based on the actual position and error and other parameters.

[0050] Specifically, the embodiment of the present application can use n disordered multi-view two-dimensional images in the scene As input, through feature extraction and matching, the point-to-point correspondence of multiple views is obtained. Subsequently, through triangulation, bundle adjustment and other steps, the 2D feature points corresponding to the two view trajectories are used as options for point cloud generation, mapped back to 3D points in 3D space, fully mining the underutilized view information in these areas, and together with the newly generated 3D points under all multi-view trajectories, form an augmented point cloud.

[0051] In one embodiment of the present application, the target point cloud is obtained by weighting the preliminary point cloud and the augmented point cloud.

[0052] Previously, it also includes: obtaining three-dimensional feature points in the preliminary point cloud and the augmented point cloud; calculating the retention probability of each three-dimensional feature point according to the error level of the three-dimensional feature point; determining the point cloud data that needs to be discarded based on the retention probability, wherein the retention probability is inversely proportional to the error level, that is, the higher the error level, the lower the retention probability, the fewer points left in the point cloud, which means that the weight contribution of this point cloud to the final point cloud is lower. The embodiment of the present application can filter part of the target point cloud data to better support the 3DGS reconstruction task.

[0053] In the actual implementation process, the embodiment of the present application can reproject each point in the point cloud back to the image domain of each view. i Located in image I i If the projection point x is within the boundary of i And the corresponding two-dimensional feature truth point The error distance can be Euclidean distance, Manhattan distance, Chebyshev distance, etc. The error level ε is defined as the sum of the error distances under different views:

[0054]

[0055] Where π() is the projection function that maps the 3D points of the scene to the image domain, ρ() is the threshold function used to filter unreasonable observations and eliminate the influence of outliers. The error level ε is the key threshold used by the algorithm to screen the threshold and determine the weight of the point cloud.

[0056] Furthermore, according to formula (1), the augmented point cloud is calculated The error level ε a and the initial point cloud The error level ε c Obviously, The probability of retention will be higher than Low, has a lower weight coefficient when combined.

[0057] Specifically, the embodiment of the present application can compare the error levels ε of the three-dimensional feature points in the two groups of point clouds. c and ε aThe point cloud weight function used to model the initial point filtering is:

[0058] p c (ε c )=θ c C(ε c ) (2)

[0059] p a (ε a )=θ a C(ε a ) (3)

[0060] Among them, pc and pa represent point clouds. The probability of an inlier being retained. The lower the point cloud weight, the higher the probability of a point being removed. θ c ,θ a is the corresponding normalization factor, which normalizes the value to [0, 1]. C is the probability function of retaining three-dimensional feature points, which is negatively correlated with the error level and can be expressed in a variety of forms, including but not limited to inverse proportional relationship, exponential relationship, Gaussian function relationship, etc.

[0061] Finally, a high-quality target point cloud It is composed of a weighted combination of the preliminary point cloud and the augmented point cloud:

[0062]

[0063] for c represents a probability, Represents a collection, Indicates that Each element in is extracted according to probability c to get a new subset.

[0064] In step S104, a three-dimensional representation of the target scene is reconstructed using the target point cloud.

[0065] In one embodiment of the present application, a three-dimensional representation of a target scene is reconstructed using a target point cloud, including: using an efficient training mechanism and a target point cloud to reconstruct a three-dimensional representation of the target scene, wherein the efficient training mechanism is obtained after adjustment according to a 3DGS training mechanism, and the adjustment parameters of the 3DGS training mechanism include: reducing the number of iterations, introducing adaptive density control, reducing the frequency of use of adaptive density control, and removing the reset opacity operation.

[0066] It can be understood that the embodiment of the present application designs a 3DGS parameter training mechanism in the scene reconstruction stage, which can better adapt to the reconstruction task of high-quality point cloud (target point cloud) under sparse angle conditions. The adjusted training mechanism includes:

[0067] First, the three-dimensional scene of the target scene is reconstructed using a high-quality point cloud. Since the high-quality point cloud provides a more accurate initial condition, the overall number of iterations can be reduced in the actual reconstruction process. Second, considering that high-quality point clouds require more iterations to converge, the embodiment of the present application delays the introduction of the ADC (Adaptive Density Control) mechanism. ADC is responsible for periodically splitting, cloning or clipping the Gaussian kernel, so that the number of Gaussian kernels changes. Therefore, by postponing the application of ADC, it is ensured that the existing Gaussian kernel representation has fully converged before the next step is performed, and then a new Gaussian kernel is added to improve the scene representation capability. Third, since the Gaussian kernel after the high-quality point cloud is initialized is relatively accurate, the embodiment of the present application can reduce the need for frequent use of ADC. Fourth, in sparse angle reconstruction, the practice of periodically clipping the low-opaque Gaussian kernel may cause the number of Gaussians to be excessively reduced, especially after resetting the opacity, thereby affecting the success rate of reconstruction. Therefore, the embodiment of the present application can remove the reset opacity operation to ensure the integrity and accuracy of the reconstruction.

[0068] The following is a detailed introduction to the use of target training mechanism and target point cloud to reconstruct the three-dimensional scene of the target scene in the embodiment of the present application with reference to specific examples. The 3DGS of the embodiment of the present application can represent the scene as a three-dimensional Gaussian kernel with several learnable parameters, wherein the three-dimensional Gaussian kernel is characterized by mean, covariance, spherical harmonic function coefficients and opacity.

[0069] The present application embodiment can be given a previously generated high-quality point cloud 3DGS models each point in it as a 3D Gaussian kernel, obtaining a preliminary representation of the scene:

[0070]

[0071] In the subsequent optimization training process, the embodiment of the present application can randomly extract one of the perspectives, splash the three-dimensional Gaussian located within the image boundary into the image domain, and reduce the dimension into a two-dimensional Gaussian with a clear center and blurred boundaries;

[0072] Query the color c of each 2D Gaussian at each pixel. Calculate the opacity α′ at each pixel:

[0073]

[0074] where ∑ 2D , μ ′ They are the three-dimensional covariance ∑ of the Gaussian kernel, the projection of the three-dimensional mean μ in the image domain, and x ′ are pixel coordinates.

[0075] According to the depth value at this perspective, Two-dimensional Gaussians are sorted in sequence;

[0076] Calculate the color value at the pixel coordinates through α-rendering, which is the predicted value of 3DGS:

[0077]

[0078] Substitute the predicted value and the true value into the loss function By updating the trainable parameters through error backpropagation and calling ADC according to the training strategy, 3DGS can train a nearly convergent scene representation based on the initial point cloud and finally obtain a realistic three-dimensional scene.

[0079] In the actual implementation process, the embodiment of the present application can reduce the overall number of iterations of sparse angle reconstruction, thereby significantly shortening the training time while maintaining a similar reconstruction effect. By increasing the periodic interval of implementing ADC, 3DGS can reasonably integrate the newly generated Gaussian kernel into the scene during the interval, reduce the number of Gaussians in the final scene representation, and effectively prevent overfitting of complex areas of the scene. Removing the opacity reset operation avoids excessive reduction in the number of Gaussians during sparse angle reconstruction.

[0080] Furthermore, the 3D scene after training and reconstruction can be visualized through professional software. By inputting the camera pose of the perspective of interest, 3DGS can present the corresponding rendering results in real time.

[0081] Combine the following Figure 2 The three-dimensional scene reconstruction method based on point cloud augmentation of the embodiment of the present application is described in detail, including an initialization stage and a reconstruction stage, wherein the initialization stage is to determine the initial point cloud density distribution required for the area within the scene by scene complexity, divide the scene area in combination with the density distribution characteristics of the preliminary point cloud, construct a scene-adaptive augmented point cloud, and finally weightedly combine them into a high-quality target point cloud, which greatly increases the number of initial points and balances their density distribution. The reconstruction stage is to perform three-dimensional reconstruction of the target scene based on the weighted target point cloud, better adapt to the reconstruction task of high-quality point clouds in sparse angle conditions, and greatly improve the computational efficiency and reduce memory overhead while maintaining sufficient resolution and accuracy.

[0082] In the actual execution process, the 3DGS algorithm used for scene reconstruction in the embodiment of the present application requires scene camera pose parameters and camera internal participating point cloud as initialization input. Among them, the camera internal parameters (focal length, principal point, pixel size, etc.) and camera pose (perspective transformation matrix) are used as the main basis for rendering pipeline direction selection and transformation. Point cloud is a collection of scenes represented as a series of points, each point containing position, color and error information. After the reconstruction phase of 3D Gaussian begins, these points are modeled into a series of three-dimensional Gaussian ellipsoids, and are characterized by the mean, covariance, spherical harmonic function coefficients, and opacity. These properties form a preliminary interpretation of the scene, and are subsequently optimized through gradient descent and density control, including the following aspects:

[0083] 1) Preliminary point cloud and camera pose estimation

[0084] For n views, after feature extraction and matching, a set of estimated initial camera poses can be obtained The input of the point cloud reconstruction stage includes n scene views and its associated camera pose estimate COLMAP triangulation method using two different views Generate preliminary point cloud

[0085]

[0086] After completing the new image registration, bundle adjustment is performed to improve the camera pose P i and the coordinate parameters of the 3D point X, thereby minimizing the reprojection error and filtering out observations with large errors:

[0087]

[0088] The 3D point must be visible in at least three views to be considered a valid point. Updated to And the new three-dimensional point X * Add to Point Cloud Re-triangulation measurement is performed to track the 2D feature points that were not successfully triangulated before the new image registration to prevent missing valid information.

[0089] Then the next image registration is started, and the bundle adjustment and triangulation measurement are performed alternately until all images are registered. After the global bundle adjustment, the optimal camera pose parameter set is finally obtained. And the preliminary point cloud Calculate the average error level of each point in the point cloud as the error level of the preliminary point cloud as a whole

[0090]

[0091] The square of the Euclidean distance was used as the error distance.

[0092] 2) Delineation of attention areas

[0093] Considering that DetectorfreeSfM uses a non-extracted feature matcher, it can greatly improve the matching degree of feature points in weak texture areas. Therefore, in the division of the attention area, the area with simple structure and sparse point cloud is retained, and the feature matcher is used to extract more structural and geometric information in the area, thereby obtaining a better reconstruction effect.

[0094] Based on preliminary point cloud Projection to View I i In the density distribution of the point cloud, the embodiment of the present application can use DBSCAN (Density-Based Spatial Clustering of Applications with Noise, density-based spatial clustering algorithm) to divide the point cloud dense area, and then use the difference set to obtain the attention area M under this perspective. i :

[0095]

[0096] Among them, ∈ and μ are two parameters of the DBSCAN method. ∈ is set to 1% of the image resolution, representing the radius of the search area around the starting point; μ is set to 15, representing the minimum number of samples for considering the starting point and surrounding points as a cluster. For each view, use formula (11) to specify the set of attention areas

[0097] 3) Define augmented point cloud

[0098] In the attention area, the DetectorfreeSfM extraction-free feature matcher is used to directly obtain the multi-view point-to-point correspondence, especially in the weak texture area, which can obtain two-dimensional feature points several times that of COLMAP. Furthermore, in the point cloud reconstruction stage, the camera pose estimated by COLMAP can be used to obtain the corresponding point-to-point relationship of the point cloud. Relax the number of views for the local track, keeping the two-view track as an option for point cloud generation:

[0099]

[0100]

[0101] The three-dimensional point X ′* Add to Point Cloud Similar to the previous steps of initial point cloud reconstruction, the augmented point cloud is constructed Calculate the error level of the overall augmented point cloud

[0102]

[0103] 4) Define high-quality target point cloud

[0104] High-quality point cloud for 3DGS initialization in the embodiment of the present application The above preliminary point cloud With augmented point cloud The point cloud weight function is related to the overall error level of the point cloud, as shown in equations (2) and (3). The retention probability function is defined as being inversely proportional to the error level, that is:

[0105]

[0106] Among them, β is a factor that controls the influence of the error level on the retention probability and is generally set to 1. Because the range of C(ε) is (0,1), the normalization factor α in equations (2) and (3) can be set to 1.

[0107] High-quality target point cloud Calculated by the following formula:

[0108]

[0109] 5) 3DGS reconstruction parameter optimization

[0110] As an example of possible implementation, the optimization training process described in the embodiment of the present application specifically adopts the following implementation method:

[0111] The iteration cycle is set to 5000 times. The training method in each iteration cycle is consistent with that described in the above embodiment. As an example, the loss function is expressed as a weighted combination of L1 loss and SSIM loss:

[0112]

[0113] In the experiment, RGB Set to 0.2. Set the rendering value obtained by 3DGS training at a certain viewing angle to I render , the corresponding truth value is I gt , then the L1 loss is defined as:

[0114]

[0115] The SSIM loss is defined as:

[0116]

[0117] The optimizer used is Adam, and the opacity reset operation is removed.

[0118] High-quality point clouds contain a lot of scene structure and detail information. Therefore, ADC is not introduced in the first 2000 iterations. 3DGS is optimized based on the initial point cloud to obtain a fully converged preliminary representation of the scene.

[0119] In the subsequent 2500 iterations, ADC is introduced to enhance the representation capability of 3DGS. The period interval for implementing ADC is set to 200 to ensure that the algorithm can effectively integrate the newly generated Gaussian kernel into the scene representation within the interval.

[0120] In the last 500 iterations, the reconstruction results are fine-tuned without increasing the number of Gaussian kernels.

[0121] According to the three-dimensional scene reconstruction method based on point cloud augmentation proposed in the embodiment of the present application, by obtaining the preliminary point cloud and scene complexity level of the target scene, the point cloud density required for each area is determined by using the complexity of the scene, and the reconstruction effect of the area with complex structure but sparse point cloud is mainly improved, and the overall density distribution of the point cloud is effectively balanced. In addition, by generating augmented point clouds in the attention area, the perspective information in the scene can be fully mined, especially in areas with sparse point cloud distribution, which greatly increases the number of initial points and makes the reconstructed three-dimensional scene more accurate and detailed. Therefore, the problems of limited point cloud representation ability and insufficient sparse perspective coverage in the related technology, which lead to poor quality of three-dimensional reconstruction, are solved.

[0122] Next, a three-dimensional scene reconstruction device based on point cloud augmentation proposed in accordance with an embodiment of the present application is described with reference to the accompanying drawings.

[0123] Figure 3 It is a block diagram of a three-dimensional scene reconstruction device based on point cloud augmentation according to an embodiment of the present application.

[0124] like Figure 3 As shown, the three-dimensional scene reconstruction device 10 based on point cloud augmentation includes: an acquisition module 100, a determination module 200, a processing module 300 and a reconstruction module 400.

[0125] Among them, the acquisition module 100 is used to obtain the preliminary point cloud and scene complexity level of the target scene; the determination module 200 is used to determine the preliminary point cloud density required for each area in the target scene according to the scene complexity, and extract the attention area in each perspective image in combination with the preliminary point cloud density; the processing module 300 is used to generate an augmented point cloud based on the attention area in each perspective image, and obtain the target point cloud according to the weighted preliminary point cloud and the augmented point cloud; the reconstruction module 400 is used to reconstruct the three-dimensional scene of the target scene using the target point cloud.

[0126] In one embodiment of the present application, the processing module 300 is further used to divide each perspective image into dense areas and sparse areas according to the preliminary point cloud density distribution; perform a two-dimensional Fourier transform on the sparse area, and define the area with a high-frequency component higher than a preset threshold as a complex structure but sparse point cloud area as an attention area.

[0127] In one embodiment of the present application, the processing module 300 is further used to obtain two-dimensional feature point pairs corresponding to multi-view views in the attention area of ​​multiple multi-view images by first extracting and then matching, or directly using a non-extraction feature matcher, to extract the two-dimensional feature points of the preliminary point cloud in the multi-view images; the two-dimensional feature points corresponding to the two view trajectories are used as options for point cloud generation and mapped to the three-dimensional space together with the multi-view trajectory to generate an augmented point cloud.

[0128] In one embodiment of the present application, the two-view trajectory is a two-dimensional feature point that appears in two different viewing angle images, and the multi-view trajectory is a two-dimensional feature point that appears in three or more viewing angle images.

[0129] In one embodiment of the present application, the processing module 300 is further used to obtain the two-dimensional coordinates of the two-dimensional feature points in images of different viewing angles; calculate the actual position and error and other parameters of the two-dimensional feature points in the three-dimensional space based on the two-dimensional coordinates; and generate an augmented point cloud based on the actual position and error and other parameters.

[0130] In one embodiment of the present application, the reconstruction module 400 is further used to reconstruct a three-dimensional scene of a target scene using an efficient training mechanism and a target point cloud, wherein the efficient training mechanism is obtained after adjustment according to the 3DGS training mechanism, and the adjustment parameters of the 3DGS training mechanism include: reducing the number of iterations, introducing adaptive density control, reducing the frequency of use of adaptive density control, and removing the reset opacity operation.

[0131] In one embodiment of the present application, the three-dimensional scene reconstruction device 10 based on point cloud augmentation also includes: a filtering module, which is used to obtain three-dimensional feature points in the preliminary point cloud and the augmented point cloud before obtaining the target point cloud by weighting the preliminary point cloud and the augmented point cloud; calculate the retention probability of each three-dimensional feature point according to the error level of the three-dimensional feature point, determine the target point cloud data based on the retention probability, and eliminate part of the target point cloud data, wherein the retention probability is inversely proportional to the error level.

[0132] It should be noted that the aforementioned explanation of the embodiment of the three-dimensional scene reconstruction method based on point cloud augmentation is also applicable to the three-dimensional scene reconstruction device based on point cloud augmentation in this embodiment, and will not be repeated here.

[0133] According to the 3D scene reconstruction device based on point cloud augmentation proposed in the embodiment of the present application, by obtaining the preliminary point cloud and scene complexity level of the target scene, the point cloud density required for each area is determined by using the complexity of the scene, thereby improving the reconstruction effect of areas with complex structures but sparse point clouds, effectively balancing the overall density distribution of the point cloud, and by generating augmented point clouds in the attention area, it is possible to fully mine the perspective information in the scene, especially in areas with sparse point cloud distribution, greatly increasing the number of initial points, making the reconstructed 3D scene more accurate and detailed. Thus, the problems of limited point cloud representation capabilities and insufficient sparse perspective coverage in related technologies, resulting in poor quality of 3D reconstruction, are solved.

[0134] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0135] Memory 401 , processor 402 , and a computer program stored in the memory 401 and executable on the processor 402 .

[0136] When the processor 402 executes the program, the three-dimensional scene reconstruction method based on point cloud augmentation provided in the above embodiment is implemented.

[0137] Furthermore, the electronic device further comprises:

[0138] The communication interface 403 is used for communication between the memory 401 and the processor 402 .

[0139] The memory 401 is used to store computer programs that can be executed on the processor 402 .

[0140] The memory 401 may include a high-speed RAM (Random Access Memory) memory, and may also include a non-volatile memory, such as at least one disk memory.

[0141] If the memory 401, the processor 402 and the communication interface 403 are implemented independently, the communication interface 403, the memory 401 and the processor 402 can be connected to each other through a bus and communicate with each other. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0142] Optionally, in a specific implementation, if the memory 401, the processor 402 and the communication interface 403 are integrated on a chip, the memory 401, the processor 402 and the communication interface 403 can communicate with each other through an internal interface.

[0143] The processor 402 may be a CPU (Central Processing Unit), or an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application.

[0144] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned three-dimensional scene reconstruction method based on point cloud augmentation.

[0145] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms are not necessarily directed to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0146] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0147] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0148] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array, a field programmable gate array, etc.

[0149] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0150] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A three-dimensional scene reconstruction method based on point cloud augmentation, characterized in that: The following steps are involved: Obtain preliminary point cloud and scene complexity level of the target scene; Determine the preliminary point cloud density required for each area in the target scene according to the scene complexity, and extract the attention area in each perspective image based on the preliminary point cloud density; Generate an augmented point cloud according to the attention area in each perspective image, and obtain a target point cloud by weighting the preliminary point cloud and the augmented point cloud; A three-dimensional representation of the target scene is reconstructed using the target point cloud.

2. The three-dimensional scene reconstruction method based on point cloud augmentation according to claim 1, characterized in that: The extracting of the attention area in each perspective image includes: Dividing each of the perspective images into a dense area and a sparse area according to the preliminary point cloud density; A two-dimensional Fourier transform is performed on the sparse area, and an area with a high-frequency component higher than a preset threshold is defined as an area with a complex structure but a sparse point cloud, which serves as the attention area.

3. The three-dimensional scene reconstruction method based on point cloud augmentation according to claim 1 or 2, characterized in that: Generating an augmented point cloud according to the attention area in each perspective image includes: In the attention area of ​​multiple multi-view images, two-dimensional feature point pairs corresponding to the multi-view views are obtained by first extracting and then matching, or directly using a feature matcher without extraction; the two-dimensional feature points corresponding to the two-view trajectories are used as options for point cloud generation and mapped to the three-dimensional space together with the multi-view trajectories to generate the augmented point cloud.

4. The three-dimensional scene reconstruction method based on point cloud augmentation according to claim 3, characterized in that: The two-view trajectory is that the two-dimensional feature point appears in two different viewing angle images, and the multi-view trajectory is that the two-dimensional feature point appears in three or more viewing angle images.

5. The three-dimensional scene reconstruction method based on point cloud augmentation according to claim 4, characterized in that: The method of mapping the two-dimensional feature points corresponding to the two view trajectories as the optional items for generating the point cloud together with the multi-view trajectories to the three-dimensional space to generate the augmented point cloud includes: Obtaining the two-dimensional coordinates of the two-dimensional feature points in images of different viewing angles; Calculate the actual position and position error of the two-dimensional feature point in the three-dimensional space according to the two-dimensional coordinates; The augmented point cloud is generated based on the actual position and the position error.

6. The three-dimensional scene reconstruction method based on point cloud augmentation according to claim 1, characterized in that: The step of reconstructing the three-dimensional representation of the target scene using the target point cloud comprises: A three-dimensional representation of the target scene is reconstructed using an efficient training mechanism and the target point cloud, wherein the efficient training mechanism is obtained after adjustment according to a 3DGS training mechanism, and adjustment parameters of the 3DGS training mechanism include: reducing the number of iterations, delaying the introduction of adaptive density control, reducing the frequency of use of adaptive density control, and removing the reset opacity operation.

7. The three-dimensional scene reconstruction method based on point cloud augmentation according to claim 1, characterized in that: Before obtaining the target point cloud by weighting the preliminary point cloud and the augmented point cloud, it also includes: Acquire three-dimensional feature points in the preliminary point cloud and the augmented point cloud; The retention probability of each three-dimensional feature point is calculated according to the error level of the three-dimensional feature point, the target point cloud data is determined based on the retention probability, and part of the target point cloud data is eliminated, wherein the retention probability is inversely proportional to the error level.

8. A three-dimensional scene reconstruction device based on point cloud augmentation, characterized in that: include: An acquisition module, used to obtain the preliminary point cloud and scene complexity level of the target scene; A determination module determines the preliminary point cloud density required for each area in the target scene according to the scene complexity, and extracts the attention area in each perspective image based on the preliminary point cloud density; A processing module, configured to generate an augmented point cloud according to the attention area in each of the perspective images, and obtain a target point cloud by weighting the preliminary point cloud and the augmented point cloud; A reconstruction module is used to reconstruct a three-dimensional representation of the target scene using the target point cloud.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the three-dimensional scene reconstruction method based on point cloud augmentation as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed, the three-dimensional scene reconstruction method based on point cloud augmentation described in any one of claims 1 to 7 is implemented.

Images