Three-dimensional reconstruction method and device based on monocular camera

Abstract

The application provides a monocular camera-based three-dimensional reconstruction method and device. The method comprises: performing format standardization processing on an image sequence collected by a monocular camera to obtain a data set containing each image sequence in a preset format, and generating a sparse point cloud based on the data set; initializing a two-dimensional Gaussian splash point cloud based on the sparse point cloud; iteratively optimizing Gaussian point cloud parameters of the two-dimensional Gaussian splash point cloud, and performing an adaptive pruning operation in the iterative optimization process to obtain an iteratively optimized Gaussian splash point cloud corresponding to the two-dimensional Gaussian splash point cloud; and outputting the iteratively optimized Gaussian splash point cloud as a three-dimensional reconstruction model corresponding to the image sequence. The application can reduce local redundancy, thereby reducing memory occupation and calculation overhead of the algorithm.

Description

Technical Field

[0001] This application relates to the fields of computer vision and robotics, and in particular to a three-dimensional reconstruction method and device based on a monocular camera. Background Technology

[0002] In the fields of computer vision and robotics, 3D reconstruction technology aims to recover the 3D structure of a scene from a 2D image and is widely used in virtual reality, autonomous driving, and drone navigation. Traditional methods rely on multi-view cameras or depth sensors, but these devices are expensive and complex to deploy. Monocular cameras have become a research hotspot due to their low cost and miniaturization advantages; however, monocular images lack depth information, and the reconstruction process often faces problems of high computational complexity and memory consumption.

[0003] Existing 3D reconstruction algorithms, such as those based on Structure of Motion (SfM) or Neural Radiation Field (NeRF), while capable of generating detailed models, require substantial memory storage for point cloud or voxel data, making real-time operation difficult on resource-constrained embedded devices. Among existing 3D reconstruction algorithms, 3D Gaussian Splashing (3DGS) is a representative method. It uses a 3D Gaussian distribution to implicitly represent the scene surface, generating a continuous model by optimizing Gaussian parameters (such as position, covariance, and weights). 3DGS produces high-quality reconstruction results, but requires storing large amounts of 3D Gaussian data, resulting in extremely high memory consumption and hindering real-time operation on resource-constrained devices. In recent years, 2DGS has evolved from 3DGS, addressing viewpoint inconsistencies and improving reconstruction accuracy by projecting Gaussian splashing onto a 2D image plane. Summary of the Invention

[0004] In view of this, embodiments of this application provide a three-dimensional reconstruction method and device based on a monocular camera to eliminate or improve one or more defects existing in the prior art.

[0005] One aspect of this application provides a 3D reconstruction method based on a monocular camera, the method comprising the following steps: The image sequences acquired by the monocular camera are format-standardized to obtain a dataset containing various image sequences in a preset format, and a sparse point cloud is generated based on the dataset. Based on the sparse point cloud, a two-dimensional Gaussian splash point cloud is initialized, wherein each Gaussian element in the two-dimensional Gaussian splash point cloud is represented by a two-dimensional Gaussian disk; The Gaussian point cloud parameters of the two-dimensional Gaussian splatter point cloud are iteratively optimized, and an adaptive pruning operation is performed during the iterative optimization process. The adaptive pruning operation includes: for each two-dimensional Gaussian disk, determining a local spatial range based on its projection information from multiple monocular camera views, and calculating the number of other two-dimensional Gaussian disks located within this local spatial range as the redundancy count of that two-dimensional Gaussian disk; dynamically determining a pruning threshold based on the statistical distribution of the redundancy counts of each two-dimensional Gaussian disk; and removing two-dimensional Gaussian disks whose redundancy counts exceed the pruning threshold based on the pruning threshold, thereby obtaining the iteratively optimized Gaussian splatter point cloud corresponding to the two-dimensional Gaussian splatter point cloud. The Gaussian splatter point cloud, after iterative optimization, is output as the 3D reconstruction model corresponding to the image sequence.

[0006] In some embodiments of this application, the loss function in the iterative optimization process is a weighted average of the L1 term and the SSIM exponent term; wherein, the L1 term is the L1 term between the 3D reconstruction model corresponding to the image sequence and the real image in the image sequence; and the SSIM exponent term is the SSIM exponent between the 3D reconstruction model corresponding to the image sequence and the real image in the image sequence.

[0007] In some embodiments of this application, determining a local spatial range for each of the two-dimensional Gaussian disks based on its projection information from the multiple monocular camera viewpoints includes: For each of the two-dimensional Gaussian disks, the radius of a query sphere is determined based on the number of pixels it occupies in the two-dimensional projection of the multiple monocular camera views. The local spatial range is determined with the two-dimensional Gaussian disk as the center and the radius of the query sphere as the reference.

[0008] In some embodiments of this application, the dynamic determination of the pruning threshold is specifically calculated using the following formula: τ = mean(C) + λ·std(C) Where τ is the pruning threshold, C is the set of redundant counts of all two-dimensional Gaussian disks, mean(C) is the mean of C, std(C) is the standard deviation of C, and λ is a preset hyperparameter.

[0009] In some embodiments of this application, the step of removing two-dimensional Gaussian disks with redundancy counts exceeding the pruning threshold based on the pruning threshold to obtain the iteratively optimized Gaussian splatter point cloud corresponding to the two-dimensional Gaussian splatter point cloud includes: Based on the pruning threshold, from the two-dimensional Gaussian disks whose redundancy count exceeds the pruning threshold, two-dimensional Gaussian disks with transparency less than a preset transparency threshold are selected and removed to obtain the iteratively optimized Gaussian splatter point cloud corresponding to the two-dimensional Gaussian splatter point cloud.

[0010] In some embodiments of this application, the iterative optimization process further includes periodically performing point cloud density control operations, the point cloud density control operations including: For a two-dimensional Gaussian disk with a gradient value greater than a preset gradient threshold, if the size of the two-dimensional Gaussian disk is smaller than a preset size threshold, then clone an identical two-dimensional Gaussian disk and adjust the positions of the two two-dimensional Gaussian disks to fill the empty part in the space range corresponding to the size threshold. If the size of the two-dimensional Gaussian disk is larger than a preset size threshold, the two-dimensional Gaussian disk is split into two two-dimensional Gaussian disks with a smaller size to fill the empty space within the space corresponding to the size threshold.

[0011] In some embodiments of this application, the method further includes: The storage and calculation precision of at least one parameter in the Gaussian splash point cloud used to adjust the geometric properties or spatial position of the Gaussian disk is converted from a first precision format to a second precision format, which is lower than the first precision.

[0012] In some embodiments of this application, the method further includes: The K-means algorithm is used to cluster a set of parameter vectors in the two-dimensional Gaussian splash point cloud that represent a geometric property or spatial location feature of the Gaussian disk into a preset number of codebook vectors, and the codebook index is used to replace the set of parameter vectors for storage.

[0013] In some embodiments of this application, the method further includes: The order of the spherical harmonic function used to represent the color of the two-dimensional Gaussian disk is dynamically adjusted based on the contribution of a single two-dimensional Gaussian disk to the reconstruction quality.

[0014] Another aspect of this application provides an electronic device including a processor and a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the aforementioned monocular camera-based 3D reconstruction method.

[0015] A third aspect of this application provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the described monocular camera-based 3D reconstruction method.

[0016] A fourth aspect of this application provides a computer program product comprising a computer program that, when executed by a processor, implements the aforementioned monocular camera-based 3D reconstruction method.

[0017] This application discloses a 3D reconstruction method based on a monocular camera, comprising the following steps: standardizing the image sequences acquired by the monocular camera to obtain a dataset containing various image sequences of a preset format, and generating a sparse point cloud based on the dataset; initializing a two-dimensional Gaussian splatter point cloud based on the sparse point cloud, wherein each Gaussian element in the two-dimensional Gaussian splatter point cloud is represented by a two-dimensional Gaussian disk; iteratively optimizing the Gaussian point cloud parameters of the two-dimensional Gaussian splatter point cloud, and performing an adaptive pruning operation during the iterative optimization process; wherein the adaptive pruning operation includes: for each of the two-dimensional Gaussian disks, A local spatial range is determined based on the projection information of the two-dimensional Gaussian disks under the multiple monocular camera views. The number of other two-dimensional Gaussian disks located within this local spatial range is calculated as the redundancy count of the two-dimensional Gaussian disks. A pruning threshold is dynamically determined based on the statistical distribution of the redundancy counts of each two-dimensional Gaussian disk. Based on the pruning threshold, two-dimensional Gaussian disks with redundancy counts exceeding the pruning threshold are removed to obtain the iteratively optimized Gaussian splatter point cloud corresponding to the two-dimensional Gaussian splatter point cloud. The iteratively optimized Gaussian splatter point cloud is output as the 3D reconstruction model corresponding to the image sequence. This method can reduce local redundancy, balance the overall reconstruction quality, and reduce the number of redundant parameters in the point cloud, thereby reducing the algorithm's memory usage and computational overhead.

[0018] Additional advantages, objectives, and features of this application will be set forth in part in the description which follows, and will in part become apparent to those skilled in the art upon review of the following description, or may be learned by practice of the application. The objectives and other advantages of this application can be realized and obtained by means of the structures specifically pointed out in the specification and drawings.

[0019] Those skilled in the art will understand that the purposes and advantages that can be achieved with this application are not limited to those specifically described above, and that the above and other purposes that this application can achieve will be more clearly understood from the following detailed description. Attached Figure Description

[0020] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, do not constitute a limitation thereof. The components in the drawings are not drawn to scale but are merely for illustrating the principles of this application. For ease of illustration and description of certain parts of this application, corresponding portions in the drawings may be enlarged, i.e., may appear larger relative to other components in an exemplary device actually manufactured according to this application. In the drawings: Figure 1 This is a schematic diagram of the first process of a three-dimensional reconstruction method based on a monocular camera in one embodiment of this application.

[0021] Figure 2 This is a schematic diagram of a second process of a three-dimensional reconstruction method based on a monocular camera in one embodiment of this application.

[0022] Figure 3 This is a flowchart illustrating a specific example of a 3D reconstruction method based on a monocular camera in this application.

[0023] Figure 4 This is a schematic diagram illustrating the cloning and splitting process of a Gaussian disk in a specific example of a monocular camera-based 3D reconstruction method in this application.

[0024] Figure 5 This is a redundancy counting diagram of the adaptive pruning algorithm for a monocular camera-based 3D reconstruction method in a specific example of this application. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and their descriptions are used to explain this application, but are not intended to limit it.

[0026] It should also be noted that, in order to avoid obscuring this application with unnecessary details, only the structures and / or processing steps closely related to the solution according to this application are shown in the accompanying drawings, while other details that are not closely related to this application are omitted.

[0027] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0028] It should also be noted that, unless otherwise specified, the term "connection" in this article can refer not only to a direct connection, but also to an indirect connection involving an intermediary.

[0029] In the following description, embodiments of the present application will be illustrated with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar parts, or the same or similar steps.

[0030] It's important to note that existing SfM and Multi-View Stereo (MVS) workflows generate sparse point clouds through feature point extraction, matching, and triangulation, and then use dense reconstruction algorithms to generate 3D models. This method relies on high-precision feature descriptors and large amounts of memory to store matching data. The SfM-MVS workflow is memory-intensive, especially when processing high-resolution images, where point cloud data and the matching matrix can consume gigabytes of memory, resulting in slow reconstruction speeds and susceptibility to feature matching errors, making it unsuitable for real-time applications. NeRF-like methods are deep learning-based approaches that implicitly represent scenes through neural networks, learning 3D structures from monocular videos. These methods can generate high-quality reconstruction results, but require large amounts of training data and GPU memory. Training and inference have huge memory requirements, the model has many parameters, and high computational resource demands, making them difficult to deploy on mobile devices and limiting their generalization capabilities. 3DGS-based methods directly optimize Gaussian distributions in 3D space, suitable for offline high-precision reconstruction. However, these methods require maintaining complete rendering scenes and Gaussian point cloud data, resulting in high memory consumption (storing millions of Gaussian parameters), poor real-time performance, and high hardware requirements, making them unsuitable for direct streaming processing with monocular cameras. Therefore, the inventors of this application first considered memory optimization for Gaussian splashing algorithms by projecting 3D Gaussian data onto a 2D image plane for calculation, avoiding 3D data redundancy in 3DGS. This application introduces sparse Gaussian management and dynamic level of detail (LOD) mechanisms to retain only the Gaussian distribution in key regions; simultaneously, quantization techniques are used to compress Gaussian parameters. Combined with monocular camera video streams, the algorithm's memory footprint and computational overhead are further reduced, enabling rapid 3D reconstruction.

[0031] The following examples will provide a detailed description.

[0032] This application provides a 3D reconstruction method based on a monocular camera, see [link to relevant documentation]. Figure 1 The method includes the following steps: Step 100: Standardize the image sequences acquired by the monocular camera to obtain a dataset containing each image sequence in a preset format, and generate a sparse point cloud based on the dataset. In step 100, the image sequence includes real-time image sequences acquired by a monocular camera or image sequences obtained by frame extraction from video captured by a monocular camera. These images, as the dataset used by the algorithm, should be processed into a project structure conforming to the COLMAP format. The sparse point cloud generation process is as follows: First, a correspondence search is performed. In this part, COLMAP first extracts features from each input image, searching for extreme points in multiple scale spaces of the image to determine the position, scale, and orientation of the feature points; then, feature matching is performed to find features representing the same physical point in different images; finally, geometric verification is performed. Pure feature matching contains a large number of outliers, which must be filtered through geometric constraints to output a verified scene graph, where nodes are images and edges are verified matching pairs. At this point, the correspondence search is completed. Next, incremental reconstruction is performed, using an incremental strategy, starting from a pair of images and gradually "growing" the entire scene. The initialization process begins by selecting a pair of optimal images from the scene graph as starting points. These images must have a sufficient number of matching points and a wide enough baseline to form a stable triangulation structure. Then, two-view reconstruction is performed on this pair of images. The initial camera pose is obtained by decomposing the fundamental matrix, and the first batch of 3D points is triangulated. After initialization, image registration is performed. In each iteration, a new image is selected that is most closely associated with the currently reconstructed model through feature matching. Once a new image is registered, it brings two main types of new information: existing 3D points are observed in the image's viewpoint, increasing observation redundancy; and new matching pairs are formed with other registered images, allowing for the triangulation of new 3D points. After image registration, Bundle Adjustment (BA) is performed. This is the core optimization step of SfM, used to eliminate accumulated errors. This is a large-scale nonlinear least-squares optimization problem, aiming to minimize the reprojection error of all observation points. After triangulation and basis recognition (BA), there will always be some erroneous 3D points (such as those located behind the camera, with excessive reprojection errors, or with excessively small triangulation angles). COLMAP performs filtering after each step to directly remove these bad points. After the above loop (registration, triangulation, BA, and filtering in sequence), until no image can be registered, the final output is: sparse point cloud (discrete 3D sampling of the scene structure), camera parameters (the precise pose (R,t) of each image represented by a matrix composed of rotation parameters R and translation parameters t), and intrinsic parameters (Focal Length, Distortion), which serve as the direct input data for the Gaussian splashing algorithm.

[0033] Step 200: Based on the sparse point cloud, initialize a two-dimensional Gaussian splash point cloud, wherein each Gaussian element in the two-dimensional Gaussian splash point cloud is represented by a two-dimensional Gaussian disk; In step 200, based on the sparse point cloud, an initial two-dimensional Gaussian splatter point cloud represented by two-dimensional Gaussian disks is created, and each two-dimensional Gaussian disk is assigned initial attribute parameters. These attribute parameters can be point coordinates, spherical harmonic features (mainly diffuse color), other spherical harmonic features, scaling factor, rotation angle, and transparency. Point coordinates are the three-dimensional coordinates of each point in the two-dimensional Gaussian splatter point cloud, which can also be understood as the location of the median of the Gaussian distribution. The spherical harmonic feature is the first spherical harmonic coefficient, which is a three-dimensional vector used to represent the base color. The other spherical harmonic features are the other spherical harmonic function coefficients, which are used to represent the details and variations of the color, and are multi-dimensional vectors. The scaling factor represents the scaling factor of each two-dimensional Gaussian disk in the horizontal and vertical coordinate directions to control the shape of the Gaussian disk, and is a two-dimensional vector. The rotation angle represents the rotation angle of each two-dimensional Gaussian disk in each direction, which can be understood as the rotation direction of the Gaussian disk, and is a four-dimensional vector. Transparency is a one-dimensional vector representing the transparency of each Gaussian disk. In addition, optimizers and learning rate schedulers can be set for the Gaussian model parameters.

[0034] Step 300: Iteratively optimize the Gaussian point cloud parameters of the two-dimensional Gaussian splatter point cloud, and perform an adaptive pruning operation during the iterative optimization process; wherein, the adaptive pruning operation includes: for each two-dimensional Gaussian disk, determining a local spatial range based on its projection information under the multiple monocular camera views, and calculating the number of other two-dimensional Gaussian disks located within the local spatial range as the redundancy count of the two-dimensional Gaussian disk; dynamically determining a pruning threshold based on the statistical distribution of the redundancy counts of each two-dimensional Gaussian disk; and removing two-dimensional Gaussian disks whose redundancy counts exceed the pruning threshold based on the pruning threshold, thereby obtaining the iteratively optimized Gaussian splatter point cloud corresponding to the two-dimensional Gaussian splatter point cloud; In step 300, the two-dimensional Gaussian splash point cloud is dynamically updated, and Gaussian disks with low contribution are split, cloned, or removed. Then, during the reconstruction process, a dynamic level of detail (LOD) is used to adjust the order of the spherical harmonic function used by the Gaussian disk based on the contribution of each individual Gaussian disk to the reconstruction quality.

[0035] Step 400: Output the Gaussian splatter point cloud after iterative optimization as the 3D reconstruction model corresponding to the image sequence.

[0036] As described above, the monocular camera-based 3D reconstruction method provided in this application can avoid the floating phenomenon during 3D voxel reconstruction by using a two-dimensional elliptical disk, thus improving the reconstruction effect. Adaptive pruning is incorporated, which removes redundant Gaussian disks that contribute little to the rendering effect or appear in large quantities within a certain range by combining multiple indicators during algorithm operation, thereby reducing memory usage. Codebook and data precision quantization are used to further reduce product storage and accelerate product streaming and rendering. The 3D reconstruction algorithm based on Gaussian splashing technology, while introducing adaptive pruning and codebook quantization, can reduce the system hardware requirements for rapid 3D reconstruction, enabling good 3D reconstruction results even on relatively inexpensive low-end devices, thereby reducing the algorithm's memory usage and computational overhead. Furthermore, it can meet the needs of lightweight application scenarios where images acquired by a monocular camera, possibly mounted on a drone, are used for 3D reconstruction.

[0037] To further reduce the memory footprint and computational overhead of the algorithm, in a monocular camera-based 3D reconstruction method provided in this application embodiment, the loss function in the iterative optimization process is a weighted average of the L1 term and the SSIM exponent term; wherein, the L1 term is the L1 difference between the 3D reconstruction model corresponding to the image sequence and the real image in the image sequence; and the SSIM exponent term is the SSIM exponent between the 3D reconstruction model corresponding to the image sequence and the real image in the image sequence.

[0038] In one or more embodiments of this application, during a single iteration optimization process, the learning rate is first updated according to the current iteration number; then, a training camera viewpoint is randomly selected, and an image from that viewpoint is rendered. Subsequently, the error between the rendered image and the real image is calculated as the loss function during the training process. The loss function is a weighted average of L1 and SSIM, where x and y represent the original image in the training set and the scene rendered from that image under the camera pose, respectively. The loss function can be expressed by the following formula: (1) (2) (3) (4) (5) in, Indicates the number of images. Indicates the image sequence number. Represents the original image. i pixel value, Indicates the rendered image number i pixel value, This represents the mean. Indicates the number of pixels. Indicates standard deviation, , It is a constant.

[0039] set up , , Constants are used to avoid instability when the value of a single item is close to 0. , , It is an empirical constant. Indicates the dynamic range of pixel values. It can be 0.01. The value can be 0.03, and L can be 1. Therefore, the actual SSIM calculation formula in this algorithm is: (6) The calculated L1 and SSIM values ​​are weighted and averaged to obtain the actual loss function. This loss function value is then used to control the iteration of the iterator through backpropagation.

[0040] To further reduce the algorithm's memory footprint and computational overhead, in a monocular camera-based 3D reconstruction method provided in this application embodiment, see [link to relevant documentation]. Figure 2 The step of determining a local spatial range for each of the two-dimensional Gaussian disks based on its projection information from the multiple monocular camera viewpoints includes: Step 310: For each of the two-dimensional Gaussian disks, determine the radius of a query sphere based on the number of pixels it occupies in the two-dimensional projection of the multiple monocular camera views; Step 320: Using the two-dimensional Gaussian disk as the center and the radius of the query sphere as the reference, determine the local spatial range.

[0041] In one or more embodiments of this application, the concept of redundancy is introduced. A Gaussian disk is considered redundant if it is surrounded by too many other Gaussian disks, all crammed into a small "pixel projection space". A query sphere is defined centered on a single Gaussian disk, and its radius is determined. The radius is determined by the number of pixels occupied in the two-dimensional projections of the Gaussian disk at the center of the sphere from various perspectives. Calculating how many Gaussian disks are within the sphere or intersect with its surface is called the redundancy count.

[0042] To further reduce the memory footprint and computational overhead of the algorithm, in the monocular camera-based 3D reconstruction method provided in this application embodiment, the dynamic determination of the pruning threshold is specifically calculated using the following formula: τ = mean(C) + λ·std(C) Where τ is the pruning threshold, C is the set of redundant counts of all two-dimensional Gaussian disks, mean(C) is the mean of C, std(C) is the standard deviation of C, and λ is a preset hyperparameter.

[0043] In one or more embodiments of this application, after calculating the redundancy count C of all Gaussian disks, the pruning threshold is dynamically determined. Users can use the hyperparameter λ to adjust the level of the pruning threshold, and the value of λ can be 1.

[0044] To further reduce the memory footprint and computational overhead of the algorithm, in a monocular camera-based 3D reconstruction method provided in this application embodiment, the step of removing two-dimensional Gaussian disks with redundancy counts exceeding the pruning threshold based on the pruning threshold to obtain the iteratively optimized Gaussian splatter point cloud corresponding to the two-dimensional Gaussian splatter point cloud includes: Based on the pruning threshold, from the two-dimensional Gaussian disks whose redundancy count exceeds the pruning threshold, two-dimensional Gaussian disks with transparency less than a preset transparency threshold are selected and removed to obtain the iteratively optimized Gaussian splatter point cloud corresponding to the two-dimensional Gaussian splatter point cloud.

[0045] In one or more embodiments of this application, after the pruning threshold is determined, the transparency of all Gaussian disks with redundancy counts exceeding the pruning threshold is checked, and the half with lower transparency will be pruned. Besides redundancy-based pruning, there are also pruning based on fixed hyperparameter thresholds such as size (too large or too small) and transparency (too small). The pruning process reduces redundant Gaussian disks and Gaussian disks in the point cloud system that contribute little to the rendering effect or even hinder iterations to achieve better rendering results, thereby reducing the number of parameters in the point cloud system and enhancing the 3D reconstruction effect to some extent.

[0046] To further reduce the memory footprint and computational overhead of the algorithm, in a monocular camera-based 3D reconstruction method provided in this application embodiment, the iterative optimization process further includes periodically performing point cloud density control operations, which include: For a two-dimensional Gaussian disk with a gradient value greater than a preset gradient threshold, if the size of the two-dimensional Gaussian disk is smaller than a preset size threshold, then clone an identical two-dimensional Gaussian disk and adjust the positions of the two two-dimensional Gaussian disks to fill the empty part in the space range corresponding to the size threshold. If the size of the two-dimensional Gaussian disk is larger than a preset size threshold, the two-dimensional Gaussian disk is split into two two-dimensional Gaussian disks with a smaller size to fill the empty space within the space corresponding to the size threshold.

[0047] In one or more embodiments of this application, in addition to the loss function calculation and backpropagation process performed in each iteration, the algorithm also periodically controls the local density of the point cloud. The point cloud density control operation includes Gaussian disk cloning and Gaussian disk splitting, both of which can improve the rendering effect. Cloning emphasizes increasing the number of points, while splitting emphasizes reducing the volume. In the code, the same size threshold is used for partitioning when performing cloning or splitting operations, represented in the code as: self.percent_dense `scene_extent`; `percent_dense` is typically set to 0.01. `scene_extent` is the radius of the bounding sphere of all training camera distributions, representing the size information of the training scene obtained from camera view initialization in this point cloud reconstruction task. It is generally normalized and falls between 1 and 10. When the reconstruction gradient of the 2D Gaussian disk is greater than or equal to `grad_threshold` (default value 0.0002), the reconstruction effect is considered unsatisfactory, triggering cloning or splitting. Furthermore, if the size is greater than `percent_dense`... If the scene_extent value is less than or equal to this threshold, splitting is triggered; otherwise, cloning is triggered. The gradient of each Gaussian disk is detected. An excessively large gradient indicates a poor fit. For Gaussian disks with excessively large gradients, two different operations are performed: splitting or cloning. Smaller Gaussian disks indicate underfitting, possibly due to incomplete detail texture. This can be addressed by cloning an identical Gaussian disk and slightly adjusting the positions of the two disks to fill in the missing details. Larger Gaussian disks indicate that the algorithm is attempting to cover complex set details with a single large disk. This can be addressed by splitting the large Gaussian disk into two smaller Gaussian disks, thereby refining the geometric details.

[0048] To further reduce the memory footprint and computational overhead of the algorithm, in a 3D reconstruction method based on a monocular camera provided in this application embodiment, the method further includes: The storage and calculation precision of at least one parameter in the Gaussian splash point cloud used to adjust the geometric properties or spatial position of the Gaussian disk is converted from a first precision format to a second precision format, which is lower than the first precision.

[0049] In one or more embodiments of this application, the first precision format can be float32, and the second precision format can be float16 or int8. Using a lower precision data format to carry some data that is not sensitive to precision reduces the amount of memory occupied by the parameters while having less impact on the reconstruction effect. To further reduce the memory footprint and computational overhead of the algorithm, in a 3D reconstruction method based on a monocular camera provided in this application embodiment, the method further includes: The K-means algorithm is used to cluster a set of parameter vectors in the two-dimensional Gaussian splash point cloud that represent a geometric property or spatial location feature of the Gaussian disk into a preset number of codebook vectors, and the codebook index is used to replace the set of parameter vectors for storage.

[0050] In one or more embodiments of this application, the K-means algorithm is used to cluster a large number of parameters into a small codebook. Assume there are M Gaussian disks, where a certain attribute of each Gaussian disk (such as the color spherical harmonic coefficient SH) is a vector. The goal is to find k codebook vectors and their corresponding index assignments. This minimizes the reconstruction error. The reconstruction error can be expressed as: (7) in, Indicates the number of codebook vectors. Indicates the first i The attribute vector of a Gaussian disk. Indicates an index. Represents the codebook vector.

[0051] To further reduce the memory footprint and computational overhead of the algorithm, in a 3D reconstruction method based on a monocular camera provided in this application embodiment, the method further includes: The order of the spherical harmonic function used to represent the color of the two-dimensional Gaussian disk is dynamically adjusted based on the contribution of a single two-dimensional Gaussian disk to the reconstruction quality.

[0052] In one or more embodiments of this application, the redundancy of Gaussian disks is used for adaptive pruning within a local range of Gaussian point clouds. This reduces local redundancy, balances overall reconstruction quality, and reduces the number of redundant parameters in the point cloud as a whole. Instead of traditional methods that rely solely on fixed threshold pruning based on hyperparameters like size and transparency, this application achieves pruning not only at a macroscopic scale, but also removes Gaussian disks that contribute very little to the rendering effect or even have negative side effects, making it insensitive to redundancy in high-density Gaussian disk regions. Furthermore, it allows for microscopic pruning, increasing the ability to adjust details. In addition, combining this with large-scene block loading and multi-threaded reconstruction further reduces the pressure on system memory and computing power.

[0053] In a specific example of a monocular camera-based 3D reconstruction method, see [link to example]. Figure 3 The method includes: First, real-time image sequences are acquired from a monocular camera or the captured video is processed by frame extraction to obtain image sequences. The obtained image sequences are then preliminarily processed to obtain a usable COLMAP format dataset, and sparse point clouds are generated.

[0054] Secondly, an initial Gaussian point cloud is created using the dataset and sparse point cloud input, and the Gaussian point cloud is dynamically updated. Low-contribution Gaussian disks are split, cloned, or removed.

[0055] Then, during the reconstruction process, a dynamic level of detail (LOD) is used to adjust the order of the spherical harmonic function used by the Gaussian disk based on the contribution of each individual Gaussian disk to the reconstruction quality.

[0056] Finally, render and output a 3D point cloud model to support real-time applications.

[0057] Specifically, after obtaining the sparse point cloud from the sparse reconstruction, the Gaussian point cloud is initialized based on this sparse point cloud. The algorithm continuously optimizes the rendering effect of the Gaussian point cloud during the iteration process, thereby outputting a high-precision reconstruction product after a certain number of iterations. Referring to Tables 1 to 4, the entire iterative calculation process of the algorithm includes the following steps: Algorithm initialization steps: First, initialize the Gaussian point cloud to represent the 3D Gaussian distribution of each point in the scene. Then, initialize the scene object, load the dataset and corresponding camera parameters. Finally, set the optimizer and learning rate scheduler for the Gaussian model parameters.

[0058] Steps in a single iteration: In a single iteration, the learning rate is first updated based on the current iteration number. Then, a training camera viewpoint is randomly selected, and an image from that viewpoint is rendered. The error between the rendered image and the real image is then calculated as the loss function during training. The loss function is a weighted average of L1 and SSIM loss functions. Let x and y represent the original image in the training set and the rendered image of that image in the camera pose, respectively. The formula is as follows: (1) (2) (3) (4) (5) set up , Constants are used to prevent instability when the value of a single item is close to 0. L is taken as 1 here; therefore, the actual SSIM calculation formula is: (6) The calculated L1 and SSIM values ​​are weighted and averaged to obtain the actual loss function. This loss function value is then used to control the iteration of the iterator through backpropagation.

[0059] The regularly performed steps, in addition to the loss function calculation and backpropagation process that occur in each iteration, are detailed in [link to relevant documentation]. Figure 4 The algorithm also periodically controls the local density of the point cloud. This operation includes Gaussian disk cloning and Gaussian disk splitting. Both can improve the rendering effect; cloning emphasizes increasing the number of points, while splitting emphasizes reducing the volume. The gradient of each Gaussian disk is detected (excessive gradients indicate poor fitting). For Gaussian disks with excessively large gradients, two different operations are performed: splitting and cloning. Smaller Gaussian disks indicate underfitting, possibly due to incomplete detail textures. Cloning an identical Gaussian disk and slightly adjusting the positions of the two disks can fill in the missing details. Larger Gaussian disks indicate that the algorithm is attempting to cover complex ensemble details with a large ellipsoid. Splitting the large Gaussian disk into two smaller Gaussian disks refines the geometric details.

[0060] Adaptive pruning steps: Introduce the concept of redundancy. If a Gaussian disk is surrounded by too many other Gaussian disks, and they are all crammed into a very small "pixel projection space," then this Gaussian disk is redundant. See also Figure 5 In the diagram, green ellipses represent the central Gaussian disk, orange ellipses represent Gaussian disks that are inside or intersect the sphere with the central Gaussian disk, and gray ellipses represent Gaussian disks that are outside or do not intersect the sphere with the central Gaussian disk. The specific algorithm is as follows: a query sphere with radius R is defined, centered on a single Gaussian disk. The determination of R depends on the number of pixels occupied in the two-dimensional projection of the central Gaussian disk from various perspectives. The number of Gaussian disks inside or intersecting the sphere is calculated; this is called the redundancy count. After calculating the redundancy count C of all Gaussian disks, the redundancy threshold for pruning is dynamically determined. Here, λmercy is a hyperparameter that users can use to adjust the redundancy threshold. After the threshold is determined, the transparency of all Gaussian disks with redundancy counts exceeding the threshold is checked, and the half with lower transparency will be pruned. In addition to redundancy-based pruning, there are also pruning methods based on size (too large or too small) and transparency (too small) with fixed hyperparameter thresholds. The pruning process reduces redundant Gaussian disks and Gaussian disks that contribute little to the rendering effect or even hinder the iteration of better rendering effects in the point cloud system, thereby reducing the number of parameters in the point cloud system and enhancing the 3D reconstruction effect to some extent.

[0061] Quantization steps: Applicable to the entire iterative process of Gaussian point clouds. It includes two quantization measures: parameter quantization and codebook clustering. Parameter quantization uses a lower-precision data format to carry some precision-insensitive data, thereby reducing the memory occupied by parameters with minimal impact on reconstruction results. Codebook quantization essentially uses the K-means algorithm to cluster a large number of parameters into a small codebook. Assume there are N Gaussian disks, where a certain attribute of each Gaussian disk (such as the color spherical harmonic coefficient SH) is a vector. The goal is to find k codebook vectors. and corresponding index allocation This minimizes the reconstruction error. The reconstruction error can be expressed as: (7) Table 1. Information on Gaussian Point Clouds

[0062] Table 2 Information on Gaussian Point Cloud Methods

[0063] Table 3 Scene Class Information Table Table 4. Information on Scenario-based Methods

[0064] This application also provides an electronic device, which may include a processor, a memory, a receiver, and a transmitter. The processor is used to execute the monocular camera-based 3D reconstruction method mentioned in the above embodiments. The processor and memory can be connected via a bus or other means, taking a bus connection as an example. The receiver can be connected to the processor and memory via wired or wireless means.

[0065] The processor can be a central processing unit (CPU). The processor can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations of the above types of chips.

[0066] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the monocular camera-based 3D reconstruction method described in the embodiments of this application. The processor executes various functional applications and data processing by running the non-transitory software programs, instructions, and modules stored in the memory, thereby implementing the monocular camera-based 3D reconstruction method described in the above method embodiments.

[0067] The memory may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created by the processor, etc. Furthermore, the memory may include high-speed random access memory and non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include memory remotely located relative to the processor, which can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0068] The one or more modules are stored in the memory, and when executed by the processor, the monocular camera-based 3D reconstruction method described in the embodiment is executed.

[0069] In some embodiments of this application, the user equipment may include a processor, a memory, and a transceiver unit. The transceiver unit may include a receiver and a transmitter. The processor, memory, receiver, and transmitter may be connected via a bus system. The memory is used to store computer instructions, and the processor is used to execute the computer instructions stored in the memory to control the transceiver unit to send and receive signals.

[0070] As one implementation method, the functions of the receiver and transmitter in this application can be implemented by transceiver circuits or dedicated transceiver chips, and the processor can be implemented by dedicated processing chips, processing circuits or general-purpose chips.

[0071] As another implementation approach, the server provided in this application embodiment can be implemented using a general-purpose computer. That is, the program code implementing the processor, receiver, and transmitter functions is stored in memory, and the general-purpose processor implements the processor, receiver, and transmitter functions by executing the code in memory.

[0072] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the aforementioned monocular camera-based 3D reconstruction method. The computer-readable storage medium can be a tangible storage medium, such as random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, floppy disks, hard disks, removable storage disks, CD-ROMs, or any other form of storage medium known in the art.

[0073] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the aforementioned monocular camera-based 3D reconstruction method.

[0074] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. The programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave.

[0075] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0076] In this application, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments.

[0077] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to the embodiments of this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A three-dimensional reconstruction method based on a monocular camera, characterized in that, include: The image sequences acquired by the monocular camera are format-standardized to obtain a dataset containing various image sequences in a preset format, and a sparse point cloud is generated based on the dataset. Based on the sparse point cloud, a two-dimensional Gaussian splash point cloud is initialized, wherein each Gaussian element in the two-dimensional Gaussian splash point cloud is represented by a two-dimensional Gaussian disk; The Gaussian point cloud parameters of the two-dimensional Gaussian splatter point cloud are iteratively optimized, and an adaptive pruning operation is performed during the iterative optimization process. The adaptive pruning operation includes: for each two-dimensional Gaussian disk, determining a local spatial range based on its projection information from multiple monocular camera views, and calculating the number of other two-dimensional Gaussian disks located within this local spatial range as the redundancy count of that two-dimensional Gaussian disk; dynamically determining a pruning threshold based on the statistical distribution of the redundancy counts of each two-dimensional Gaussian disk; and removing two-dimensional Gaussian disks whose redundancy counts exceed the pruning threshold based on the pruning threshold, thereby obtaining the iteratively optimized Gaussian splatter point cloud corresponding to the two-dimensional Gaussian splatter point cloud. The Gaussian splatter point cloud, after iterative optimization, is output as the 3D reconstruction model corresponding to the image sequence.

2. The method according to claim 1, characterized in that, The loss function in the iterative optimization process is the weighted average of the L1 term and the SSIM exponent term; wherein, the L1 term is the L1 coefficient between the 3D reconstruction model corresponding to the image sequence and the real image in the image sequence; and the SSIM exponent term is the SSIM exponent between the 3D reconstruction model corresponding to the image sequence and the real image in the image sequence.

3. The method according to claim 2, characterized in that, For each of the two-dimensional Gaussian disks, determining a local spatial range based on its projection information from the multiple monocular camera viewpoints includes: For each of the two-dimensional Gaussian disks, the radius of a query sphere is determined based on the number of pixels it occupies in the two-dimensional projection of the multiple monocular camera views. The local spatial range is determined with the two-dimensional Gaussian disk as the center and the radius of the query sphere as the reference.

4. The method according to claim 3, characterized in that, The dynamic determination of the pruning threshold is specifically calculated using the following formula: τ = mean(C) + λ·std(C) Where τ is the pruning threshold, C is the set of redundant counts of all two-dimensional Gaussian disks, mean(C) is the mean of C, std(C) is the standard deviation of C, and λ is a preset hyperparameter.

5. The method according to claim 2, characterized in that, The step of removing two-dimensional Gaussian disks with redundancy counts exceeding the pruning threshold based on the pruning threshold, to obtain the iteratively optimized Gaussian splatter point cloud corresponding to the two-dimensional Gaussian splatter point cloud, includes: Based on the pruning threshold, from the two-dimensional Gaussian disks whose redundancy count exceeds the pruning threshold, two-dimensional Gaussian disks with transparency less than a preset transparency threshold are selected and removed to obtain the iteratively optimized Gaussian splatter point cloud corresponding to the two-dimensional Gaussian splatter point cloud.

6. The method according to claim 2, characterized in that, The iterative optimization process also includes periodically performing point cloud density control operations, which include: For a two-dimensional Gaussian disk with a gradient value greater than a preset gradient threshold, if the size of the two-dimensional Gaussian disk is smaller than a preset size threshold, then clone an identical two-dimensional Gaussian disk and adjust the positions of the two two-dimensional Gaussian disks to fill the empty part in the space range corresponding to the size threshold. If the size of the two-dimensional Gaussian disk is larger than a preset size threshold, the two-dimensional Gaussian disk is split into two two-dimensional Gaussian disks with a smaller size to fill the empty space within the space corresponding to the size threshold.

7. The method according to claim 2, characterized in that, The method further includes: The storage and calculation precision of at least one parameter in the Gaussian splash point cloud used to adjust the geometric properties or spatial position of the Gaussian disk is converted from a first precision format to a second precision format, which is lower than the first precision.

8. The method according to claim 2, characterized in that, The method further includes: The K-means algorithm is used to cluster a set of parameter vectors in the two-dimensional Gaussian splash point cloud that represent a geometric property or spatial location feature of the Gaussian disk into a preset number of codebook vectors, and the codebook index is used to replace the set of parameter vectors for storage.

9. The method according to claim 8, characterized in that, The method further includes: The order of the spherical harmonic function used to represent the color of the two-dimensional Gaussian disk is dynamically adjusted based on the contribution of a single two-dimensional Gaussian disk to the reconstruction quality.

10. An electronic device, characterized in that, It includes a processor and a memory; when the processor executes the running program stored in the memory, it implements the monocular camera-based three-dimensional reconstruction method as described in any one of claims 1 to 9.

Images