A new perspective synthesis method based on sparse views

Abstract

The present invention proposes a new perspective synthesis method based on sparse views, which relates to the technical field of three-dimensional reconstruction. The input image guides the color allocation of the point cloud, and the optical flow and camera parameters are used to establish the connection between the pole line and the depth in the 3DGS adaptive density control process to accurately calculate the depth and obtain an accurate depth value; the depth information such as the depth value is embedded into the point cloud, and the pole representation capability of 3DGS is used to effectively fuse the depth value using the deep fusion algorithm that resists optical flow errors to achieve accurate geometric modeling; a relatively dense point cloud is formed in the initial stage of geometric modeling, and the point cloud is subjected to deep shearing based on optical flow filtering to reduce the randomness of placement and reduce over-smoothing or over-fitting in sparse view synthesis. The influence caused by the errors in the depth information is avoided to a great extent, so that the results of three-dimensional reconstruction can obtain higher quality and details.

Description

Technical Field

[0001] The invention proposes a new perspective synthesis method based on sparse views, and relates to the technical field of three-dimensional reconstruction. Background Art

[0002] 3D reconstruction technology has high practicality, but dense multi-view image input is often difficult to meet. To address this problem, existing technologies have proposed a variety of solutions, including NeRF-based methods and 3DGS-based methods. Among them, benefiting from the better reconstruction quality of 3DGS, this type of method has better performance. The mainstream solutions are mainly as follows: Figure 1 The flowchart of DNGaussian is shown in the figure.

[0003] Neural Radiance Fields (NeRF) revolutionizes 3D scene reconstruction and synthesis of new perspectives by leveraging neural representations through volume rendering. While NeRF is capable of producing high-quality images, its computational intensity results in slow training and inference times. To address this, multiple techniques have been developed to improve efficiency, as well as methods aimed at enhancing image quality and generalization. In parallel, 3D Gaussian Splattering (3DGS) offers an explicit alternative that utilizes anisotropic 3D Gaussian volumes for fast, high-quality, real-time rendering. 3DGS excels in capturing details in complex scenes and continues to develop as a robust approach in parallel to implicit methods such as NeRF.

[0004] There are two main problems in the existing technology. One is that the introduction of depth information only depends on the back propagation of depth loss. Although existing technologies, such as DNGaussian, have made special improvements to 3DGS so that depth loss can guide the training of 3DGS, this strategy of using back propagation to constrain geometric shapes cannot fully utilize the feature that the geometric shapes of 3DGS can be directly operated. At the same time, the depth information obtained by using the monocular depth estimation network is not completely accurate. Under the constraint of this depth information, more errors can be introduced. The second problem is that the point cloud density control strategy of 3DGS is not suitable for sparse attempt generation tasks, and the existing technologies have hardly improved this point. Although FSGS proposes an additional point addition strategy, this strategy is not stable and has a lot of randomness, resulting in it not playing a positive role in most cases. In 3DGS, if a dense point cloud cannot be obtained, the final reconstruction quality and the fineness of the detailed texture will be greatly reduced, which leads to the fact that the advantages of 3DGS cannot be fully utilized. Summary of the invention

[0005] Based on the above problems, the present invention proposes a new perspective synthesis method based on sparse views. The present invention embeds depth information into the point cloud, and uses the extreme point representation capability of 3DGS to achieve accurate geometric modeling without complex manual regularization. The core of the present invention is the point cloud densification technology. Unlike the traditional method of gradually densifying the point cloud through splitting operations, the present invention aims to form a relatively dense point cloud in the initial stage, reduce placement randomness, and alleviate over-smoothing or over-fitting in sparse view synthesis. This technology includes three key steps: extreme point deep network, deep fusion to resist optical flow errors, and deep pruning based on optical flow filtering.

[0006] The extremal deep network step uses optical flow and camera pose within the extremal geometry framework to accurately and explicitly compute depth. Given that optical flow estimates can be inaccurate, leading to inconsistent depth maps, we propose a deep fusion strategy that is resistant to optical flow errors, selecting the depth with the least sensitivity to flow errors by evaluating the relative rate of change between the reference and projected distances. Depth pruning based on optical flow filtering further refines the depth map by removing points that are too far from the extremal line, thereby mitigating the impact of optical flow inaccuracies.

[0007] Although optical flow predictions may be inaccurate, they are generally more stable than depth maps because their solution space is restricted to the image resolution, while depth maps span the real number domain. By leveraging optical flow for extreme depth computation and combining it with the proposed strategy, a reliable initial dense point cloud coverage is achieved. This foundation supports stable 3DGS training under sparse view conditions, resulting in highly accurate depth maps and improved image quality.

[0008] The specific steps include:

[0009] S1. Use the optical flow estimator to predict the optical flow between the source view and the target view, use the optical flow and camera parameters to establish the connection between the epipolar line and the depth, and calculate the depth value of the pixel in the source view;

[0010] S2, using the anti-optical flow error depth fusion algorithm to fuse the depth value calculated in step S1 to obtain a fused depth value;

[0011] S3, pruning the fused depth values ​​based on optical flow filtering, removing points whose distance to the epipolar line exceeds the threshold to refine the depth map.

[0012] In the preferred technical solution, in step S1: using an optical flow estimator to predict the optical flow between the source view and the target view image :

[0013] ;

[0014] in, represents a pre-trained optical flow estimator, and are the source view image i and the target view image j respectively;

[0015] For the tth pixel in the source view image i , the predicted matching points in the target view image j for:

[0016] .

[0017] In the preferred technical solution, by selecting the polar line in the target view The foot of the To correct , vertical foot The calculation process is as follows:

[0018] ;

[0019] in is the predicted matching point The coordinates in the coordinate system, It's a pixel The coordinates in the coordinate system, It is a vertical foot Coordinates in a coordinate system;

[0020] Among them, a, b, c are epipolar parameters, and epipolar geometry is used to calculate the t-th pixel in the source view image i Depth value :

[0021] ;

[0022] in, represents the cross product, are the external parameters of the source camera corresponding to the source view image i and the target camera corresponding to the target view image j, and and are the intrinsic parameters of the source camera corresponding to source view image i and the target camera corresponding to target view image j.

[0023] In the preferred technical solution, in step S2: let the potential real position in space be P, is a three-dimensional point in space, , is the point where the camera is located, , and is the intersection of the line and the imaging plane, Representative Points To Camera The length of e represents The length of the mapping on the imaging plane, and Represents the change between two values;

[0024] Deep fusion strategy to resist optical flow error uses gradient Sort the depth values, gradient for:

[0025] ;

[0026] Among them, t is the number of cameras , The distance between the cameras, m is Intersection The distance They are angle.

[0027] In the preferred technical solution, the error along the epipolar line Converted to depth error :

[0028] ;

[0029] For a point in the source view , by selecting from the other N-1 views the view with the minimum The depth value is used to initialize the 3D point.

[0030] In the preferred technical solution, the formula is used Represents pixel The gradient corresponding to the target view image j ;use Represents a point on the source view image i The value of e calculated on the target view image j;

[0031] The final fused depth value is Determined by the following formula:

[0032] .

[0033] Among them, k represents The value of j with the smallest value.

[0034] In the preferred technical solution, in step S3: the point To the pole Distance :

[0035] ;

[0036] Setting the Threshold To filter out points with larger distances;

[0037] The color of each pixel in the input source view image is used as the initial color of the Gaussian kernel, and the calculated depth value is used to project the pixel into three-dimensional space as the initial position of each Gaussian kernel. :

[0038] ;

[0039] in, Represents the projection function, which transforms a two-dimensional point into a three-dimensional space.

[0040] Compared with the existing technology, the present invention has the following beneficial technical effects:

[0041] The present invention enhances sparse view synthesis by embedding extreme depth prior into 3DGS, avoiding the errors of monocular depth estimation and the need for complex implicit regularization.

[0042] Combining the characteristics of 3DGS, the problems existing in the existing technology are studied, and a brand-new way is designed to introduce external information. The advantages of 3DGS as an explicit representation are fully utilized. For the first time, depth information is calculated through optical flow and an initial point cloud is constructed.

[0043] We designed a stable technique to create a dense accurate point cloud from optical flow, estimate depth values ​​through epipolar lines, and determine the accuracy of depth values ​​based on the distance from the corresponding point to the epipolar line. We also protected the method of calculating depth through optical flow and the strategy of filtering out more accurate points.

[0044] After acquiring multiple depth values ​​from different viewing angles, the present invention designs a strategy for combining the depths of different views to find the most likely true depth value in order to obtain a more accurate depth value. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0046] Figure 1 This is the flowchart of DNGaussian.

[0047] Figure 2 It is a flow chart of the present invention.

[0048] Figure 3 It is the 3DGS flow chart of the present invention.

[0049] Figure 4 Schematic diagram of small block depth regularization of the present invention.

[0050] Figure 5 It is a schematic diagram comparing the result diagram of the present invention with the existing solution. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0052] In the drawings of the specific embodiments of the present invention, in order to better and more clearly describe the working principles of the various components in the system, the connection relationship of the various parts in the device is shown, which only clearly distinguishes the relative position relationship between the various components, and cannot constitute a limitation on the signal transmission direction, connection sequence and size, dimensions and shape of the components or structures.

[0053] The process of the new perspective synthesis method based on sparse views of the present invention is as follows: Figure 1 As shown, the following steps are included:

[0054] S1. Use the optical flow estimator to predict the optical flow between the source view and the target view, use the optical flow and camera parameters to establish the connection between the epipolar line and the depth, and calculate the depth value of the pixel in the source view.

[0055] In 3DGS, these problems are difficult to mitigate due to the randomness introduced by the segmentation operation during adaptive density control, coupled with insufficient supervision in the sparse view new view synthesis task. To address this problem, the present invention proposes to use dense point clouds directly for initialization. Although the input image can guide the color assignment of the point cloud, accurate depth values ​​are crucial to determine the location of each point - these values ​​are usually not reliably provided by monocular depth estimation networks.

[0056] Therefore, this step uses optical flow and camera parameters to establish the connection between epipolar lines and depth to accurately calculate the depth values ​​of pixels in the source view.

[0057] Use an optical flow estimator to predict the optical flow between the source view and the target view images :

[0058] ;

[0059] in, represents a pre-trained optical flow estimator, and are the source view image i and the target view image j respectively.

[0060] For the tth pixel in the source view image i , the predicted matching points in the target view image j for:

[0061] ;

[0062] Ideally, the true matching point The epipolar lines that should be located in the target view image j However, the optical flow estimator predicts This condition is often not met due to estimation errors.

[0063] These errors introduce additional randomness, making it very difficult to perform effective optimization under sparse input cues. To solve this problem, the present invention selects the epipolar lines in the target view. The foot of the (Vertical foot It is the distance nearest point) to correct , as a relaxation treatment. The calculation process is as follows:

[0064] ;

[0065] in is the predicted matching point The coordinates in the coordinate system, It's a pixel The coordinates in the coordinate system, It is a vertical foot Coordinates in the coordinate system.

[0066] Among them, a, b, c are epipolar parameters, and epipolar geometry is used to calculate the t-th pixel in the source view image i Depth value :

[0067] ;

[0068] here, represents the cross product, are the external parameters of the source camera corresponding to the source view image i and the target camera corresponding to the target view image j, and and are the intrinsic parameters of the source camera corresponding to source view image i and the target camera corresponding to target view image j.

[0069] It needs to be explained that epipolar line is an important concept in the field of stereo vision in computer vision. It describes the correspondence between points in one camera and the image of another camera in a binocular vision system. Specifically, epipolar line refers to the set of possible positions of the image of a point in a pair of stereo images in another image, and this set is a straight line.

[0070] S2. Use the anti-optical flow error depth fusion algorithm to fuse the depth value calculated in step S1 to obtain a fused depth value.

[0071] The depth value information of the pixel point, as a bridge between the 2D and 3D domains, is used through the depth map generated by the monocular depth estimator. These depth maps are incorporated into the depth regularization to correct the inaccuracy and scale distortion caused by the mismatch of camera poses.

[0072] Suppose the set of N input source view images is , for the source view image The tth pixel in , a matching point will be obtained in each of the N-1 target view images j except i , use the equation in step S1 and N-1 matching points to calculate the corresponding depth value.

[0073] Despite the dangling It is to solve The suboptimal form designed with a high probability of not being located on the extreme line is different from the potential real matching point There may still be distance errors between them. Therefore, it is necessary to effectively fuse these depth values ​​to minimize the errors.

[0074] The most straightforward approach in the prior art is to average the depth values, but this approach is susceptible to erroneous values ​​and is often significantly biased compared to the underlying true position in 3D space. While choosing the minimum depth value is often more efficient, determining which value corresponds to the true position with the least error is challenging.

[0075] In order to solve this problem, the present invention adopts a deep fusion algorithm that is resistant to optical flow errors, such as Figure 3 The potential true position in the space shown is P, is a three-dimensional point in space, , is the point where the camera is located, , and is the intersection of the line and the imaging plane, Representative Points To Camera The length of e represents The length of the mapping on the imaging plane, and Represents the difference between two values.

[0076] The deep fusion algorithm that is resistant to optical flow errors uses gradient Sort the depth values. Changes in e will affect ,Will Defined as a function of e.

[0077] The present invention uses geometric relationships to derive The expression of :

[0078] ;

[0079] Where t is Figure 3 Two cameras shown , The distance between the cameras, m is Intersection The distance They are angle.

[0080] According to the above formula, the present invention applies the fundamental theorem of calculus to convert the error along the polar line into Converted to depth error Since the goal is to To sort rather than obtain its exact value, it can be expressed in an approximate form as follows:

[0081] ;

[0082] For the tth pixel in the source view The present invention aims to select the view with the minimum The depth value is used to initialize the 3D point .

[0083] The present invention assumes that when using the same optical flow predictor, the effective prediction points have similar Based on this, the present invention can eliminate The influence of , directly compare the gradient value To approximate The value of e depends on the source view image i and the target view image j as well as the pixel points in the source view image i. .

[0084] Therefore, the present invention uses the formula Represents pixel Corresponding target view image Gradient , It means the gradient value corresponding to e when the source view image is i and the target view image is j.

[0085] It should be noted that the value of e is related to the label of the source view image and the label of the target view image, and and The e is different, so use Represents a point on the source view image i The value of e computed on the target view image j.

[0086] The final fused depth value is Determined by the following formula:

[0087] ;

[0088] k means The purpose of the argmin function is to calculate the minimum value of j, which is the definition of k.

[0089] By fusing the depth values, the inconsistency of the depth map caused by inaccurate optical flow estimation can be effectively avoided.

[0090] S3, pruning the fused depth values ​​based on optical flow filtering, removing points whose distance to the epipolar line exceeds the threshold to refine the depth map.

[0091] Since optical flow estimation may fail due to similar colors or pixels deviating from the target view, the assumption that the projection distance errors are similar may not always hold. In this case, these pixels need to be removed. To this end, the present invention calculates the point To the pole Distance :

[0092] ;

[0093] The greater the distance, the lower the confidence in the optical flow estimate of the point. Therefore, the present invention sets a threshold This not only improves the reliability of the hypothesis, but also enhances the quality of the point cloud and reduces the impact of erroneous points on reconstruction.

[0094] Finally, the color of each pixel in the input source view image is used as the initial color of the Gaussian kernel, and the calculated depth value is used to project the pixel into three-dimensional space as the initial position of each Gaussian kernel. :

[0095] ;

[0096] in, Represents a projection function that can transform a 2D point into 3D space using a depth value.

[0097] By performing a depth clipping method based on optical flow filtering on the point cloud, points that are too far from the pole line are removed to further refine the depth map, thereby alleviating the impact of optical flow inaccuracy, reducing point aggregation, and forming a reliable and dense point cloud, resulting in a highly accurate depth map and improved image quality.

[0098] The present invention uses optical flow to initialize the point cloud, avoiding the influence of depth map and scale on reconstruction quality, and increases the number of points and improves the position accuracy of points through splitting operation during adaptive density control. By performing deep shearing based on optical flow filtering on the point cloud, rich supervision effectively reduces the randomness of point placement in dense view new perspective synthesis and alleviates over-smoothing or over-fitting in sparse view synthesis.

[0099] The results are as follows Figure 5 As shown, compared with the prior art, the results of the present invention on color images are better than other solutions and are most similar to the original images, while other solutions in the prior art will have more or less distortion; in terms of depth reconstruction, the present invention is also more complete, while other solutions will lack in geometric shapes.

[0100] In an alternative embodiment, an optical flow estimator is used to predict the optical flow and calculate the depth, wherein the optical flow estimator may be provided by any method and other methods that can obtain matching points between two views may be used instead.

[0101] In an alternative embodiment, the three-dimensional scene is created based on the 3DGS as the final three-dimensional representation, and any point cloud-based three-dimensional explicit representation may be used instead.

[0102] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

[0103] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A new perspective synthesis method based on sparse views, characterized in that: The steps include: S1. Use the optical flow estimator to predict the optical flow between the source view and the target view, use the optical flow and camera parameters to establish the connection between the epipolar line and the depth, and calculate the depth value of the pixel in the source view; S2, using the anti-optical flow error depth fusion algorithm to fuse the depth value calculated in step S1 to obtain a fused depth value; Let the potential true position in space be P, is a three-dimensional point in space, , is the point where the camera is located, , and is the intersection of the line and the imaging plane, Representative Points To Camera The length of e represents The length of the mapping on the imaging plane, and Represents the change between two values; Deep fusion strategy to resist optical flow error uses gradient Sort the depth values, gradient for: ； Where t is the number of cameras , The distance between the cameras, m is Intersection The distance , , They are Angle The error along the epipolar line Converted to depth error : ； For a point in the source view , by selecting from the other N-1 views the view with the minimum The depth value is used to initialize the 3D point ; Using the formula Represents pixel Corresponding target view image Gradient ;use Represents a point on the source view image i The value of e calculated on the target view image j; the final fused depth value is Determined by the following formula: ； Among them, k represents The value of j with the smallest value; S3. Prune the fused depth values ​​based on optical flow filtering and remove points whose distance to the epipolar line exceeds the threshold to refine the depth map.

2. The new perspective synthesis method according to claim 1, characterized in that: In step S1: using an optical flow estimator to predict the optical flow between the source view and the target view image : ； in, represents a pre-trained optical flow estimator, and are the source view image i and the target view image j respectively; For the tth pixel in the source view image i , the predicted matching points in the target view image j for: 。 3. The new perspective synthesis method according to claim 2, characterized in that: By selecting the polar line in the target view The foot of the To correct , vertical foot The calculation process is as follows: ； in is the predicted matching point The coordinates in the coordinate system, It's a pixel The coordinates in the coordinate system, It is a vertical foot Coordinates in a coordinate system; Among them, a, b, c are epipolar parameters, and epipolar geometry is used to calculate the t-th pixel in the source view image i Depth value : ； in, represents the cross product, are the external parameters of the source camera corresponding to source view image i and the target camera corresponding to target view image j, and and are the intrinsic parameters of the source camera corresponding to source view image i and the target camera corresponding to target view image j.

4. The new perspective synthesis method according to claim 3, characterized in that: In step S3: the point To the pole Distance : ； Setting the Threshold To filter out points with larger distances; The color of each pixel in the input source view image is used as the initial color of the Gaussian kernel, and the calculated depth value is used to project the pixel into three-dimensional space as the initial position of each Gaussian kernel. : ； in, Represents the projection function, which transforms a two-dimensional point into a three-dimensional space.

Images