Point cloud fusion 3D reconstruction method and system based on multivariate confidence filtering

Abstract

The present application relates to a point cloud fusion 3D reconstruction method and system based on multivariate confidence filtering, wherein the point cloud fusion 3D reconstruction method based on multivariate confidence filtering includes: acquiring an image, determining the multivariate confidence of the depth observation to be fused based on the RGB information, posture information and corresponding depth map of each frame image, and merging the multivariate confidence to obtain a merged confidence; for each observation, performing fusion according to its corresponding merged confidence; and determining the reconstruction point based on the fusion result, and performing 3D reconstruction based on the reconstruction point; through the present application, the problem of low reconstruction accuracy for 3D reconstruction of AR large-scene navigation in the related art is solved, and the accuracy of AR large-scene 3D reconstruction is improved.

Description

Technical Field

[0001] The present application relates to the field of three-dimensional reconstruction technology, and in particular to a point cloud fusion three-dimensional reconstruction method and system based on multivariate confidence filtering. Background Art

[0002] Dense scene reconstruction has long been a core issue in 3D vision, playing a crucial role in applications such as augmented reality (AR). Achieving a realistic, immersive virtual-reality fusion experience in AR requires correctly handling occlusion between the real scene and virtual AR objects, as well as rendering effects like shadows. This allows for the proper placement of virtual content and its interaction with the real scene. In short, achieving these effects requires real-time, accurate 3D reconstruction of the scene.

[0003] Currently, commonly used 3D reconstruction solutions, such as KinectFusion and BundleFusion, rely heavily on depth measurements provided by depth sensors. However, due to their high cost and power consumption, depth sensors remain relatively unpopular, typically only being included in a few high-end mobile devices. Therefore, using monocular multi-view images to achieve real-time 3D reconstruction holds great potential. Without adding additional sensors, it can be directly integrated into existing smart devices.

[0004] Multi-View-Stereo (MVS) is a fundamental task in the computer field. Its goal is to deduce the three-dimensional information of objects in the real environment through images taken by the camera and camera parameters. The basic principle is that there will be some common observation parts in images taken from different angles. Reasonable analysis and utilization of the 2D associations of different images are the basis of three-dimensional reconstruction. The 3D coordinates of spatial objects can be recovered from 2D-2D triangulation. The mainstream solution is to first restore the depth map of each frame based on a series of images and corresponding poses, and then obtain a dense point cloud three-dimensional model through a point cloud fusion algorithm for all depth maps. Among them, depth map fusion includes voxel fusion (represented by TSDF (truncated signed distance function)) and point cloud fusion.

[0005] TSDF is a surface reconstruction algorithm that uses structured point cloud data and expresses the surface with parameters. The core is to map the point cloud data into a predefined three-dimensional space and use the truncated signed distance function to represent the area near the surface of the real scene. There is an implicit function F in each voxel, which is continuously fitted through tsdf and weights. When all voxels near all point clouds are fitted, all voxel points with F equal to 0 are found. These point clouds can represent the surface point clouds of the scene model. However, for three-dimensional reconstruction applied to large-scale AR experiences, the scene contains a large number of irregular and complex objects (such as winding roads, vegetation cover, art sculptures, etc.). The voxel volume that needs to be constructed is very large and discrete. In addition, since the depth of field of outdoor scenes often varies greatly, it is difficult to use reasonable truncation parameters to process the depth value of each pixel.

[0006] The open source algorithm libraries colmap and openMVS both use point cloud fusion technology, the core of which is to average the 3D points obtained by back-projecting all depth maps that meet the reprojection error. Figure 1 This is a schematic diagram of point cloud fusion based on the classic open source algorithm library of related technologies, such as Figure 1 As shown in Figure 2, by observing the reconstruction error along the ray, we can see that this method often leads to large reconstruction errors in the observation direction. For general scenes, it can achieve good restoration. However, when using depth maps with a large depth range, high noise levels, and dynamic objects, the results are often unsatisfactory.

[0007] Table 1

[0008]

[0009] Table 1 lists the industry's mainstream commercial 3D reconstruction software. As can be seen, these software packages are primarily designed for aerial photography and object-level / indoor reconstruction. They all offer a reasonable depth range for reconstruction, planned camera acquisition trajectories, and controllable image frame rates. However, 3D reconstruction for large-scale AR scene navigation (hundreds of meters, kilometers, or even larger) often requires images with a large depth of field and uneven local acquisition, which poses significant challenges to reconstruction accuracy.

[0010] In terms of reconstruction accuracy, general deep fusion algorithms have more discrete points in the image for visual depth restoration, and the fused model has greater uncertainty. The fused model often contains flying points, and thus cannot be directly applied to grid reconstruction. The existing probabilistic fusion method REMODE takes into account the difference in depth uncertainty caused by the error of matching 1 pixel, but this method is aimed at fusing multiple frames of depth maps into a high-confidence depth map and cannot directly reconstruct dense point clouds. Methods based on deep learning implicit spatial expression have good reconstruction completeness and accuracy. However, current academic frontier work, such as NeuralRecon, is mainly developed for indoor scenes. It performs poorly for outdoor scenes with a large depth and is extremely dependent on the computing power and video memory size of the GPU.

[0011] Regarding the problem of low reconstruction accuracy of three-dimensional reconstruction of AR large-scene navigation in related technologies, no effective solution has been proposed yet. Summary of the Invention

[0012] The embodiments of the present application provide a point cloud fusion three-dimensional reconstruction method and system based on multivariate confidence filtering, so as to at least solve the problem of low reconstruction accuracy in the three-dimensional reconstruction of AR large-scene navigation in the related art.

[0013] In a first aspect, an embodiment of the present application provides a point cloud fusion 3D reconstruction method based on multivariate confidence filtering, the method comprising:

[0014] Acquire an image, determine the multivariate confidence of the depth observation to be fused based on the RGB information, pose information and corresponding depth map of each frame image, and merge the multivariate confidence to obtain a merged confidence;

[0015] For each observation, fusion is performed according to its corresponding combined confidence; a reconstruction point is determined according to the fusion result; and a three-dimensional reconstruction is performed based on the reconstruction point.

[0016] In some embodiments, the process of performing fusion and determining the reconstruction point based on the fusion result includes: fusing the observations on each group of common view frames to the reference frame through Bayesian filtering, and determining the reconstruction point based on the convergence of the point cloud distribution after fusion.

[0017] In some embodiments, the process of determining the merge confidence includes:

[0018] Determine the geometric confidence of the depth observation to be fused based on the pose information of the image reference frame and the common view frame; determine the texture matching confidence of the depth observation to be fused based on the image RGB information and the pose information of the reference frame and the common view frame; determine the semantic confidence of the depth observation to be fused based on the semantic information calculated based on the image RGB information;

[0019] The geometric confidence, texture matching confidence and semantic confidence are combined to obtain a combined confidence.

[0020] In some embodiments, before determining the confidence of the multivariate depth observations to be fused, the method includes:

[0021] Traversing pixels of the reference frame depth map to obtain a first pixel, and determining the coordinates of a target point of the first pixel in a reference frame coordinate system by back-projecting the depth value of the first pixel to obtain a first coordinate;

[0022] The coordinates of the target point in the world coordinate system are determined by posture transformation to obtain a second coordinate; one of the unoperated common view frames of the reference frame is selected, and the observation value is determined according to the second coordinate.

[0023] In some embodiments, the process of determining the observed value includes:

[0024] According to the second coordinate, determining the pixel position of the target point on the common view frame through posture transformation and projection equation to obtain a second pixel;

[0025] Determine the coordinates of the target point of the second pixel in the world coordinate system by back-projecting the depth value of the second pixel to obtain a third coordinate;

[0026] According to the first coordinate, the third coordinate and the optical center of the reference frame, the coordinates of the projection of the position corresponding to the third coordinate on the observation vector of the reference frame are determined, and the result is used as the observation value.

[0027] In some embodiments, the process of determining the confidence of the multivariate depth observations to be fused includes:

[0028] Determine the matching confidence of the depth observation texture to be fused and express it as a matching standard deviation;

[0029] Determine a transformation matrix of the relative posture according to the posture information of the reference frame and the common view frame, and determine the distance between the optical center of the reference frame and the optical center of the common view frame according to the transformation matrix to obtain the optical center distance;

[0030] Determine a geometric confidence level based on the square of the observation value, the focal length of the reference frame camera intrinsic parameter, the optical center distance, and the matching standard deviation, and express it as a geometric standard deviation;

[0031] The semantic category label of each pixel to which an observation belongs is queried to determine the semantic confidence of the observation, where the standard deviation corresponding to the observation of dynamic objects is infinite.

[0032] In some embodiments, the process of determining the reconstruction point based on the convergence of the fused point cloud distribution includes:

[0033] For Gaussian distribution, determine whether the standard deviation after fusion is less than the preset threshold. If so, it is considered to be converged. If converged, determine the reconstruction point in the world coordinate system based on the average value of the Gaussian distribution and mark the merge position of the corresponding common view frame;

[0034] If convergence has not occurred, one of the unoperated common view frames of the reference frame is continuously selected to determine the observation value and perform fusion until all the common view frames under the reference frame have been operated.

[0035] In a second aspect, an embodiment of the present application provides a point cloud fusion 3D reconstruction system based on multivariate confidence filtering, the system comprising:

[0036] A determination module is used to obtain an image, determine the multivariate confidence of the depth observation to be fused based on the RGB information, pose information and corresponding depth map of each frame image, and merge the multivariate confidence to obtain a merged confidence;

[0037] The fusion module is used to perform fusion for each observation according to its corresponding combined confidence; determine the reconstruction point according to the fusion result; and perform three-dimensional reconstruction based on the reconstruction point.

[0038] In a third aspect, an embodiment of the present application provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the point cloud fusion three-dimensional reconstruction method based on multivariate confidence filtering.

[0039] In a fourth aspect, an embodiment of the present application provides a storage medium storing a computer program, wherein the computer program is configured to execute the point cloud fusion three-dimensional reconstruction method based on multivariate confidence filtering when running.

[0040] Compared with the problem of low reconstruction accuracy for three-dimensional reconstruction of AR large-scene navigation and guidance in related technologies, the embodiment of the present application obtains images, determines the multivariate confidence of the depth observation to be fused based on the RGB information, posture information and corresponding depth map of each frame image, and merges the multivariate confidence to obtain a merged confidence; for each observation, fusion is performed according to its corresponding merged confidence; and the reconstruction point is determined according to the fusion result, and three-dimensional reconstruction is performed based on the reconstruction point; since the present application takes the influence of confidence into consideration, the reconstruction accuracy is better guaranteed, which solves the problem of low reconstruction accuracy for three-dimensional reconstruction of AR large-scene navigation and guidance in related technologies and improves the accuracy of three-dimensional reconstruction of AR large scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0042] Figure 1 This is a schematic diagram of point cloud fusion based on the classic open source algorithm library of related technologies;

[0043] Figure 2 2 is a schematic diagram of an application environment of a point cloud fusion 3D reconstruction method based on multivariate confidence filtering according to an embodiment of the present application;

[0044] Figure 3 is a schematic diagram of a point cloud fusion 3D reconstruction method based on multivariate confidence filtering according to the first embodiment of the present application;

[0045] Figure 4 is a schematic diagram of the preceding steps of determining the confidence of the depth observation to be fused according to the second embodiment of the present application;

[0046] Figure 5 is a geometric diagram of a method for parameterizing 3D point fusion into 1D observation values ​​on a ray according to the second embodiment of the present application;

[0047] Figure 6 is a schematic diagram of a process for determining a merge confidence level according to a third embodiment of the present application;

[0048] Figure 7 is a schematic diagram of the transmission process of the point cloud fusion 3D reconstruction model according to the third embodiment of the present application;

[0049] Figure 8 is a schematic diagram of a process for determining reconstruction points based on the convergence of the distribution of the fused point cloud according to the fourth embodiment of the present application;

[0050] Figure 9 This is a schematic diagram of the colmap AR large scene 3D reconstruction effect based on related technologies;

[0051] Figure 10 2 is a schematic diagram of an AR large-scene 3D reconstruction effect according to an embodiment of the present application;

[0052] Figure 11 Schematic diagram of the internal structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solutions and advantages of this application more clearly understood, the present application is described and illustrated below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely used to explain this application and are not intended to limit this application. Based on the embodiments provided in this application, all other embodiments obtained by those of ordinary skill in the art without making any creative efforts are within the scope of protection of this application.

[0054] Obviously, the drawings described below are merely examples or embodiments of the present application. Those skilled in the art can, without inventive effort, apply the present application to other similar scenarios based on these drawings. Furthermore, it is also understood that, although the effort involved in such a development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, changes in design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as an insufficiency of the content disclosed in this application.

[0055] References to "embodiments" in this application mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it refer to independent or alternative embodiments that are mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described in this application may be combined with other embodiments unless there is a conflict.

[0056] Unless otherwise defined, the technical or scientific terms used in this application should have the ordinary meaning understood by a person of ordinary skill in the technical field to which this application belongs. The words "one", "a", "the" and the like used in this application do not indicate a limit on quantity and may indicate the singular or plural. The terms "include", "comprise", "have" and any variations thereof used in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units that are inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The word "multiple" used in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships can exist. For example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific order for the objects.

[0057] The point cloud fusion 3D reconstruction method based on multivariate confidence filtering provided in this application can be applied to Figure 2 In the application environment shown, Figure 2 is a schematic diagram of the application environment of the point cloud fusion 3D reconstruction method based on multivariate confidence filtering according to an embodiment of the present application, such as Figure 2 As shown, terminal 202 communicates with server 204 via a network. Server 204 acquires images through terminal 202. Server 204 determines the multivariate confidence of the depth observation to be fused based on the RGB information, pose information, and corresponding depth map of each frame, and merges the multivariate confidences to obtain a merged confidence. For each observation, server 204 performs fusion based on its corresponding merged confidence. Based on the fusion result, server 204 determines a reconstruction point and performs 3D reconstruction based on the reconstruction point. Terminal 202 may be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. Server 204 may be implemented as a standalone server or a server cluster consisting of multiple servers.

[0058] This application provides a point cloud fusion 3D reconstruction method based on multivariate confidence filtering, which can perform high-precision point cloud fusion of large AR scenes. Figure 3 is a schematic diagram of a point cloud fusion 3D reconstruction method based on multivariate confidence filtering according to the first embodiment of the present application, such as Figure 3 As shown, the process includes the following steps:

[0059] Step S301: Acquire an image, determine the multivariate confidence of the depth observation to be fused based on the RGB information, pose information and corresponding depth map of each frame image, and merge the multivariate confidence to obtain a merged confidence;

[0060] For example, the input is the pose information of each frame image and its corresponding depth map, and the output is the scene point cloud reconstruction. In the intermediate process, the following three confidence levels are solved: first, the geometric confidence level of a depth observation to be fused is solved using the pose information of the reference frame and the common view frame. Second, the texture matching confidence level of a depth observation to be fused is solved using the RGB image information and the pose information of the reference frame and the common view frame. Third, the semantic information calculated using the RGB image information is used to obtain the semantic confidence level. For each observation to be fused, this algorithm merges the above confidence levels to obtain a merged confidence level, so that each observation contains a depth value and a confidence value (or uncertainty value).

[0061] Step S302: For each observation, perform fusion according to its corresponding merging confidence; determine a reconstruction point based on the fusion result; and perform 3D reconstruction based on the reconstruction point;

[0062] Optionally, the observations on each group of common view frames can be fused to the reference frame through Bayesian filtering, and the reconstruction points can be determined based on the convergence of the point cloud distribution after fusion. The final fusion result allows each fused 3D point cloud to be represented by a probability distribution, and the convergence of the distribution can be used to determine which are high-quality reconstructed points and which are discrete outliers, thus ensuring the reconstruction accuracy.

[0063] Through the above steps S301 to S302, compared with the problem of low reconstruction accuracy for three-dimensional reconstruction of AR large-scene navigation and guidance in related technologies, the embodiment of the present application obtains images, determines the multivariate confidence of the depth observation to be fused according to the RGB information, posture information and corresponding depth map of each frame image, and merges the multivariate confidence to obtain a merged confidence; for each observation, fusion is performed according to its corresponding merged confidence; and the reconstruction point is determined according to the fusion result, and three-dimensional reconstruction is performed based on the reconstruction point; since the present application takes the influence of confidence into consideration, the reconstruction accuracy is better guaranteed, which solves the problem of low reconstruction accuracy for three-dimensional reconstruction of AR large-scene navigation and guidance in related technologies and improves the accuracy of three-dimensional reconstruction of AR large scenes.

[0064] In some embodiments, before determining the multivariate confidence of the depth observations to be fused, the method further includes selecting a fusion prior, selecting a common view frame, and determining the observation value of the point to be fused. Figure 4is a schematic diagram of the preceding steps of determining the confidence of the depth observation to be fused according to the second embodiment of the present application, such as Figure 4 As shown, the process includes the following steps:

[0065] Step S401, traversing the pixels of the reference frame depth map to obtain a first pixel, and determining the coordinates of the target point of the first pixel in the reference frame coordinate system by back-projecting the depth value of the first pixel to obtain a first coordinate;

[0066] For example, traverse the pixel (u, v) of the reference frame ref depth map, determine the depth value depth corresponding to the pixel (u, v), and back-project the depth value to the 3D point X in the reference frame coordinate system according to formula 1. ref ;

[0067]

[0068] Step S402, determining the coordinates of the target point in the world coordinate system through posture transformation to obtain a second coordinate;

[0069] For example, according to Formula 2, the camera pose T w-ref Transform to the world coordinate system to get 3D point X w ;

[0070] X w =T w-ref *X ref =T w-ref *π -1 (u, v, depth) Formula 2

[0071] Step S403: Select one of the unoperated common view frames of the reference frame, and determine the pixel position of the target point on the common view frame according to the second coordinate through posture transformation and projection equation to obtain the second pixel;

[0072] For example, an unoperated common view frame src of the reference frame ref is selected, and the following steps S303 to S305 are performed; the 3D point X w , through the pose T src-w Transform to the coordinate system of the common view frame src, and according to the

[0073] Formula 3, the pixel position (u', v') on the common view frame is obtained through the projection equation;

[0074]

[0075] Step S404, determining the coordinates of the target point of the second pixel in the world coordinate system by back-projecting the depth value of the second pixel to obtain a third coordinate;

[0076] For example, according to Formula 4, the depth value d of the pixel in the common viewing frame is obtained by the pixel position (u', v'): src ;

[0077] d src =depth Map src (u′, v′) Formula 4

[0078] According to formula 5, d src Calculate the depth value and back-project it into the world coordinate system to get the 3D point X src ;

[0079] X src =T w-src *π -1 (u′,v′,d src ) Formula 5

[0080] Step S405, determining the coordinates of the projection of the position corresponding to the third coordinate on the observation vector of the reference frame according to the first coordinate, the third coordinate, and the optical center of the reference frame, and using the result as the observation value;

[0081] For example, the observation vector on the current reference frame can be represented by (X ref -C ref ) indicates that C ref Represents the optical center of the reference frame; the observation vector on the current common view frame can be expressed by (X src -C ref ) represents; according to formula 6, solve the angle between the two vectors;

[0082]

[0083] Finally, according to Formula 7, the 3D position calculated on the common view frame is projected onto the observation vector (X ref -C ref ), and use the result as the posterior observation value d obs ;

[0084] d obs =|X src -C ref |*cos(θ) Formula 7

[0085] In some embodiments, Figure 5 FIG. 1 is a geometric diagram of a method for parameterizing 3D point fusion into 1D observation values ​​on a ray according to the second embodiment of the present application, as shown in FIG. Figure 5As shown, the embodiment of the present application models the 3D point positions to be fused as a distribution on the observation vector of the reference frame. This transforms a three-dimensional distribution problem into a one-dimensional distribution problem, greatly simplifying the computational complexity and significantly improving computational efficiency. Furthermore, the distribution can be expressed using a Gaussian distribution, so that the distribution can be represented by two parameters: mean and variance, as shown in Formula 8.

[0086] N(x|μ k , σ 2 ) Formula 8 By choosing a reasonable error distribution assumption, such as Gaussian distribution, the calculation can be greatly accelerated; of course, other distributions can also be used to replace the Gaussian distribution model mentioned above, such as mixed Gaussian model, Beta-Gaussian mixed model, and Uniform-Gaussian mixed model.

[0087] Specifically, Figure 6 FIG. 1 is a schematic diagram of a process for determining a merge confidence level according to a third embodiment of the present application. Figure 6 As shown, the process includes the following steps:

[0088] Step S601, determining the texture matching confidence of the depth observation to be fused based on the image RGB information and the pose information of the reference frame and the common view frame;

[0089] Texture matching confidence can be measured in different ways, even using deep learning methods, as long as the local texture of the pixel can be characterized to provide a richer and more qualitative match. For example, for texture matching confidence, the matching standard deviation e is used. d To express, the following solution e d :

[0090] A pair of matching points x, x' on the reference frame and the common view frame conforms to the epipolar geometry constraint and can be represented by the fundamental matrix F according to Formula 9;

[0091] x*F*x′=0 Formula 9

[0092] According to formula 10, the epipolar line 1 on the reference frame is solved by the basic matrix;

[0093] l=F T x′ Formula 10

[0094] The epipolar line can be decomposed into its direction. The matching point on the reference frame must be on the epipolar line. This is the epipolar geometry constraint. If the gradient direction of the texture of the matching point on the epipolar line is consistent with the epipolar line, a good matching accuracy can be found. However, if the two directions are perpendicular, the matching accuracy will be very poor. According to Formulas 11 and 12, the Sobel operator can be used to calculate the gradients Gx and Gy of image I in the x and y directions respectively.

[0095]

[0096]

[0097] According to formula 13, the gradient magnitude G is calculated from the gradients Gx and Gy in the x and y directions;

[0098]

[0099] According to formula 14 and formula 15, calculate the reverse of the gradient;

[0100] α=atan(l y / l x ) Formula 14

[0101] β=atan(G y / G x ) Formula 15

[0102] According to formula 16, the texture matching standard deviation e of the observation is defined by the gradient direction and the epipolar direction. d ; It can be seen that when the gradient direction and the epipolar line direction are relatively consistent, the standard deviation can be controlled at a lower value, and the larger the gradient amplitude G, the smaller the standard deviation; when the gradient direction and the epipolar line direction are perpendicular, we assign a larger value, which is set to 10 here, representing a matching error of 10 pixels;

[0103]

[0104] Step S602: Determine the geometric confidence of the depth observation to be fused based on the pose information of the image reference frame and the common view frame; for example, the geometric standard deviation e is used for the geometric confidence. z To express, the following solution e z :

[0105] According to formula 17, for the reference frame and the common view frame, the relative pose can be solved by their poses;

[0106] T ref-src =T ref-w * (T src-w ) -1 Formula 17

[0107] According to formula 18, from the last column of the 4*4 transformation matrix of the relative pose, a 3D vector can be decomposed to find its modulus, that is, the baseline length b; the baseline length is also the distance between the optical centers of the reference frame and the common view frame;

[0108] b=||T ref-src[:,3]|| Formula 18

[0109] According to formula 19, solve the geometric standard deviation e z , where f is the focal length of the reference frame camera intrinsic parameter;

[0110]

[0111] From formula 19, we can see that the geometric standard deviation is the square of the distance between the observations (d obs The square of the geometric standard deviation indicates that the farther the point, the worse the reconstruction accuracy. The geometric standard deviation is negatively correlated with the baseline length b. The larger the baseline, the better the 3D position can be triangulated. The geometric standard deviation is negatively correlated with the focal length f of the reference frame camera intrinsic parameter. This indicates that the higher the resolution of the image, the finer the reconstruction. Finally, the geometric standard deviation is negatively correlated with the matching standard deviation e calculated in the previous step. d Positive correlation, so Formula 19 combines texture matching confidence and geometric confidence together;

[0112] Step S603: Determine the semantic confidence of the depth observation to be fused based on the semantic information calculated based on the image RGB information. Semantic segmentation is a pixel-level classification process where pixels belonging to the same class are grouped together. Therefore, semantic segmentation understands images at the pixel level. For example, pixels belonging to people are grouped together, pixels belonging to walls are grouped together, and even pixels belonging to elevators are grouped together. Currently, semantic segmentation technology based on deep learning can segment scenes with high accuracy and controllable computational cost.

[0113] For example, for semantic confidence, the final fusion standard deviation ef is used to represent it, and the following solves ef:

[0114] The semantic map is calculated for the RGB image of the common view frame. The solution of the semantic map does not rely on a specific method. All existing state-of-the-art solutions are applicable. Optionally, the classic deep learning FCN (Fully Convolutional Network) can be used for calculation. The calculated semantic map has the same resolution as the image. The semantic category label can be queried by the pixel position (u', v') according to Equation 20.

[0115] label=sematicMap(u′,v′) Formula 20

[0116] A semantic category label is an integer value that is bound to unique semantic information. All semantic category labels can be divided into two categories: static objects and dynamic objects. For example, people, cars, and airplanes are dynamic objects, while walls, ground, and buildings are static objects.

[0117] According to Formula 21, by querying the semantic category label of each observation pixel, a semantic confidence value can be assigned to the observation. For observations belonging to static objects, the previous standard deviation is inherited; for observations belonging to dynamic objects, an infinite standard deviation is assigned to it and no fusion is performed.

[0118]

[0119] Step S604: Merge the geometric confidence, texture matching confidence, and semantic confidence to obtain a merged confidence.

[0120] Through steps S601 to S604, in related technologies, large AR scenes often have a large number of dynamic objects, such as pedestrians, cars, etc., and existing commercial software technologies do not consider the impact of these dynamic objects. The embodiment of the present application models the texture confidence, semantic confidence and geometric confidence of three-dimensional reconstruction from multiple aspects, and proposes a fusion method based on multi-dimensional confidence. This method can effectively cope with different scenarios, especially the complex scenes required for large AR scene applications, such as shopping malls, parks, etc.

[0121] In some embodiments, Figure 7 is a schematic diagram of the transmission process of the point cloud fusion 3D reconstruction model according to the third embodiment of the present application, such as Figure 7 As shown in the figure, the transmission process of the point cloud fusion 3D reconstruction model based on multi-level confidence filtering includes inputting the camera pose and RGB image, as well as the depth map to be fused, thereby determining the geometric confidence, matching confidence, and semantic confidence, and performing multi-confidence fusion to finally output the scene point cloud reconstruction.

[0122] In some embodiments, Figure 8 FIG. 4 is a schematic diagram of a process for determining reconstruction points according to the convergence of the distribution of the fused point cloud according to the fourth embodiment of the present application. Figure 8 As shown, the process includes the following steps:

[0123] Step S801: for the Gaussian distribution, determine whether the fused standard deviation is less than a preset threshold. If so, it is determined to be converged.

[0124] For example, after obtaining the observation value and the fused observation standard deviation, for Gaussian distribution, the fusion can be performed using Formula 22;

[0125]

[0126] If the fused standard deviation is less than the threshold, it is determined to be converged, and the iterative update is stopped;

[0127] Step S802: If convergence is achieved, the reconstruction point in the world coordinate system is determined based on the average value of the Gaussian distribution, and the merging position of the corresponding common view frame is marked. If convergence is not achieved, one of the unoperated common view frames of the reference frame is continuously selected, the observation value is determined, and fusion is performed until all common view frames of the reference frame have been operated.

[0128] For example, if converged, the average value of the current Gaussian distribution is substituted into Formula 23 to obtain a reconstructed point X in the world coordinate system. new ;

[0129] X new =T w-ref *π -1 (u, v, μ′) Formula 23

[0130] The final reconstructed point cloud is composed of all the reconstructed points after the above steps; on the other hand, if the standard deviation after fusion is greater than the threshold, continue to search for other common view frames, continue in the same way, determine the new observation value and the fused observation standard deviation, and continue iterating the fusion.

[0131] This application is more suitable for depth maps with a large depth range, a lot of noise, and dynamic objects. The embodiment of this application also provides a comparison between the reconstruction effect of colmap in the related art and the reconstruction effect of this application. Figure 9 This is a schematic diagram of the colmapAR large scene 3D reconstruction effect based on related technologies. Figure 10 Schematic diagram of the AR large scene 3D reconstruction effect according to an embodiment of the present application, as shown in FIG. Figure 9 and Figure 10 As shown, it can be seen that the reconstruction produced by colmap has obvious reconstruction errors along the rays, and the reconstruction produced by the technical solution of the present application has better results.

[0132] The following compares the technical solution of this application with other solutions in the related art to illustrate the advantages of the embodiment of this application, such as high quality, low memory consumption, and high timeliness:

[0133] First, in terms of reconstruction accuracy, conventional depth fusion algorithms use a large number of discrete points in visual depth restoration images, resulting in a fused model with significant uncertainty. The fused model often contains flying points, making it difficult to directly apply to grid-based reconstruction. The embodiments of the present application consider the impact of confidence from multiple perspectives, enabling the removal of divergent discrete points and dynamic objects such as pedestrians. This ensures better reconstruction accuracy, and the fusion reconstruction method is suitable for the reconstruction of large AR scenes and is also compatible with other reconstruction scenarios.

[0134] Second, compared with the voxel fusion method, the depth distance range of visual depth is larger. If the TSDF method is used, too many voxels are required, which means extremely high memory consumption. Moreover, different scenes vary greatly, and it is difficult to use a suitable voxel resolution to represent the reconstructed scene. In addition, the traditional method requires each depth image to be loaded into the memory for fusion. For large AR scenes, there are often tens of thousands of depth maps with a resolution of one million (or even ten million) to be fused, which consumes a lot of memory. The algorithm of the embodiment of the present application has a smaller memory usage than the TSDF solution and can be deployed on a cloud server. Due to its lightweight computation and low memory consumption, it can also be deployed on a mobile terminal.

[0135] Third, compared with the existing probabilistic fusion method REMODE, which considers the difference in depth uncertainty caused by the error of matching one pixel, this method is aimed at fusing multiple frames of depth maps into a high-confidence depth map and cannot directly reconstruct a dense point cloud. The algorithm of the embodiment of the present application is aimed at point cloud fusion of image recovery depth map. If the depth map is collected by other sensing devices (such as ToF camera), it is also compatible. It should be noted that the embodiment of the present application is applicable to scenarios that do not rely on additional depth sensors. Without adding sensors, the technical solution can be directly used in existing smart devices. In addition, the embodiment of the present application is also compatible with data collected by applicable depth sensors.

[0136] Fourth, compared with the method based on deep learning implicit spatial expression, the deep learning method has good reconstruction completeness and accuracy. However, the current academic frontier work, such as Neural Recon, is mainly developed for indoor scenes. It performs poorly for outdoor scenes with a large depth, and is extremely dependent on the computing power and memory size of the GPU. The algorithm of the embodiment of the present application does not rely on the support of specific devices such as GPU. For the first time, it constructs a unified confidence model from the three aspects of texture matching, geometry, and semantics, and proposes a confidence fusion method, that is, fusion in a filtering manner, which greatly improves the computational efficiency. At the same time, the three-dimensional point cloud observations are integrated into one dimension, which greatly simplifies the computational complexity of the problem and greatly improves the computational efficiency. In addition, by selecting a reasonable error distribution assumption, such as Gaussian distribution, the computational efficiency is further improved.

[0137] In conjunction with the point cloud fusion 3D reconstruction method based on multivariate confidence filtering in the above embodiments, embodiments of the present application may provide a storage medium for implementation. The storage medium stores a computer program; when executed by a processor, the computer program implements any of the point cloud fusion 3D reconstruction methods based on multivariate confidence filtering in the above embodiments.

[0138] In one embodiment, a computer device is provided, which may be a terminal. The computer device includes a processor, memory, a network interface, a display screen, and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a point cloud fusion three-dimensional reconstruction method based on multivariate confidence filtering is implemented. The display screen of the computer device may be a liquid crystal display or an electronic ink display screen. The input device of the computer device may be a touch layer covering the display screen, or may be a key, trackball, or touchpad provided on the computer device housing, or may be an external keyboard, touchpad, or mouse.

[0139] In one embodiment, Figure 11 is a schematic diagram of the internal structure of an electronic device according to an embodiment of the present application, such as Figure 11 As shown, an electronic device is provided, which may be a server, and its internal structure diagram may be as shown in FIG. Figure 11 As shown. The electronic device includes a processor, a network interface, an internal memory, and a non-volatile memory connected via an internal bus, wherein the non-volatile memory stores an operating system, a computer program, and a database. The processor is used to provide computing and control capabilities, the network interface is used to communicate with external terminals via a network connection, the internal memory is used to provide an environment for the operation of the operating system and the computer program. When the computer program is executed by the processor, it implements a point cloud fusion 3D reconstruction method based on multivariate confidence filtering, and the database is used to store data.

[0140] Those skilled in the art will understand that Figure 11 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the electronic device to which the solution of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0141] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, which can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0142] Those skilled in the art should understand that, in order to keep the description concise, not all possible combinations of the various technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0143] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A point cloud fusion 3D reconstruction method based on multivariate confidence filtering, characterized in that: The method comprises: Acquire an image, determine the multivariate confidence of the depth observation to be fused based on the RGB information, pose information and corresponding depth map of each frame image, and merge the multivariate confidence to obtain a merged confidence. Before determining the multivariate confidence of the depth observation to be fused, the method includes: traversing the pixels of the reference frame depth map to obtain a first pixel, determining the coordinates of the target point of the first pixel in the reference frame coordinate system by back-projecting the depth value of the first pixel to obtain a first coordinate, determining the coordinates of the target point in the world coordinate system by pose transformation to obtain a second coordinate, selecting one of the unoperated common view frames of the reference frame, and determining the observation value according to the second coordinate, including: According to the second coordinate, the pixel position of the target point on the common view frame is determined by posture transformation and projection equation to obtain the second pixel, and the coordinates of the target point of the second pixel in the world coordinate system are determined by back-projection of the depth value of the second pixel to obtain the third coordinate. According to the first coordinate, the third coordinate and the optical center of the reference frame, the coordinates of the projection of the position corresponding to the third coordinate on the observation vector of the reference frame are determined, and the result is used as the observation value. The multivariate confidence of the depth observation to be fused includes a geometric confidence, and the geometric confidence is positively correlated with the square of the observation value; For each observation, fusion is performed according to its corresponding combined confidence; a reconstruction point is determined according to the fusion result; and a three-dimensional reconstruction is performed based on the reconstruction point.

2. The method according to claim 1, characterized in that The process of performing fusion and determining the reconstruction point based on the fusion result includes: fusing the observations on each group of common view frames to the reference frame through Bayesian filtering, and determining the reconstruction point based on the convergence of the point cloud distribution after fusion.

3. The method according to claim 2, characterized in that The process of determining the merge confidence level includes: Determine the geometric confidence of the depth observation to be fused based on the pose information of the image reference frame and the common view frame; determine the texture matching confidence of the depth observation to be fused based on the image RGB information and the pose information of the reference frame and the common view frame; determine the semantic confidence of the depth observation to be fused based on the semantic information calculated based on the image RGB information; The geometric confidence, texture matching confidence and semantic confidence are combined to obtain a combined confidence.

4. The method according to claim 3, characterized in that The process of determining the multivariate confidence of the depth observations to be fused includes: Determine the matching confidence of the depth observation texture to be fused and express it as a matching standard deviation; Determine a transformation matrix of the relative posture according to the posture information of the reference frame and the common view frame, and determine the distance between the optical center of the reference frame and the optical center of the common view frame according to the transformation matrix to obtain the optical center distance; Determine a geometric confidence level based on the square of the observation value, the focal length of the reference frame camera intrinsic parameter, the optical center distance, and the matching standard deviation, and express it as a geometric standard deviation; The semantic category label of each pixel to which an observation belongs is queried to determine the semantic confidence of the observation, where the standard deviation corresponding to the observation of dynamic objects is infinite.

5. The method according to claim 4, characterized in that The process of determining the reconstruction point according to the convergence of the fused point cloud distribution includes: For Gaussian distribution, determine whether the standard deviation after fusion is less than the preset threshold. If so, it is considered to be converged. If converged, determine the reconstruction point in the world coordinate system based on the average value of the Gaussian distribution and mark the merge position of the corresponding common view frame; If convergence has not occurred, one of the unoperated common view frames of the reference frame is continuously selected to determine the observation value and perform fusion until all the common view frames under the reference frame have been operated.

6. A point cloud fusion 3D reconstruction system based on multivariate confidence filtering, characterized in that: The system comprises: The determination module is used to obtain an image, determine the multivariate confidence of the depth observation to be fused based on the RGB information, pose information and corresponding depth map of each frame image, and merge the multivariate confidence to obtain a merged confidence. Before determining the multivariate confidence of the depth observation to be fused, the determination module is further used to: traverse the pixels of the reference frame depth map to obtain a first pixel, determine the coordinates of the target point of the first pixel in the reference frame coordinate system by back-projecting the depth value of the first pixel, obtain the first coordinate, determine the coordinates of the target point in the world coordinate system by pose transformation, obtain the second coordinate, select one of the unoperated common view frames of the reference frame, and determine the observation value according to the second coordinate, including: According to the second coordinate, the pixel position of the target point on the common view frame is determined by posture transformation and projection equation to obtain the second pixel, and the coordinates of the target point of the second pixel in the world coordinate system are determined by back-projection of the depth value of the second pixel to obtain the third coordinate. According to the first coordinate, the third coordinate and the optical center of the reference frame, the coordinates of the projection of the position corresponding to the third coordinate on the observation vector of the reference frame are determined, and the result is used as the observation value. The multivariate confidence of the depth observation to be fused includes a geometric confidence, and the geometric confidence is positively correlated with the square of the observation value; The fusion module is used to perform fusion for each observation according to its corresponding combined confidence; determine the reconstruction point according to the fusion result; and perform three-dimensional reconstruction based on the reconstruction point.

7. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to execute the point cloud fusion three-dimensional reconstruction method based on multivariate confidence filtering according to any one of claims 1 to 5.

8. A storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program is configured to execute the point cloud fusion three-dimensional reconstruction method based on multivariate confidence filtering according to any one of claims 1 to 5 when running.

Images