A three-dimensional reconstruction method based on foreground extraction

Abstract

The application is suitable for the field of three-dimensional reconstruction, and provides a three-dimensional reconstruction method based on foreground extraction.The application has the advantages that: the pre-processing method used in the application concentrates more filtering functions in the pre-processing stage compared with the existing three-dimensional reconstruction algorithm, the picture-based pre-processing consumes less resources than the point cloud-based filtering, and improves the real-time performance and stability of the system; compared with the traditional registration method, the time spent in registration is too long, and the error is large, the registration method used in the application consumes less computing resources and has higher accuracy; the application uses a facet model, which occupies less system memory resources than the traditional mesh TSDF model; at the same time, the facet model is more suitable for the random fern algorithm in the closed loop detection.In summary, the application has the advantages of high real-time performance and strong anti-interference ability for the three-dimensional reconstruction algorithm system.

Description

Technical Field

[0001] This invention relates to three-dimensional reconstruction technology, and more particularly to a three-dimensional reconstruction method based on foreground extraction. Background Technology

[0002] 3D reconstruction technology is a process of converting real-world objects, scenes, or environments into 3D models. 3D reconstruction enables environmental perception and navigation; it can construct environmental maps for path planning and obstacle avoidance. In the medical field, 3D reconstruction can be used for medical image processing and visualization, aiding in diagnosis and surgical planning. In recent years, 3D reconstruction has been increasingly widely applied in the integration of automation and medicine, enabling intelligent healthcare and improving its convenience.

[0003] The main drawbacks of current 3D reconstruction technology lie in its computationally complex process and poor resistance to interference. During 3D reconstruction, the processing of large-scale point cloud data requires significant computational resources and complex mathematical algorithms, limiting real-time performance. Furthermore, the sensitivity of sensors to external stimuli can lead to data loss and discrepancies in reconstruction, resulting in distortion and incomplete reconstruction results, thus affecting accuracy. Therefore, multiple sampling sessions are necessary to obtain accurate data and achieve satisfactory results. Summary of the Invention

[0004] This invention provides a three-dimensional reconstruction method based on foreground extraction, aiming to solve the problems of weak real-time performance and weak anti-interference ability in existing three-dimensional reconstruction technologies.

[0005] The three-dimensional reconstruction method includes the following steps:

[0006] By calibrating the camera, the parameters of the color camera and the depth camera are obtained, a camera model is established, and spatial coordinate transformation relationships are introduced to complete the alignment of RGB-D data;

[0007] Multiple frames of aligned RGB-D data are acquired using an RGB-D sensor; wherein, the frame sets include color images and depth images;

[0008] For each set of frames, data is completed by preprocessing through a preset training network to obtain the first set of data frames.

[0009] The first data frame set is filtered to obtain the second data frame set;

[0010] The second data frame set is converted into a target point cloud, and the highly correlated parts of the target point cloud are registered based on a preset registration algorithm to obtain the registration result.

[0011] A point cloud fusion front-end controller is established, which is used to predict the trajectory of the target point cloud and determine whether to fuse the target point cloud based on the registration result and preset conditions.

[0012] A surface element model is established based on the fused target point cloud;

[0013] The surface model is globally optimized using a deformation graph, and loop closure detection is performed on the globally optimized surface model using an enhanced random fern algorithm based on directional gradient histograms.

[0014] Preferably, the preset training network is a YOLO network.

[0015] Preferably, in the step of preprocessing each of the frame sets through a preset training network to obtain the first data frame set, the preprocessing process includes:

[0016] The color image is processed using a YOLO network to select the first image region in the color image that is expected to be reconstructed.

[0017] Find the second image region corresponding to the first image region in the depth image and traverse it to retrieve regions with missing depth in the second image region.

[0018] Determine whether there are any regions with missing depth in the second image region. If so, fill in the depth image using the YuPnP method; otherwise, skip this search and continue searching the next frame set.

[0019] Preferably, the filtering process includes:

[0020] The outlier removal algorithm marks pixels that deviate from the foreground in the first data frame as obvious background pixels.

[0021] Traverse the regions in the depth image, determine the sparsity based on the depth value of each pixel in the region, and mark the sparse points as probabilistic background points according to a preset threshold.

[0022] Grabcut segmentation is performed on all the aforementioned probabilistic background points;

[0023] The first data frame set is subjected to bilateral filtering to obtain the second data frame set.

[0024] Preferably, the preset configuration algorithm is an improved ICP algorithm based on kd-tree.

[0025] Preferably, the improved ICP algorithm satisfies the following relationship:

[0026]

[0027] Where, x i y i Let X and Y represent a pair of points in different sets of target point clouds, respectively. Let R represent the rotation matrix, t represent the translation vector, and F represent the number of points in the target point cloud.

[0028] Preferably, the preset condition is: whether the distance between the two registered target point clouds is less than a preset distance threshold; if so, the fusion is abandoned; if not, the fusion is performed.

[0029] Preferably, the set of all facets of the facet model is defined as follows: One of the face elements is Affected nodes are In the step of globally optimizing the surface element model using the deformation diagram, the deformation position and normal vector of the surface element model satisfy the following relationship:

[0030]

[0031]

[0032]

[0033] in, This is a deformed diagram. For nodes, each node The transformation matrix of this node is stored. Translation vector Location Represents a node right Influence weight, d max Indicated to The distance to the (k+1)th nearest neighbor node. Representation of face element The position after deformation Representation of face element The deformed normal vector.

[0034] Compared with existing technologies, the advantages of this invention are as follows: The preprocessing method used in this invention, compared with existing 3D reconstruction algorithms, concentrates more filtering functions in the preprocessing stage. Image-based preprocessing consumes fewer resources than point cloud-based filtering, improving the real-time performance and stability of the system. Compared with traditional registration methods, which are too time-consuming and have large errors, the registration method used in this invention uses fewer computational resources and has higher accuracy. This invention uses a pixel model, which consumes less system memory resources compared with the traditional mesh-based TSDF model. At the same time, the pixel model is more suitable for the random fern algorithm in loop closure detection. In summary, for 3D reconstruction algorithm systems, this invention has the advantages of high real-time performance and strong anti-interference ability. Attached Figure Description

[0035] The present invention will now be described in detail with reference to the accompanying drawings. The above and other aspects of the present invention will become clearer and more readily understood through the detailed description following the accompanying drawings. In the drawings:

[0036] Figure 1 This is a flowchart of the three-dimensional reconstruction method based on foreground extraction provided in an embodiment of the present invention;

[0037] Figure 2 This is a schematic diagram of the coordinate transformation relationship of the three-dimensional reconstruction method based on foreground extraction provided in the embodiments of the present invention;

[0038] Figure 3 This is a schematic diagram of the RGB-D data preprocessing flow of the three-dimensional reconstruction method based on foreground extraction provided in the embodiments of the present invention. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0040] Please refer to Figures 1-3 This invention provides a three-dimensional reconstruction method based on foreground extraction, characterized in that the three-dimensional reconstruction method includes the following steps:

[0041] S101. Obtain the parameters of the color camera and the depth camera by calibrating the camera, establish a camera model and introduce spatial coordinate transformation relationship to complete the alignment of RGB-D data;

[0042] In this embodiment of the invention, the camera is first calibrated, and the parameters of the color camera and the depth camera are obtained, respectively K. rgb and K dFor a given spatial point P, whose coordinates are (X, Y, Z), its pixel is p.

[0043] By creating a camera model, the problem of two-dimensional pixels can be transformed into a problem of three-dimensional points in space.

[0044]

[0045] Introducing spatial coordinate transformation relationships to complete RGB-D data alignment mainly involves obtaining the transformation matrix from the color coordinate system to the depth coordinate system.

[0046] For the color camera coordinate system C, the depth camera coordinate system D, and the world coordinate system W, extrinsic parameter calibration can be performed on the color camera and the depth camera to obtain the transformation matrix between them. C T D The transformation relation matrix T consists of the rotation matrix R and the translation vector t.

[0047] For a spatial point P, its three-dimensional coordinates in the world coordinate system W are Pi. W Its three-dimensional coordinates in the color coordinate system C are P. C Its three-dimensional coordinates in the depth coordinate system D are P D .

[0048] Please refer to Figure 2 The transformation relationships from the world coordinate system to the color coordinate system and the depth coordinate system can be easily obtained through calibration. C T W and D T W Spatial coordinate point P W The following transformation relationships exist:

[0049] P C = C R W P W + C t W (1);

[0050] P D = D R W P W + D t W (2);

[0051] From transformation relations (1) and (2), we can obtain transformation relation (3).

[0052]

[0053] That is what is sought

[0054] In summary, the alignment of the color image and the depth image was completed, maintaining the pixel correspondence between the two images.

[0055] S102. Acquire a set of multiple aligned RGB-D data frames using an RGB-D sensor; wherein the set of frames includes color images and depth images;

[0056] In this embodiment of the invention, a frame set I after alignment is obtained from an RGB-D sensor. F One set of frames consists of a color image I. C and a depth image I D composition.

[0057] I F ={I C I D};

[0058] S103. For each set of frames, preprocessing is performed through a preset training network to complete the data completion, and a first set of data frames is obtained.

[0059] In this embodiment of the invention, the preset training network is a YOLO network. The color image is processed by the YOLO network to select the first image region in the color image that is to be reconstructed;

[0060] Find the second image region corresponding to the first image region in the depth image and traverse it to retrieve regions with missing depth in the second image region.

[0061] Determine whether there are any regions with missing depth in the second image region. If so, fill in the depth image using the YuPnP method; otherwise, skip this search and continue searching the next frame set.

[0062] Specifically, a pre-trained YOLO network is used to process color images I. C After preprocessing, the bounding box P containing the location of the foreground in the segmented image is output. CC That is, check the color image I. C The first image region P to be reconstructed CC By ignoring the invalid information that does not need to be reconstructed, the image filtering function is initially realized, simplifying the computational workload of the process and ensuring the real-time performance of the entire process.

[0063] In depth image I D Find color image I C The first image region P CC The corresponding second image region P DAC Traverse the second image region P of the depth imageDAC The system searches for missing regions in the depth image. If no depth loss is found, it skips the current region and continues to the next set of frames.

[0064] For areas where depth values ​​are lost, the YuPnP method is used to complete the depth map, resulting in the completed selected region P. DAP .

[0065] The first data frame set after completing the foreground selection is

[0066] S104. Filter the first data frame set to obtain the second data frame set;

[0067] In this embodiment of the invention, the filtering process includes: marking pixels in the first data frame that deviate from the foreground as obvious background points using an outlier removal algorithm;

[0068] Traverse the regions in the depth image, determine the sparsity based on the depth value of each pixel in the region, and mark the sparse points as probabilistic background points according to a preset threshold.

[0069] Grabcut segmentation is performed on all the aforementioned probabilistic background points;

[0070] The first data frame set is subjected to bilateral filtering to obtain the second data frame set.

[0071] Specifically, extract the first data frame set of the foreground after padding. Using depth images, an outlier removal algorithm is applied to the image within the region, marking pixels that are significantly deviated from the foreground as obvious background points.

[0072] Based on this, the entire depth image is traversed, and for each pixel, a search is conducted within a certain range around it to find whether there are background points. If the range exceeds a certain threshold, the pixel is marked as a probabilistic background pixel, and these probabilistic background points are recorded as p = {p1, ..., p2}. n}

[0073] Even after using the outlier removal algorithm, some background points may still remain. The probabilistic background points obtained using sparsity assessment methods may be misidentified. Therefore, a grab-cut segmentation is performed on these probabilistic background points to remove those clearly identified as background points, thus eliminating redundant noise.

[0074] Perform bilateral filtering on the data frame set after foreground extraction to obtain the second data frame set I after foreground extraction. P This results in a smoother image, further enhancing its anti-interference capabilities, and this step effectively reduces the computational scale.

[0075] S105. Convert the second data frame set into a target point cloud, and perform registration processing on the highly correlated parts of the target point cloud based on a preset registration algorithm to obtain the registration result.

[0076] In this embodiment of the invention, the second data frame set I input at time K... P A set of frames Convert it into a point cloud The partially reconstructed point cloud is P g .

[0077] If P is directly g and Registration work is not only time-consuming but also has a higher probability of error. Therefore, this invention only extracts the point cloud parts with high correlation for registration.

[0078] Therefore, before registration, the motion trajectory predicted by the point cloud fusion front-end controller is used. The reconstructed point cloud model P g Based on the probability of motion prediction, the reconstructed area is divided into high-probability overlapping point cloud partitions and low-probability overlapping point cloud partitions, named respectively. and Low probability overlapping point cloud partitioning and The relevance is low, so no update is needed.

[0079] To improve reconstruction speed, only extract and To perform overlapping region detection, after extracting the overlapping regions of the two point clouds, feature point matching and corresponding descriptor calculation are performed on them respectively, and the point cloud corresponding to the above feature points is extracted, X = {x1, ..., x2}. n}, Y = {y1, ..., y2} n Point cloud registration was performed using an improved ICP algorithm based on kd-tree.

[0080]

[0081] Where, x i y i Let X and Y represent a pair of points in different sets of target point clouds, respectively. Let R represent the rotation matrix, t represent the translation vector, and F represent the number of points in the target point cloud.

[0082] The optimal rotation matrix R and translation vector t are obtained through iterative registration algorithms, thereby calculating the relative motion between the two point clouds.

[0083] S106. Establish a point cloud fusion front-end controller, which is used to predict the trajectory of the target point cloud and determine whether to fuse the target point cloud according to the registration result and preset conditions.

[0084] In this embodiment of the invention, the preset condition is: whether the distance between the two registered target point clouds is less than a preset distance threshold. If yes, the fusion is abandoned; otherwise, the fusion is performed.

[0085] Specifically, according to the calculation T g，K To determine whether the fusion threshold has been reached, the actual distance between the two registered point clouds is calculated using an improved projection method.

[0086]

[0087] Where, d u It is the distance between the two point clouds being registered, V k N k V represents the set of points and the normal map (normal set) obtained from the point cloud map. k It contains three-dimensional information of all pixels, where g represents the coordinates in global coordinates, u represents the pixel, and V is the point cloud. k A point in the matrix is ​​registered at time k, where k represents time k. This represents the coordinates of pixel u in the global coordinate system at time k-1. This indicates that at time k-1, in the global coordinate system, The normal vector of the corresponding pixel u.

[0088] The controller is set with a distance threshold d. θ Calculate its relationship with d u The difference between the values ​​determines whether to fuse the new point cloud. If the controller determines d u <d θ If the range of motion is small, the fusion process is abandoned; otherwise, the point cloud fusion process is executed.

[0089] S107. Establish a surface element model based on the fused target point cloud;

[0090] In this embodiment of the invention, a surface element model is used to store the model, converting the model to be registered from point cloud form into surface element form. This simplifies the fusion process and makes subsequent loop closure detection more convenient.

[0091] Compared to complex mesh models, lightweight pixel models are 3D surface models based on basic point representations. The reconstructed 3D surface is represented by multiple overlapping pixels, each containing attributes such as point position p, normal vector n, radius r, and color c. The current global pixel is represented by the value s, the newly measured value by l, and the weights used for fusion are ω. s and ω m .

[0092] The formula for blending overlapping facets is shown below:

[0093]

[0094]

[0095] p sl =ω s p s +ω l p l ;

[0096] n sl =ω s n s +ω l n l ;

[0097]

[0098] S108. The surface element model is globally optimized using the deformation graph, and the globally optimized surface element model is subjected to loop closure detection using the enhanced random fern algorithm based on the directional gradient histogram.

[0099] In this embodiment of the invention, after the point cloud trajectory is acquired, it is also necessary to achieve global optimization trajectory consistency of the three-dimensional surface model. The surface element model representation method can achieve this function by constructing a deformation diagram to achieve non-rigid deformation.

[0100] Define the set of all facets of the facet model as follows: One of the face elements is Affected nodes are The deformation position and normal vector of the surface element are calculated as follows:

[0101]

[0102]

[0103]

[0104] in This is a deformed diagram. For nodes, each node The transformation matrix of this node is stored. Translation vector Location It represents a node right Influence weight, d max Is to The distance to the (k+1)th nearest neighbor node. Representation of face element The position after deformation Representation of face element The deformed normal vector. Note that here, 'n' represents the normal vector, and 's' refers to a single face element in the entire face element set, named 's'. This is similar to extracting a pixel 'u' from the pixel set in the previous steps, i.e., naming it 'u'.

[0105] The Gauss-Newton method is used to optimize the iterative global loop closure detection process. Obtain the optimized deformation diagram Optimize all face elements.

[0106] In this embodiment of the invention, an enhanced random fern algorithm based on HOG (Histogram of Oriented Gradients) is used to determine whether an already observed region in the scene has been visited.

[0107] Compared with existing technologies, the advantages of this invention are as follows: The preprocessing method used in this invention, compared with existing 3D reconstruction algorithms, concentrates more filtering functions in the preprocessing stage. Image-based preprocessing consumes fewer resources than point cloud-based filtering, improving the real-time performance and stability of the system. Compared with traditional registration methods, which are too time-consuming and have large errors, the registration method used in this invention uses fewer computational resources and has higher accuracy. This invention uses a pixel model, which consumes less system memory resources compared with the traditional mesh-based TSDF model. At the same time, the pixel model is more suitable for the random fern algorithm in loop closure detection. In summary, for 3D reconstruction algorithm systems, this invention has the advantages of high real-time performance and strong anti-interference ability.

[0108] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0109] The embodiments of the present invention have been described above with reference to the accompanying drawings. The disclosed embodiments are merely preferred embodiments of the present invention. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many equivalent changes in form without departing from the spirit and scope of the claims of the present invention, and all such changes are within the protection scope of the present invention.

Claims

1. A three-dimensional reconstruction method based on foreground extraction, characterized in that, The three-dimensional reconstruction method includes the following steps: By calibrating the camera, the parameters of the color camera and the depth camera are obtained, a camera model is established, and spatial coordinate transformation relationships are introduced to complete the alignment of RGB-D data; Multiple frames of aligned RGB-D data are acquired using an RGB-D sensor; wherein, the frame sets include color images and depth images; For each set of frames, data is completed by preprocessing through a preset training network to obtain the first set of data frames. The first data frame set is filtered to obtain the second data frame set; The second data frame set is converted into a target point cloud, and the highly correlated parts of the target point cloud are registered based on a preset registration algorithm to obtain the registration result. A point cloud fusion front-end controller is established, which is used to predict the trajectory of the target point cloud and determine whether to fuse the target point cloud based on the registration result and preset conditions. A surface element model is established based on the fused target point cloud; The surface model is globally optimized using a deformation graph, and the optimized surface model is then subjected to loop closure detection using an enhanced random fern algorithm based on the directional gradient histogram. The preset training network is a YOLO network; In the step of preprocessing each of the frame sets through a preset training network to obtain the first data frame set, the preprocessing process includes: The color image is processed using a YOLO network to select the first image region in the color image that is expected to be reconstructed. Find the second image region corresponding to the first image region in the depth image and traverse it to retrieve regions with missing depth in the second image region. Determine whether there are any regions with missing depth in the second image region. If so, fill in the depth image using the YuPnP method; otherwise, skip this search and continue searching the next frame set. The filtering process includes: The outlier removal algorithm marks pixels that deviate from the foreground in the first data frame as obvious background pixels. Traverse the regions in the depth image, determine the sparsity based on the depth value of each pixel in the region, and mark the sparse points as probabilistic background points according to a preset threshold. Grabcut segmentation is performed on all the aforementioned probabilistic background points; The first data frame set is subjected to bilateral filtering to obtain the second data frame set.

2. The three-dimensional reconstruction method based on foreground extraction as described in claim 1, characterized in that, The preset configuration algorithm is based on kd-tree Improvements ICP algorithm.

3. The three-dimensional reconstruction method based on foreground extraction as described in claim 2, characterized in that, The improved ICP algorithm satisfies the following relationship: ； in, Let X and Y represent a pair of points in different sets of target point clouds, respectively. R Represents the rotation matrix. t Represents the translation vector. F This indicates the number of points in the target point cloud.

4. The three-dimensional reconstruction method based on foreground extraction as described in claim 1, characterized in that, The preset condition is: whether the distance between the two registered target point clouds is less than a preset distance threshold. If yes, the fusion is abandoned; otherwise, the fusion is performed.

5. The three-dimensional reconstruction method based on foreground extraction as described in claim 1, characterized in that, Define the set of all facets of the facet model as follows: One of the face elements is The affected nodes are Then, in the step of globally optimizing the surface element model using the deformation diagram, the deformation position and normal vector of the surface element model satisfy the following relationship: ； ； in, This is a deformed diagram. For nodes, each node The transformation matrix of this node is stored. ,Location Represents a node right Influence weight, arrive The nearest k +1 node distance, Representation of face element The position after deformation Representation of face element The deformed normal vector.

Images