A Two-Stage Subject Point Cloud Filtering Method Based on RGBD Camera

Abstract

This invention discloses a two-stage point cloud filtering method based on an RGBD camera, belonging to the field of point cloud filtering technology. The method includes: reading RGBD camera data containing the subject to be photographed, and obtaining aligned RGB and depth image pairs; generating an original 3D point cloud image based on camera calibration intrinsic parameters and the aligned RGB and depth image pairs; and using a learning-based foreground matting technique to extract the foreground from the RGB image pair and generate a transparency image of the foreground object. This invention can accurately extract the point cloud of the subject, overcoming the shortcomings of traditional filtering algorithms that cannot effectively remove edge noise at discontinuities in the depth domain of the point cloud caused by the manufacturing process and measurement principle of RGBD cameras. Furthermore, it can be applied to point cloud filtering tasks with cameras of any depth and with any subject.

Description

Technical Field

[0001] This invention relates to the field of point cloud filtering technology, and in particular to a two-stage subject point cloud filtering method based on an RGBD camera. Background Technology

[0002] With the latest advancements in 3D scanning technology, RGBD cameras, capable of acquiring aligned RGB and depth images in real time, have been widely used in the field of 3D vision. However, due to the influence of camera manufacturing processes, measurement principles, and external environments, point clouds generated from depth images acquired by RGBD cameras contain significant noise, particularly noticeable at discontinuous edges in the depth domain. Only by effectively removing this noise through filtering can subsequent point cloud-related tasks be performed more efficiently.

[0003] Most existing filtering requirements are met through traditional point cloud filtering methods, such as pass-through filtering, voxel filtering, and statistical filtering, or directly based on widely used digital image processing algorithms, such as mean filtering, median filtering, and bilateral filtering. However, these algorithms have two significant drawbacks. First, they usually need to be used in combination to ensure effective removal of most noise. Furthermore, the second and most critical drawback of these methods is their inability to handle edge noise at discontinuities in the depth domain. This type of noise is attached to the edges of the main point cloud, is close to it, and is numerous; the methods described above are clearly ineffective at removing this type of noise. Summary of the Invention

[0004] The present invention provides a two-stage subject point cloud filtering method based on an RGBD camera, which is highly versatile and overcomes some of the defects of the prior art.

[0005] The present invention adopts the following technical solution:

[0006] A two-stage subject point cloud filtering method based on an RGBD camera includes the following specific steps:

[0007] Step 1: Read the RGBD camera image data containing the subject being photographed to obtain an image pair consisting of an aligned RGB image and a depth image;

[0008] Step 2: Based on the camera intrinsic data and aligned RGB and depth image pairs, restore the pixels to 3D space to generate the original 3D point cloud image to be filtered in the camera coordinate system. The mapping formula between the image and the point cloud is:

[0009]

[0010] in,( () represents the pixel coordinates on the depth image or RGB image. For the camera intrinsic parameter matrix, These are the coordinates of the 3D point in the point cloud corresponding to the current pixel coordinates;

[0011] Step 3: Use a learning-based foreground matting technique to extract the foreground from the RGB image of the image pair and generate a transparency image of the foreground object;

[0012] Step 4: Determine the spatial neighborhood of the pixel to which the main object belongs based on the transparency image. R and spatial neighborhood weight set W Depth restoration is performed on the main object in the spatial domain to obtain a complete point cloud of the main object. The spatial neighborhood of the pixels to which the main object belongs... R The specific components are as follows:

[0013] First, construct a c*c pixel original spatial neighborhood centered on the current pixel. If elements are stored at the same pixel position in the transparency image... If the neighboring pixel is not found, it is discarded; otherwise, it is retained, thus forming the current pixel's neighborhood. R Neighborhood R The weights of each pixel in the restoration depth The formula is as follows:

[0014]

[0015] Where c is the side length of the original rectangular spatial neighborhood. The preset optimal threshold, The coordinates of the current pixel. The coordinates of the neighboring pixels, Constructing a neighborhood weight set W ;

[0016] Step 5: Based on prior information such as ensuring no obstruction between the subject and the camera during shooting, and the subject's size, determine the coordinates in the camera coordinate system. The point cloud filtering interval along the axis completes the first stage of coarse filtering for the main point cloud;

[0017] Step 6: Superimpose the second-stage fine filtering, indicated by the transparency image, onto the result of the first-stage coarse filtering to obtain the final subject point cloud filtering result.

[0018] As a further optimization of the present invention, the specific steps of step 3 are as follows:

[0019] The foreground segmentation network is used to segment the foreground of the acquired RGB image. The network output is a transparency map S with the same resolution as the input image and is aligned pixel by pixel with the original image. The element a stored at each pixel position represents the degree of blending between the foreground and the background at the current pixel.

[0020] As a further optimization of the present invention, the specific steps of step 5 are as follows:

[0021] First, based on the prior condition that there are no other objects obstructing the view between the subject and the camera, the coordinates of all points in the original point cloud are traversed. Value, extract the minimum depth value Then the point containing the current minimum depth value is the point in the main object that is closest to the camera coordinate system xoy plane;

[0022] Next, based on the prior size information b of the current main object in the Z direction, the following coarse filtering formula is obtained:

[0023]

[0024] in, This is the point cloud after coarse filtering; This refers to any point in the original cloud. This represents the Z-coordinate value of the current point;

[0025] As a further optimization of the present invention, the specific steps of step 6 are as follows:

[0026] The foreground segmentation network is used to segment the foreground of the acquired RGB image. The network output is a transparency map S with the same resolution as the input image. The element a stored at each pixel position in S represents the degree of blending between the foreground and the background of the current pixel.

[0027] The transparency values ​​at each pixel location in the transparency map S are iterated. To ensure accurate removal of edge noise, the transparency value 'a' is used as the basis for the coarse filtering result. As an indicator of whether something belongs to the foreground object, fine filtering is performed. The specific formula is as follows:

[0028]

[0029] in, The preset optimal threshold, This is the point cloud after fine filtering; is the projection function, representing the two-dimensional pixel in the image corresponding to the three-dimensional point in the point cloud; express The transparency value of the projection point corresponding to the point;

[0030] Compared with existing technologies, the algorithm of this invention has a clear and effective process. It effectively removes overall non-subject noise by using only prior information. For the first time, it combines image matting technology as a guide for subsequent depth restoration and edge noise removal. It overcomes the shortcomings of traditional filtering algorithms that cannot effectively remove edge noise at discontinuities in the depth domain of point clouds. It can accurately extract the subject point cloud and is reproducible for point cloud extraction from any RGBD camera and any subject. Attached Figure Description

[0031] Figure 1 This is a flowchart of the method of the present invention;

[0032] Figure 2 It consists of aligned RGB and depth images acquired by an RGBD camera;

[0033] Figure 3 It is the original 3D point cloud image with color information to be filtered;

[0034] Figure 4 This is the result of foreground extraction from an RGB image;

[0035] Figure 5 It is a 3D point cloud map with color information after coarse filtering;

[0036] Figure 6 It is a 3D point cloud map with color information after fine filtering; Detailed Implementation

[0037] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0038] like Figure 1 As shown, this invention provides a two-stage subject point cloud filtering method based on an RGBD camera, comprising the following specific steps:

[0039] Step 1: Read the RGBD camera image data containing the subject being photographed, and obtain image pairs consisting of aligned RGB images and depth images, such as... Figure 2 As shown; the operation of aligning the RGB image and the depth image is implemented by the camera pose parameters provided by the camera manufacturer or the camera development SDK.

[0040] Step 2: Based on the camera intrinsic data and aligned RGB and depth image pairs, reconstruct the pixels into 3D space to generate the original 3D point cloud image to be filtered in the camera coordinate system, such as... Figure 3 As shown, the mapping formula between the image and the point cloud is:

[0041]

[0042] in,( () represents the pixel coordinates on the depth image or RGB image. For the camera intrinsic parameter matrix, These are the coordinates of the 3D point in the point cloud corresponding to the current pixel coordinates;

[0043] Step 3: Using a learning-based foreground matting technique, the foreground of the RGB image in the image pair is extracted, and a transparency image of the foreground object is generated, such as... Figure 4 As shown.

[0044] Step 4: Determine the spatial neighborhood of the pixel to which the main object belongs based on the transparency image. R and spatial neighborhood weight set W Depth restoration is performed on the main object in the spatial domain to obtain a complete point cloud of the main object. The spatial neighborhood of the pixels to which the main object belongs... R The specific components are as follows:

[0045] First, construct a c*c pixel original spatial neighborhood centered on the current pixel. If elements are stored at the same pixel position in the transparency image... If the neighboring pixel is not found, it is discarded; otherwise, it is retained, thus forming the current pixel's neighborhood. R Neighborhood R The weights of each pixel in the restoration depth The formula is as follows:

[0046]

[0047] Where c is the side length of the original rectangular spatial neighborhood. The preset optimal threshold, The coordinates of the current pixel. The coordinates of the neighboring pixels, Constructing a neighborhood weight set W ;

[0048] Finally, the repair depth of the area where the main object belongs. The specific formula is as follows:

[0049]

[0050] in This represents the depth value corresponding to the neighboring pixels;

[0051] Step 5: Based on prior information such as ensuring no obstruction between the subject and the camera during shooting, and the subject's size, determine the coordinates in the camera coordinate system. The point cloud filtering interval along the axis completes the first stage of coarse filtering for the main point cloud, such as... Figure 5As shown; since the subject captured in the subject point cloud filtering task is known, it is only necessary to make a rough preset of the size of the subject target in the Z-axis direction (i.e., the depth direction) to obtain the specific filtering range of the subject target in the Z-axis direction by combining the content in this step.

[0052] Step 6: A second-stage fine filtering, guided by the segmentation results, is superimposed on the first-stage coarse filtering result. This effectively removes edge noise at discontinuities in the point cloud depth domain, yielding the final filtered subject point cloud result. Since a higher transparency value indicates a more pronounced foreground object color in the current pixel, and vice versa (theoretically, the transparency transition area should appear in the foreground-background separation region), the transparency value is used as the confidence level for belonging to a foreground object. That is, the higher the transparency value, the greater the probability that the pixel belongs to a foreground object, thus filtering out uncertain pixels (i.e., edge noise) in the foreground-background separation region. The final result of the superposition of the two filtering stages is the final filtered subject point cloud, as shown below. Figure 6 As shown.

[0053] In this embodiment, the camera intrinsic parameters in step 2 are calibrated by capturing multiple images of a complete planar chessboard calibration board using an RGBD camera, and then calculating them using Zhang Zhengyou's planar calibration method. This calibration process only needs to be performed once for any RGBD camera used.

[0054] In this embodiment, the specific steps of step 3 are as follows:

[0055] The foreground segmentation network is used to segment the foreground of the acquired RGB image. The network output is a transparency map S with the same resolution as the input image and is aligned pixel by pixel with the original image. The element a stored at each pixel position represents the degree of blending between the foreground and the background at the current pixel.

[0056] In this embodiment, the specific steps of step 5 are as follows:

[0057] First, based on the prior condition that there are no other objects obstructing the view between the subject and the camera, the coordinates of all points in the original point cloud are traversed. Value, extract the minimum depth value Then the point containing the current minimum depth value is the point in the main object that is closest to the camera coordinate system xoy plane;

[0058] Next, based on the prior size information b of the current main object in the Z direction, the following coarse filtering formula is obtained:

[0059]

[0060] in, This is the point cloud after coarse filtering; This refers to any point in the original cloud. This represents the Z-coordinate value of the current point;

[0061] In this embodiment, the specific steps of step 6 are as follows:

[0062] The transparency values ​​at each pixel location in the transparency map S are iterated. To ensure accurate removal of edge noise, the transparency value 'a' is used as the basis for the coarse filtering result. As an indicator of whether something belongs to the foreground object, fine filtering is performed. The specific formula is as follows:

[0063]

[0064] in, The preset optimal threshold, This is the point cloud after fine filtering; is the projection function, representing the two-dimensional pixel in the image corresponding to the three-dimensional point in the point cloud; express The transparency value of the projection point corresponding to the point;

[0065] The above description is merely illustrative of the invention and its embodiments, but the scope of protection of the invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art without departing from the spirit of the invention should be included within the scope of protection of the invention. Therefore, the scope of protection of the invention should be determined by the scope of the claims.

Claims

1. A two-stage subject point cloud filtering method based on an RGBD camera, characterized in that, The method includes the following specific steps: Step 1: Read the RGBD camera image data containing the photographed subject to obtain an image pair composed of an aligned RGB image and a depth image; Step 2: According to the camera intrinsic parameter data and the aligned RGB and depth image pair, restore the pixel points to the three-dimensional space to generate an original three-dimensional point cloud map to be filtered in the camera coordinate system. The mapping formula between the image and the point cloud is: where (u, v) are the pixel coordinates on the depth image or the RGB image, K is the camera intrinsic parameter matrix, X, Y, and Z are the point cloud coordinates in the camera coordinate system to be restored, and P is the three-dimensional point coordinate in the point cloud corresponding to the current pixel coordinate; Step 3: Adopt a learning-based foreground matting technique to extract the foreground of the RGB image in the image pair and generate a transparency image of the foreground object; Step 4: Based on the transparency image, determine the spatial neighborhood R of the pixels belonging to the main object and the spatial neighborhood weight set W, and perform depth restoration on the main object in the spatial domain to obtain a complete main object point cloud. The composition of the spatial neighborhood R of the pixels belonging to the main object is as follows: First, construct an original spatial neighborhood of c*c pixels centered on the current pixel. If the element α stored at the same pixel position in the transparency image is less than l, the neighborhood pixels lower than the threshold l are discarded, otherwise they are retained. Finally, the current pixel neighborhood R is formed. The formula for the weight w(i, j) contributed by each pixel in the neighborhood R to the restored depth is as follows: Where c is the side length of the original rectangular spatial neighborhood, l is the preset optimal threshold, (x,y) are the coordinates of the current pixel, (i,j) are the coordinates of the neighboring pixels, and α (i，j) Let w(i,j) be the transparency value at pixel coordinates (i,j) in the transparency image, and w(i,j) constitute the neighborhood weight set W. Finally, the repair depth D of the area to which the main object belongs. * The specific formula for (x, y) is as follows: where D(i, j) is the depth value corresponding to the neighborhood pixel; Step 5: Based on the prior information of no occlusion between the main object and the camera and the size of the main object during shooting, determine the point cloud filtering interval in the Z-axis direction of the camera coordinate system to complete the first-stage coarse filtering of the main object point cloud; Step 6: Superimpose the second-stage fine filtering indicated by the transparency image on the result of the first-stage coarse filtering to obtain the final main object point cloud filtering result.

2. The two-stage subject point cloud filtering method based on an RGBD camera according to claim 1, characterized in that, The specific steps of Step 3 are as follows: Use a foreground matting network to perform foreground segmentation on the obtained RGB image. Its network output is a transparency map S with the same resolution as the input image and is pixel-aligned with the original image. The element α stored at each pixel position represents the mixing degree of the current pixel foreground and background.

3. The two-stage subject point cloud filtering method based on an RGBD camera according to claim 2, characterized in that, The specific steps of Step 5 are as follows: First, based on the prior condition that there are no other objects obstructing the view between the subject and the camera, the minimum depth value D is extracted by iterating through the Z values ​​of all point coordinates in the original point cloud. min Then the point containing the current minimum depth value is the point in the main object that is closest to the camera coordinate system xoy plane; Secondly, according to the prior information β of the size of the current main object in the Z direction, the following coarse filtering formula can be obtained: Where C represents the point cloud after coarse filtering; p represents any point in the original point cloud; p z This represents the Z-coordinate value of the current point.

4. The two-stage subject point cloud filtering method based on an RGBD camera according to claim 3, characterized in that, The specific steps of Step 6 are as follows: Traverse the transparency values at each pixel position in the transparency map S. To ensure the accurate removal of edge noise, perform fine filtering in the result of the coarse filtering with the transparency value α≥l as an indication of whether it belongs to the foreground object. The specific formula is as follows: Among them, C * This is the finely filtered point cloud; Projection(·) is the projection function, representing the two-dimensional pixel in the image corresponding to the three-dimensional point in the point cloud; Projection(p) α This represents the transparency value of the projection point corresponding to point p.

Images