Three-dimensional point cloud denoising method, device, computer-readable storage medium and electronic device based on normal vector

Abstract

The present invention is applicable to the field of image processing, and provides a three-dimensional point cloud denoising method, device, computer-readable storage medium and electronic device based on normal vectors. The method includes: acquiring a depth image and a corresponding three-dimensional point cloud; obtaining multiple second three-dimensional point clouds according to the depth image; obtaining the normal vector of the second three-dimensional point cloud according to the second three-dimensional point cloud; calculating the ray from the center of the visible camera to each three-dimensional point in the second three-dimensional point cloud; calculating the angle between the normal vector of the second three-dimensional point cloud and the ray from the center of the visible camera to each three-dimensional point in the second three-dimensional point cloud; judging whether the angle is greater than a preset angle threshold, and if so, deleting the corresponding three-dimensional point in the three-dimensional point cloud. The present invention adopts a three-dimensional point cloud denoising method based on normal vectors, and performs filtering processing on the three-dimensional point cloud corresponding to the depth image, which can effectively and quickly eliminate radial noise in the three-dimensional point cloud, obtain a cleaner three-dimensional point cloud, and improve the accuracy of the point cloud for three-dimensional reconstruction.

Description

Technical Field

[0001] The present invention belongs to the field of image processing, and in particular relates to a three-dimensional point cloud denoising method, device, computer-readable storage medium and electronic device based on normal vectors. Background Art

[0002] A depth image, also known as a range image, is an image that uses the distance (depth value) from the image collector to each point in the scene as a pixel value. It directly reflects the geometric shape of the visible surface of the scene. The depth image can be calculated as point cloud data after coordinate transformation, and point cloud data with rules and necessary information can also be inversely calculated into depth image data.

[0003] At present, the methods for acquiring depth images include laser radar depth imaging, computer stereo vision imaging, coordinate measuring machine method, moiré fringe method, structured light method, etc. No matter which method is used to obtain the depth image, it contains a lot of noise. Generally, a filtering algorithm is used for denoising. However, traditional depth filtering mainly includes guided filtering and bilateral filtering. Both types of filtering use RGB images as guides to filter the depth map. They are not effective in eliminating radial noise in three-dimensional point clouds (radial noise is point cloud noise that is radially scattered on the rays emitted by the optical center of the camera), and the speed is relatively slow. Summary of the invention

[0004] The purpose of the present invention is to provide a three-dimensional point cloud denoising method, device, computer-readable storage medium and electronic device based on normal vectors, aiming to solve the problem that traditional deep filtering methods are not effective in eliminating radial noise in three-dimensional point clouds and are slow.

[0005] In a first aspect, the present invention provides a three-dimensional point cloud denoising method based on normal vectors, the method comprising:

[0006] Get the depth image and the corresponding 3D point cloud;

[0007] Obtain a plurality of second three-dimensional point clouds according to the depth image;

[0008] Obtaining a normal vector of the second three-dimensional point cloud according to the second three-dimensional point cloud;

[0009] Calculate the ray from the center of the visible camera to each three-dimensional point in the second three-dimensional point cloud;

[0010] Calculate the angle between the normal vector of the second three-dimensional point cloud and the ray from the center of the visible camera to each three-dimensional point in the second three-dimensional point cloud;

[0011] Determine whether the angle is greater than a preset angle threshold, and if so, delete the corresponding three-dimensional point in the three-dimensional point cloud;

[0012] There is a one-to-one correspondence between each pixel point on the depth image and a three-dimensional point on the three-dimensional point cloud.

[0013] In combination with the first aspect, in a possible implementation manner, obtaining a plurality of second three-dimensional point clouds according to the depth image specifically includes:

[0014] Performing super-pixel segmentation on the depth image to obtain N super-pixel blocks, where N≥2;

[0015] The three-dimensional point corresponding to each pixel in each superpixel block is obtained in turn to obtain multiple second three-dimensional point clouds.

[0016] In combination with the first aspect, in a possible implementation manner, obtaining a normal vector of the second three-dimensional point cloud according to the second three-dimensional point cloud specifically includes:

[0017] Performing principal component analysis (PCA) on the second three-dimensional point cloud, and taking the eigenvector with the smallest eigenvalue of the covariance matrix as the normal vector of the second three-dimensional point cloud;

[0018] In combination with the first aspect, in a possible implementation manner, the visible camera is located at the center of the second three-dimensional point cloud.

[0019] In a second aspect, the present invention provides a three-dimensional point cloud denoising device based on a normal vector, the device comprising:

[0020] Acquisition module: used to obtain depth images and corresponding three-dimensional point clouds;

[0021] A first processing module: used for obtaining a plurality of second three-dimensional point clouds according to the depth image;

[0022] A second processing module: configured to obtain a normal vector of the second three-dimensional point cloud according to the second three-dimensional point cloud;

[0023] A first calculation module: used for calculating rays from the center of the visible camera to each three-dimensional point in the second three-dimensional point cloud;

[0024] A second calculation module: used to calculate the angle between the normal vector of the second three-dimensional point cloud and the ray from the center of the visible camera to each three-dimensional point in the second three-dimensional point cloud;

[0025] The third processing module is used to determine whether the angle is greater than a preset angle threshold, and if so, delete the corresponding three-dimensional point in the three-dimensional point cloud.

[0026] In conjunction with the second aspect, in a possible implementation manner, the first processing module specifically includes:

[0027] Segmentation unit: used to perform super-pixel segmentation on the depth image to obtain N super-pixel blocks, where N≥2;

[0028] Generation unit: used to sequentially obtain the three-dimensional point corresponding to each pixel in the superpixel block to obtain a second three-dimensional point cloud.

[0029] In a third aspect, the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the three-dimensional point cloud denoising method based on normal vectors as described in the first aspect.

[0030] In a fourth aspect, the present invention provides an electronic device comprising: one or more processors; a memory; and one or more computer programs, wherein the processor and the memory are connected via communication, wherein the one or more computer programs are stored in the memory and are configured to be executed by the one or more processors, and when the processor executes the computer program, the steps of the normal vector-based three-dimensional point cloud denoising method as described in the first aspect are implemented.

[0031] In the present invention, a normal vector-based 3D point cloud denoising method is adopted to filter the 3D point cloud corresponding to the depth image, which can effectively and quickly eliminate the radial noise in the 3D point cloud (radial noise is point cloud noise radially distributed on the rays emitted by the camera optical center), obtain a cleaner 3D point cloud, and improve the point cloud accuracy of 3D reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a flow chart of a three-dimensional point cloud denoising method based on normal vectors provided in Example 1 of the present invention.

[0033] Figure 2 It is a specific structural diagram of a normal vector-based three-dimensional point cloud denoising device provided in Embodiment 2 of the present invention.

[0034] Figure 3 It is a specific structural block diagram of an electronic device provided in Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solution and beneficial effects of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0036] In order to better understand the present application scheme, the relevant terms and concepts are first introduced below.

[0037] (1) Superpixel segmentation: The concept of superpixel is an image segmentation technology proposed and developed by Xiaofeng Ren in 2003. It refers to irregular pixel blocks with certain visual significance composed of adjacent pixels with similar texture, color, brightness and other characteristics. It groups pixels by using the similarity of features between pixels, and uses a small number of superpixels instead of a large number of pixels to express image features, which greatly reduces the complexity of image post-processing. Therefore, it is usually used as a preprocessing step of the segmentation algorithm. It has been widely used in computer vision applications such as image segmentation, pose estimation, target tracking, and target recognition.

[0038] (2) Principal Component Analysis (PCA) is a statistical method that transforms a set of possibly correlated variables into a set of linearly uncorrelated variables through orthogonal transformation. The transformed set of variables is called principal components.

[0039] (3) Depth image: The difference between a depth image and a grayscale image is that the information contained in the pixels is different. The pixels in a depth image contain the three-dimensional information of each point on the surface of an object or in a scene, that is, the actual distance from the point's position in reality to the vertical plane where the camera scanner is located. In addition, there is a one-to-one correspondence between the pixels in the depth image and each pixel in the RGB image taken by the camera.

[0040] (4) 3D point cloud: a data set containing the 3D information and color information of each point on the surface of an object or scene photographed by the camera;

[0041] In order to illustrate the technical solution of the present invention, a specific embodiment is provided below for illustration.

[0042] Embodiment 1:

[0043] See also Figure 1 The three-dimensional point cloud denoising method based on normal vectors provided in the first embodiment of the present invention comprises the following steps:

[0044] S101, obtaining a depth image and a corresponding three-dimensional point cloud;

[0045] In the first embodiment of the present invention, the method for acquiring the depth image may include one or more of the following common methods: lidar depth imaging method, computer stereoscopic vision imaging, coordinate measuring machine method, moiré fringe method, structured light method, etc.; of course, the depth image here can also be stored in a memory, and the processor can obtain the depth image data from the memory; of course, the depth image here can also be directly obtained by a depth camera.

[0046] In the first embodiment of the present invention, the three-dimensional point cloud corresponding to the depth image can be directly acquired by laser radar, and of course, it can also be obtained by converting the acquired depth image. The specific conversion idea is: first combine the two-dimensional coordinates, depth value and camera intrinsic parameters of the pixel point in the depth image to generate the three-dimensional coordinates of the pixel point in the camera coordinate system, and then combine the external parameters of the camera to generate the three-dimensional coordinates in the world coordinate system. For details, please refer to the following:

[0047] Given the two-dimensional coordinates (u, v), depth value d, and camera intrinsic parameter K of a pixel in the depth image, the three-dimensional coordinates P of the pixel in the camera coordinate system that captured the depth image can be generated by the relevant formula. n ;

[0048] Then, the camera external parameters (generally the camera's rotation matrix and translation matrix) are combined to obtain the three-dimensional coordinates P of the pixel point in the world coordinate system through relevant formulas.

[0049] In the first embodiment of the present invention, there is a one-to-one correspondence between each pixel point on the depth image and a three-dimensional point on the three-dimensional point cloud.

[0050] , obtaining a plurality of second three-dimensional point clouds according to the depth image;

[0051] In the first embodiment of the present invention, S102 may specifically be:

[0052] S1021, performing super-pixel segmentation on the depth image to obtain N super-pixel blocks, where N≥2;

[0053] S1022. Obtain the three-dimensional point corresponding to each pixel in the superpixel block in sequence to obtain a second three-dimensional point cloud.

[0054] In the first embodiment of the present invention, the superpixel segmentation method in S1021 may adopt a commonly used superpixel segmentation method, for example, SLIC, Ncut and other algorithms.

[0055] It should be understood that in the first embodiment of the present invention, the superpixel segmentation method in S1021 can also be replaced by other segmentation methods. It is only necessary to ensure that the size of each sub-region in the segmentation result is as uniform as possible and the boundary of the sub-region respects the boundary information of the original image.

[0056] , obtaining a normal vector of the second three-dimensional point cloud according to the second three-dimensional point cloud;

[0057] In the first embodiment of the present invention, obtaining a normal vector of the second three-dimensional point cloud according to the second three-dimensional point cloud in S103 specifically includes:

[0058] Performing principal component analysis (PCA) on the second three-dimensional point cloud, and taking the eigenvector with the smallest eigenvalue of the covariance matrix as the normal vector of the second three-dimensional point cloud;

[0059] The principal component analysis (PCA) is performed on the second three-dimensional point cloud, and the eigenvector with the smallest eigenvalue of the covariance matrix is ​​taken as the normal vector of the second three-dimensional point cloud. Specifically, the following steps can be used to implement the PCA:

[0060] 1) Centralize the second three-dimensional point cloud data;

[0061] 2) Calculate the covariance matrix of the second three-dimensional point cloud data;

[0062] 3) Perform eigenvalue decomposition on the covariance matrix;

[0063] 4) Take out the eigenvector corresponding to the smallest eigenvalue and use it as the normal vector of the superpixel block.

[0064] , calculating rays from the center of the visible camera to each three-dimensional point in the second three-dimensional point cloud;

[0065] In the first embodiment of the present invention, the visual camera in S104 is specifically:

[0066] The visible camera is located at the center of the second three-dimensional point cloud.

[0067] , calculating the angle between the normal vector of the second three-dimensional point cloud and the ray from the center of the visible camera to each three-dimensional point in the second three-dimensional point cloud.

[0068] In the first embodiment of the present invention, in S105, the angle between the normal vector of the second three-dimensional point cloud and the ray from the center of the visible camera to each three-dimensional point in the second three-dimensional point cloud is calculated, which may be specifically:

[0069] In the second 3D point cloud, there are m (m ≥ 2) 3D points. The normal vector of the second 3D point cloud is P, and the rays from the center of the visible camera to each 3D point are M1, M2, M3, ... M m , calculate M1, M2, M3...M respectively m The angles between P and θ1, θ2, θ3...θ m , the angle can be any angle value.

[0070] , determining whether the angle is greater than a preset angle threshold, and if so, deleting the corresponding three-dimensional point in the three-dimensional point cloud;

[0071] In the first embodiment of the present invention, the preset angle threshold in S106 may be any angle. Generally speaking, the smaller the preset angle threshold is, the more point clouds are filtered out. However, considering the effect, it is recommended that the preset angle threshold is greater than 85°.

[0072] In the first embodiment of the present invention, in S106, it is determined whether the angle is greater than a preset angle threshold, and if so, the corresponding three-dimensional point in the three-dimensional point cloud is deleted, specifically:

[0073] If the angle θ i (i=1, 2, 3...m) is greater than a preset threshold, the corresponding three-dimensional point i in the second three-dimensional point cloud is removed from the original three-dimensional point cloud to obtain a new three-dimensional point cloud.

[0074] In the first embodiment of the present invention, after deleting the three-dimensional points that do not meet the conditions, a new three-dimensional point cloud can be obtained, and the new three-dimensional point cloud can improve the point cloud accuracy of three-dimensional reconstruction.

[0075] Embodiment 2:

[0076] Figure 2 It shows that a three-dimensional point cloud denoising device 100 based on normal vectors is provided in the second embodiment of the present invention, and the device comprises:

[0077] 101 acquisition module: used to acquire a depth image and a corresponding three-dimensional point cloud;

[0078] 102 A first processing module: configured to obtain a plurality of second three-dimensional point clouds according to the depth image;

[0079] 103 a second processing module: configured to obtain a normal vector of the second three-dimensional point cloud according to the second three-dimensional point cloud;

[0080] 104 A first calculation module: used to calculate rays from the center of the visible camera to each three-dimensional point in the second three-dimensional point cloud;

[0081] 105 A second calculation module: used to calculate the angle between the normal vector of the second three-dimensional point cloud and the ray from the center of the visible camera to each three-dimensional point in the second three-dimensional point cloud;

[0082] 106 A third processing module: used to determine whether the angle is greater than a preset angle threshold, and if so, delete the corresponding three-dimensional point in the three-dimensional point cloud.

[0083] In the second embodiment of the present invention, the first processing module specifically includes:

[0084] Segmentation unit: used to perform super-pixel segmentation on the depth image to obtain N super-pixel blocks, where N≥2;

[0085] Generation unit: used to sequentially obtain the three-dimensional point corresponding to each pixel in the superpixel block to obtain a second three-dimensional point cloud.

[0086] Those skilled in the art should understand that each module of the above-mentioned three-dimensional point cloud denoising device based on normal vectors can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor of any electronic device in the form of hardware, or can be stored in the memory of any electronic device in the form of software, so that the processor can call and execute the operations corresponding to the above modules. The electronic device here can be a computer, a mobile phone, an IPAD, a server, a computer, a smart watch, smart glasses, a camera, etc.

[0087] Embodiment three:

[0088] Embodiment 3 of the present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the three-dimensional point cloud denoising method based on normal vectors provided in Embodiment 1 of the present invention are implemented.

[0089] Embodiment 4:

[0090] Figure 3 A specific structural block diagram of an electronic device provided in Embodiment 4 of the present invention is shown. The electronic device here may be a computer, a mobile phone, an IPAD, a server, a computer, a smart watch, smart glasses, a camera, etc. An electronic device 200 includes: one or more processors 201, a memory 202, and one or more computer programs, wherein the processor 201 and the memory 202 are connected via communication or a bus, the one or more computer programs are stored in the memory 202, and are configured to be executed by the one or more processors 201, and when the processor 201 executes the computer program, the steps of the three-dimensional point cloud denoising method based on normal vectors provided in Embodiment 1 of the present invention are implemented.

[0091] In the present invention, a normal vector-based 3D point cloud denoising method is adopted to filter the 3D point cloud corresponding to the depth image, which can effectively and quickly eliminate the radial noise in the 3D point cloud (radial noise is point cloud noise radially distributed on the rays emitted by the camera optical center), obtain a cleaner 3D point cloud, and improve the point cloud accuracy of 3D reconstruction.

[0092] A person skilled in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0093] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A three-dimensional point cloud denoising method based on normal vectors, characterized in that: The method comprises: Get the depth image and the corresponding original 3D point cloud; Obtaining a plurality of second three-dimensional point clouds according to the depth image specifically includes: Performing super-pixel segmentation on the depth image to obtain N super-pixel blocks, where N≥2; Sequentially obtain the three-dimensional point corresponding to each pixel in each superpixel block to obtain multiple second three-dimensional point clouds; Obtaining a normal vector of the second three-dimensional point cloud according to the second three-dimensional point cloud; Calculate the ray from the center of the visible camera to each three-dimensional point in the second three-dimensional point cloud; Calculate the angle between the normal vector of the second three-dimensional point cloud and the ray from the center of the visible camera to each three-dimensional point in the second three-dimensional point cloud; Determine whether the angle is greater than a preset angle threshold, and if so, delete the corresponding three-dimensional point in the original three-dimensional point cloud; There is a one-to-one correspondence between each pixel point on the depth image and a three-dimensional point on the original three-dimensional point cloud.

2. The method according to claim 1, characterized in that Obtaining a normal vector of the second three-dimensional point cloud according to the second three-dimensional point cloud specifically includes: A principal component analysis (PCA) is performed on the second three-dimensional point cloud, and an eigenvector with the smallest eigenvalue of the covariance matrix is ​​taken as the normal vector of the second three-dimensional point cloud.

3. A three-dimensional point cloud denoising device based on normal vectors, characterized in that: The device comprises: Acquisition module: used to obtain depth images and corresponding original 3D point clouds; The first processing module is used to obtain a plurality of second three-dimensional point clouds according to the depth image, specifically comprising: Segmentation unit: used to perform super-pixel segmentation on the depth image to obtain N super-pixel blocks, where N≥2; A generating unit: used to sequentially obtain the three-dimensional point corresponding to each pixel point in the superpixel block to obtain multiple second three-dimensional point clouds; A second processing module: configured to obtain a normal vector of the second three-dimensional point cloud according to the second three-dimensional point cloud; A first calculation module: used for calculating rays from the center of the visible camera to each three-dimensional point in the second three-dimensional point cloud; A second calculation module: used to calculate the angle between the normal vector of the second three-dimensional point cloud and the ray from the center of the visible camera to each three-dimensional point in the second three-dimensional point cloud; The third processing module is used to determine whether the angle is greater than a preset angle threshold, and if so, delete the corresponding 3D point in the original 3D point cloud; There is a one-to-one correspondence between each pixel point on the depth image and a three-dimensional point on the original three-dimensional point cloud.

4. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the normal vector-based three-dimensional point cloud denoising method as described in any one of claims 1 to 2 are implemented.

5. An electronic device, comprising: one or more processors; Memory; and one or more computer programs, the processor and the memory are connected via communication or a bus, wherein the one or more computer programs are stored in the memory and are configured to be executed by the one or more processors, characterized in that when the processor executes the computer program, the steps of the normal vector-based three-dimensional point cloud denoising method as described in any one of claims 1 to 2 are implemented.

Images