A method for denoising multi-view ToF depth measurement combined with RGB pictures

Abstract

The application discloses a kind of multi-view ToF depth measurement denoising methods of joint RGB picture, comprising:1 utilizes RGB-D camera to obtain the imaging result under the measurement scene multiple views, 2 calculate the corresponding camera light of each pixel point in imaging result, and 3D coordinate is sampled on camera light, 3 the density value of each coordinate point is predicted by neural network, radiation value, infrared intensity value and normal direction, 4 the imaging result under the rendering of each camera light under multipath interference is obtained by the prediction result of neural network, 5 the imaging result of rendering is constructed loss function training network with the imaging result of acquisition, 6 generate depth measurement data removing multipath interference effect using the network of training.This application removes the noise caused by multipath interference in ToF imaging process by the imaging result of multiple views joint RGB picture, to obtain more accurate depth measurement data and overcome the disadvantage of needing a large amount of real depth data as supervision.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer vision, and particularly relates to a method for removing noise caused by multipath interference in the imaging process of a ToF camera through multi-view RGB-D pictures. BACKGROUND

[0002] In recent years, Time-of-Fight (TOF) based RGB-D camera modules have been widely used in mobile devices. They provide a reliable way of measuring depth data. Compared with structured light cameras or binocular imaging systems, TOF cameras provide more accurate depth data in a short distance range.

[0003] TOF devices calculate the depth of a geometric scene by emitting modulated infrared light to the scene and calculating the measurement on the sensor with different phase shifts. However, ToF devices are affected by multipath interference (MPI): the signal of a single pixel point is composed of multiple light reflection path signals, which can cause errors when obtaining depth information, thereby reducing the application range of the TOF camera. In order to eliminate the influence of MPI as much as possible, most previous work uses additional measures to increase the accuracy of the obtained signal, such as encoding the probe light signal or using multiple modulation frequencies with different phase shifts, which can eliminate the error caused by multipath effects, but this requires hardware changes (such as modifying the built-in infrared light emitter, using a sensor that can receive multiple modulation frequencies), or using multiple scans of the same standard ToF camera.

[0004] Due to the rapid development of deep learning in recent years, more and more researchers want to solve the multipath effect through deep learning, so researchers have begun to use deep learning methods to solve the error problem in ToF imaging, which is very dependent on the data set used for training. However, this method requires a large amount of real depth data as supervision, and a model can only be used on a single type of camera, which is not universal. SUMMARY

[0005] The present application is to solve the above-mentioned problems in the prior art that a large amount of real depth map data is required as supervision when denoising ToF, and it is only applicable to a single type of camera. A multi-view ToF depth measurement denoising method combined with RGB pictures is proposed, which can remove the noise caused by multipath interference in the ToF imaging process through multi-view imaging results combined with RGB pictures, thereby obtaining more accurate depth measurement data, and overcoming the disadvantage of requiring a large amount of real depth data as supervision.

[0006] To achieve the above-mentioned application purposes, the present application adopts the following technical solutions: TOF devices calculate the depth of a geometric scene by emitting modulated infrared light to the scene and calculating the measurement on the sensor with different phase shifts. However, ToF devices are affected by multipath interference (MPI): the signal of a single pixel point is composed of multiple light reflection path signals, which can cause errors when obtaining depth information, thereby reducing the application range of the TOF camera. In order to eliminate the influence of MPI as much as possible, most previous work uses additional measures to increase the accuracy of the obtained signal, such as encoding the probe light signal or using multiple modulation frequencies with different phase shifts, which can eliminate the error caused by multipath effects, but this requires hardware changes (such as modifying the built-in infrared light emitter, using a sensor that can receive multiple modulation frequencies), or using multiple scans of the same standard ToF camera.

[0007] The present invention provides a multi-view ToF depth measurement and noise reduction method for combining RGB images, characterized by the following steps:

[0008] Step 1: Obtain N sets of RGB images and ToF phase measurement images using the calibrated and aligned RGB-D imaging system {I n ,P n |n=1,2,…,N}, where I n Let P represent the nth RGB image. n This represents the nth ToF phase measurement image;

[0009] The nth RGB image I n The pixel in the i-th column and j-th row is denoted as in, Represents the nth RGB image I n The R value of the pixel in the i-th column and j-th row. Represents the nth RGB image I n The G value of the pixel in the i-th column and j-th row. Represents the nth RGB image I n The B value of the pixel in the i-th column and j-th row;

[0010] The nth ToF phase measurement image P n The pixel in the i-th column and j-th row is denoted as in, P represents the nth ToF phase measurement image. n The sinusoidal measurement component of the pixel in the i-th column and j-th row. P represents the nth ToF phase measurement image. n The cosine measurement component of the pixel in the i-th column and j-th row;

[0011] Step 2: Take the camera optical center of the nth image as the origin o. n , take the origin o n The direction pointing to the pixel (i,j) in the i-th column and j-th row is denoted as Therefore, using equation (1), we can obtain the value from the origin o. n A ray passing through pixel (i,j) As camera light:

[0012]

[0013] In equation (1), x represents a ray. Any point on the origin o n The distance between them; and:

[0014] o n =-t n (2)

[0015]

[0016] In formula (2) and formula (3), K represents a camera intrinsic parameter; R n represents a rotation matrix of a camera pose E n under the n-th group of images; t n represents a translation vector of the camera pose E n under the n-th group of images, n = 1, 2, …, N;

[0017] Step 3, using the hierarchical sampling method to sample A position points from the ray :

[0018] Step 3.1, set the sampling interval as [x near , x far ], and uniformly divide [x near , x far ] into A interval blocks; wherein x near represents the nearest distance between the sampling point and the origin o n , and x far represents the farthest distance between the sampling point and the origin o n ;

[0019] Step 3.2, randomly sample a sample x a from the a-th block interval, wherein x a represents the distance between the current sampling position point and the origin o n , and has:

[0020]

[0021] In formula (2), x represents a uniform distribution; and U represents a uniform distribution.

[0022] Step 3.3, substitute the sample x a into formula (1) to obtain the a-th 3D coordinate point

[0023] Step 3.4, according to the process of step 3.2-step 3.3, obtain each 3D coordinate point of the A interval and constitute a 3D coordinate point set

[0024] Step 4, construct a multi-layer perception network , and each layer adopts ReLU as an activation function; and input the a-th 3D coordinate point into the multi-layer perception network , so as to obtain the corresponding density value σ and the radiation value c of the a-th 3D coordinate point a by using formula (5) and formula (6) respectively.a , the infrared intensity value b a , and the normal direction n a :

[0025]

[0026]

[0027] In formula (5) and formula (6), represents the gradient;

[0028] Step 5, respectively, using formula (7), formula (8), formula (9) and formula (10) to calculate the camera light corresponding RGB value ToF intensity value Camera light The distance between the intersection point of the plane and the origin o n Camera light The plane normal vector at the plane intersection point

[0029]

[0030]

[0031]

[0032]

[0033] In formula (7), formula (8), formula (9) and formula (10), c a represents the radiation value of the a th 3D coordinate point b a represents the infrared intensity value of the a th 3D coordinate point x a represents the distance between the a th 3D coordinate point and the origin o n n a represents the normal vector value of the a th 3D coordinate point w a represents the weight of the a th 3D coordinate point And:

[0034] w a = T a (1-exp(-σ a δ a )) (11)

[0035] In formula (11), T a represents the first 3D coordinate point ​The distance between the (a+1)th 3D coordinate point and the (a)th 3D coordinate point a is denoted by formula (12), δ a The distance between the (a+1)th 3D coordinate point and the (a)th 3D coordinate point is denoted by formula (13).

[0036]

[0037] δ a = |x a+1 -x a | (13)

[0038] In formula (13), x a+1 denotes the distance between the (a+1)th 3D coordinate point and the origin o n .

[0039] Step 6, constructing the camera light ray using formula (14)-(16) The reflected light ray at the intersection of the plane:

[0040]

[0041]

[0042]

[0043] In formula (14)-(16), <,> denotes the vector angle cosine value operator; denotes the origin of the reflected ray , and denotes the direction of the reflected ray .

[0044] Step 7, obtaining the reflected light ray using formula (7)-(9) The corresponding RGB value is the distance of the intersection point of the intersection plane and the infrared intensity value , so as to calculate the path MPI of multipath reflection using formula (17):

[0045]

[0046] Step 8, obtaining the RGB measurement value at the pixel point of the i-th column and the j-th row in the n-th RGB image I n and the phase measurement value at the pixel point of the i-th column and the j-th row in the n-th ToF phase measurement image ToF respectively using formula (18) and formula (19) under the setting of multipath interference.

[0047]

[0048]

[0049] In formula (17), λ is the wavelength of the infrared light modulation of the ToF camera;

[0050] Step 9, constructing a multi-layer perception network by constructing formula (20) Loss function of the nth group of images

[0051]

[0052] Step 10, based on the RGB images and ToF phase measurement images of N groups {I n ,P n |n=1,2,…,N}, the multi-layer perception network is trained by using the gradient descent method, and the loss function is calculated to update the network parameters until the loss function converges, thereby obtaining the trained multi-layer perception network for calculating the denoised depth measurement result of any camera light line.

[0053] The electronic device comprises a memory and a processor, and the memory is used to store a program supporting the processor to execute the multi-view ToF depth measurement denoising method, and the processor is configured to execute the program stored in the memory.

[0054] The computer readable storage medium stores a computer program, and the computer program is configured to execute the steps of the multi-view ToF depth measurement denoising method when the processor runs.

[0055] Compared with the prior art, the beneficial effects of the present application are as follows:

[0056] 1. The present application combines the RGB image to denoise the ToF depth measurement result under multi-view, optimizes the depth measurement result of the ToF camera through the depth information obtained by multi-view geometry, removes the noise caused by multipath interference in the imaging process of the ToF camera, and obtains more accurate depth measurement result.

[0057] 2. The present application introduces the RGB image as an auxiliary in the ToF denoising task, and compared with only using the phase image obtained by the ToF camera, the RGB image contains rich texture information, and reliable depth information can be obtained through multi-view geometry to assist the denoising task.

[0058] ​​3. This invention is a self-supervised denoising method that does not require real depth maps as supervision data. Instead, it uses measurement results from different perspectives to supervise each other. Its performance is not limited by the training dataset and has a wider range of applications. Attached Figure Description

[0059] Figure 1 This is a flowchart illustrating the noise reduction process according to an embodiment of the present invention;

[0060] Figure 2 This is a depth map calculated from a ToF phase measurement image;

[0061] Figure 3 This is the denoised depth map of this invention. Detailed Implementation

[0062] In this embodiment, as Figure 1 As shown, a denoising method for multi-view ToF depth measurement results combining RGB images is performed according to the following steps:

[0063] Step 1: Obtain N sets of RGB images and ToF phase measurement images using the calibrated and aligned RGB-D imaging system {I n ,P n |n=1,2,…,N}, where I n Let P represent the nth RGB image. n This represents the nth ToF phase measurement image; Figure 2 The image shown is a depth map calculated from a noisy phase measurement map.

[0064] The nth RGB image I n The pixel in the i-th column and j-th row is denoted as in, Represents the nth RGB image I n The R value of the pixel in the i-th column and j-th row. Represents the nth RGB image I n The G value of the pixel in the i-th column and j-th row. Represents the nth RGB image I n The B value of the pixel in the i-th column and j-th row;

[0065] The nth ToF phase measurement image P n The pixel in the i-th column and j-th row is denoted as in, P represents the nth ToF phase measurement image. n The sinusoidal measurement component of the pixel in the i-th column and j-th row. P represents the nth ToF phase measurement image. n The cosine measurement component of the pixel in the i-th column and j-th row;

[0066] Step 2: Take the camera optical center of the nth image as the origin o. n , take the origin o n The direction pointing to the pixel (i,j) in the i-th column and j-th row is denoted as Therefore, using equation (1), we can obtain the value from the origin o. n A ray passing through pixel (i,j) As camera light:

[0067]

[0068] In equation (1), x represents a ray. Any point on the origin o n The distance between them; and:

[0069] o n =-t n (2)

[0070]

[0071] In equations (2) and (3), K represents the camera intrinsic parameters; R n Represents the camera pose E of the nth image group. n The rotation matrix under; t n Represents the camera pose E of the nth image group. n The translation vector is given by n = 1, 2, ..., N; the camera intrinsic parameters can be obtained by Matlab calibration, and the camera pose can be obtained by inputting N RGB images into COLMAP.

[0072] Step 3: Use stratified sampling to sample from X-rays. Upsampling to 128 locations will result in a more accurate depth value, but it will also increase the training time of the network.

[0073] Step 3.1: Set the sampling interval to [0,10], and divide [0,10] into 128 evenly distributed blocks; where 0 represents the sampling point and the origin o. n The closest distance is 0, and 10 indicates that the sampling point is close to the origin. n The furthest distance is 10;

[0074] Step 3.2: Randomly sample a sample x from the a-th block interval. a , where x a Represents the current sampling position point and the origin o n The distance between them, and:

[0075]

[0076] In equation (2), U represents uniform distribution; and

[0077] Step 3.3, obtaining the 3D coordinate point of the a-th sample x a Substituting into equation (1) to obtain the 3D coordinate point of the a-th

[0078] Step 3.4, obtaining the 3D coordinate point of each interval of 128 intervals according to the process of step 3.2-step 3.3 and forming a 3D coordinate point set

[0079] Step 4, constructing a multi-layer perception network containing 8 fully connected layers Each layer contains 256 nodes and uses ReLU as the activation function; and the 3D coordinate point of the a-th Inputting the multi-layer perception network Thus, the 3D coordinate point of the a-th is obtained by equation (5) and equation (6) respectively The corresponding density value σ a , the radiation value c a , the infrared intensity value b a and the normal direction n a :

[0080]

[0081]

[0082] In equation (5) and equation (6), represents the gradient; in actual operation, it is the partial derivative of the output result σ a , with respect to the input coordinate in the x, y, z three directions respectively;

[0083] Step 5, calculating the camera light corresponding to the RGB value ToF intensity value The intersection point of the camera light passing through the plane and the distance n from the origin o The plane normal vector of the camera light at the intersection point of the plane

[0084]

[0085]

[0086]

[0087]

[0088] in formula (7), formula (8), formula (9) and formula (10), c a represents the radiation value of the a-th 3D coordinate point , b a represents the infrared intensity value of the a-th 3D coordinate point , x a represents the distance of the a-th 3D coordinate point from the origin o n , n a represents the normal vector value of the a-th 3D coordinate point , w a represents the weight of the a-th 3D coordinate point , and has:

[0089] w a = T a (1-exp(-σ a δ a )) (11)

[0090] in formula (11), T a represents the transparency between the 1st 3D coordinate point and the a-th 3D coordinate point , and is obtained by formula (12), δ a represents the distance between the a+1-th 3D coordinate point and the a-th 3D coordinate point , and is obtained by formula (13);

[0091]

[0092] δ a = |x a+1 -x a | (13)

[0093] in formula (13), x a+1 represents the distance of the a+1-th 3D coordinate point from the origin o n ; and δ 128 takes the average value of the distance between sampling points, calculated as

[0094] Step 6, construct the camera light ray at the intersection of the plane reflection light ray:

[0095]

[0096]

[0097]

[0098] In formula (14) to formula (16), <,> represents a vector angle cosine value operator; represents the origin of the reflected ray , represents the direction of the reflected ray ;

[0099] Step 7, the reflected light ray corresponding to the RGB value is obtained by using formula (7) to formula (9) and the distance of the intersection point of the intersection plane is and the infrared intensity value , so as to calculate the path MPI of the multipath reflection by using formula (17):

[0100]

[0101]

[0102] Step 8, the nth RGB image I n under the multipath interference setting is obtained by using formula (18) and formula (19) respectively and the phase measurement value of the pixel point in the i-th column and the j-th row in the nth ToF phase measurement image ToF

[0103]

[0104]

[0105] In formula (17), λ is the wavelength of the infrared light modulation of the ToF camera, and in the present embodiment, λ in the acquisition device used is 16 m;

[0106] Step 9, a multi-layer perception network is constructed by using the construction formula (20) Loss function of the nth group of images

[0107]

[0108] Step 10, based on the N groups of RGB images and ToF phase measurement images {I n , n | n = 1, 2, …, N}, the multi-layer perception network is trained by using the gradient descent method, and the loss function is calculated to update the network parameters until the loss function converges, so as to obtain the trained multi-layer perception network The depth measurement result of any one camera light ray after denoising is calculated. The denoising result is as shown in FIG. 8, so that the noise data in the phase measurement map collected by the ToF camera is removed, and a smoother result is obtained. Figure 3

[0109] In this embodiment, an electronic device includes a memory for storing a program supporting a processor to execute the multi-view ToF depth measurement denoising method described above, and the processor is configured to execute the program stored in the memory.

[0110] In this embodiment, a computer readable storage medium has a computer program stored thereon, and the computer program is run by a processor to execute the steps of the multi-view ToF depth measurement denoising method described above.​

Claims

1. A multi-view ToF depth measurement denoising method combining RGB images, characterized by: Follow these steps: Step 1: Obtain the image using the calibrated and aligned RGB-D imaging system. RGB image and ToF phase measurement image of the group ,in, This represents the nth RGB image. This represents the nth ToF phase measurement image; The nth RGB image The Middle Liede The number of pixels in a row is denoted as ;in, Represents the nth RGB image The Middle Liede row of pixels value, Represents the nth RGB image The Middle Liede row of pixels value, Represents the nth RGB image The Middle Liede row of pixels value; The nth ToF phase measurement image The Middle Liede The number of pixels in a row is denoted as = ;in, This represents the nth ToF phase measurement image. The Middle Liede The sinusoidal measurement component of the row's pixels. This represents the nth ToF phase measurement image. The Middle Liede The cosine measurement component of the row's pixels; Step 2, with the first The camera's optical center in the series of photos serves as the origin. , to the origin Pointing to the Liede row pixels The direction is denoted as Thus, by using equation (1), we can obtain the value from the origin. After pixels A ray As camera light: (1) In equation (1), Represents rays Any point on the origin The distance between them; and: (2) (3) In equations (2) and (3), Indicates camera intrinsic parameters; Indicates the first Camera pose in the series of images The rotation matrix below; Indicates the first Camera pose in the series of images The translation vector below, ; Step 3: Use stratified sampling to sample from X-rays. Upsampling Location points: Step 3.1: Set the sampling interval as and will Evenly divided into There are 1000 interval blocks; among them... Indicates the sampling point and the origin The closest distance, Indicates the sampling point and the origin The farthest distance; Step 3.2: Randomly sample a sample from the a-th block interval. ,in, Represents the current sampling location point and the origin. The distance between them, and: (4) In equation (2), To show obedience; Indicates a uniform distribution; Step 3.3: Prepare the sample Substituting into equation (1) yields the a-th 3D coordinate point. ; Step 3.4: Obtain the results following the process in steps 3.2-3. The 3D coordinate points of each interval are combined to form a set of 3D coordinate points. ; Step 4: Construct a multilayer perceptron network Furthermore, ReLU is used as the activation function for each layer; and the a-th 3D coordinate point is... Input multilayer perceptron network Thus, the a-th 3D coordinate point can be obtained using equations (5) and (6) respectively. Corresponding density value radiation value Infrared intensity value and the direction of the normal : (5) (6) In equations (5) and (6), Represents the gradient; Step 5: Calculate the camera light using equations (7), (8), (9), and (10). Corresponding RGB values ToF intensity value Camera lighting The intersection point of the plane and the origin distance Camera lighting Plane normal vector at the intersection of planes : (7) (8) (9) (10) In equations (7), (8), (9), and (10), Represents the a-th 3D coordinate point The radiation value, Represents the a-th 3D coordinate point The infrared intensity value, Represents the a-th 3D coordinate point and the origin distance, Represents the a-th 3D coordinate point The normal vector value, Represents the a-th 3D coordinate point The weights are: (11) In equation (11), Represents the first 3D coordinate point With the a-th 3D coordinate point The transparency between them is obtained from equation (12). Represents the (a+1)th 3D coordinate point With the a-th 3D coordinate point The distance between them is obtained from equation (13); (12) (13) In equation (13), Represents the (a+1)th 3D coordinate point and the origin The distance; Step 6: Construct camera rays using equations (14)-(16) Reflected light rays at the intersection of the planes: (14) (15) (16) In equations (14)-(16), The operator represents the cosine value of the angle between vectors; Indicates reflected rays The origin, Indicates reflected rays The direction; Step 7: Obtain the reflected light using equations (7)-(9) Corresponding RGB values The distance between the intersection point of the planes and the intersection point is and infrared intensity value Therefore, the path of multipath reflection can be calculated using equation (17). : (17) Step 8: Use equations (18) and (19) to obtain the nth RGB image under the multipath interference setting. The Middle Liede RGB measurement value at the pixel of the row And the nth ToF phase measurement image in the ToF Liede Phase measurement value at the pixel of the row : (18) (19) In equation (19), The wavelength for infrared light modulation in a ToF camera; Step 9: Construct a multilayer perceptron network using constructive (20) method. The Middle Loss function of graph group : (20) Step 10, based on RGB image and ToF phase measurement image of the group The gradient descent method is used to analyze the multilayer perceptron network. Perform training and calculate the loss function. To update network parameters until the loss function The training of the multilayer perceptron network is obtained by continuing until convergence. It is used to calculate the depth measurement result after denoising any camera ray.

2. An electronic device, comprising a memory and a processor, characterized in that, The memory is used to store a program that supports the processor in executing the multi-view ToF depth measurement denoising method of claim 1, and the processor is configured to execute the program stored in the memory.

3. A computer-readable storage medium storing a computer program, characterized in that, The computer program is executed by the processor to perform the steps of the multi-view ToF depth measurement denoising method of claim 1.

Images