A ToF depth image denoising method based on a graph Laplacian regularized network

Abstract

This application provides a ToF depth image denoising method based on a Graph Laplacian Regularized Network (GLR). The method includes: acquiring raw data from a ToF sensor; removing noise from the raw data using a GLR to obtain a denoised depth map; and post-processing the denoised depth map to remove regions with confidence scores below a threshold, resulting in a final depth map. This approach utilizes the GLR to achieve better denoising performance and does not require actual data acquisition for training; it can be trained directly using simulated Gaussian noise, exhibiting good generalization ability.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, and specifically relates to a ToF depth image denoising method based on a graph Laplacian regularization network. Background Technology

[0002] 3D imaging technology has broad application prospects and commercial value in fields such as mobile robotics, telemedicine, and 3D television. Accurate acquisition of depth information is a crucial technology essential for 3D research and practical applications. Therefore, fast and high-quality depth sensors are highly sought after. Among various depth acquisition technologies (binocular, structured light, LiDAR, etc.), Time-of-Flight (ToF) cameras have become an efficient, low-cost, and versatile depth imaging solution.

[0003] Noise exists in depth images, originating from various processes such as image acquisition, compression, and transmission. Image denoising is the process of reducing noise in an image. The principle followed by image denoising algorithms is to preserve as much detail and edge information as possible from the original image while separating noise, that is, to separate useful information from noisy data in the image.

[0004] Existing denoising methods do not have adaptive denoising intensity adjustment. In addition, existing methods using neural networks for denoising have limited generalization ability and require the acquisition of actual data. Summary of the Invention

[0005] The purpose of the embodiments in this specification is to provide a ToF depth image denoising method based on a graph Laplacian regularization network.

[0006] To solve the above-mentioned technical problems, the embodiments of this application are implemented in the following ways:

[0007] In a first aspect, this application provides a ToF depth image denoising method based on a Graph Laplacian Regularization Network, the method comprising:

[0008] Acquire raw data collected by the ToF sensor;

[0009] The raw data is used to remove noise based on a graph Laplacian regularization network to obtain a denoised depth map.

[0010] The denoised depth map is then processed by removing regions with confidence scores below a threshold to obtain the final depth map.

[0011] In one embodiment, the raw data is single-frequency raw data;

[0012] The raw data is used to remove noise using a graph Laplacian regularization network to obtain a denoised depth map, including:

[0013] Single-frequency raw data is converted into corresponding IQ data;

[0014] IQ data is denoised using a graph Laplacian regularization network to obtain denoised IQ data;

[0015] The denoised IQ data is converted into a denoised depth map.

[0016] In one embodiment, single-frequency raw data is converted into corresponding IQ data using the following formula:

[0017]

[0018] Among them, V k This refers to the output signal corresponding to different phase shifts k = 1, 2, 3... N for single-frequency raw data.

[0019] In one embodiment, the IQ data is denoised using a graph Laplacian regularization network to obtain denoised IQ data, including:

[0020] IQ data is input into a feature extraction network to obtain feature maps;

[0021] Determine the graph adjacency matrix based on the feature map;

[0022] Determine the Laplace matrix based on the graph adjacency matrix;

[0023] IQ data is input into a pre-filtering network to obtain pre-filtered IQ data;

[0024] IQ data is input into a coefficient prediction network to obtain predicted coefficients;

[0025] The denoised IQ data is determined based on the Laplacian matrix, the pre-filtered IQ data, and the prediction coefficients.

[0026] In one embodiment, the denoised IQ data is determined based on the Laplacian matrix, the pre-filtered IQ data, and the prediction coefficients, including:

[0027] Equations are constructed based on the Laplacian matrix, pre-filtered IQ data, prediction coefficients, and denoised IQ data.

[0028] The equations are solved using a QP solver to obtain the denoised IQ data.

[0029] In one embodiment, the denoised IQ data is converted into a denoised depth map, including:

[0030] Convert the denoised IQ data into phase data;

[0031] The depth of the denoised depth map is calculated based on the phase data.

[0032] In one embodiment, the denoised depth map is post-processed to remove regions with confidence levels below a threshold, resulting in a final depth map, including:

[0033] Determine the confidence level based on the denoised IQ data;

[0034] Regions with confidence scores below the threshold are set as invalid values ​​to obtain the final depth map.

[0035] In one embodiment, if the raw data is multi-frequency raw data, the multi-frequency raw data includes at least low-frequency raw data and high-frequency raw data;

[0036] The raw data is used to remove noise using a graph Laplacian regularization network to obtain a denoised depth map, including:

[0037] Low-frequency raw data is converted into corresponding low-frequency IQ data, and high-frequency raw data is converted into corresponding high-frequency IQ data.

[0038] Low-frequency IQ data is denoised using a graph Laplacian regularization network to obtain denoised low-frequency IQ data, and high-frequency IQ data is denoised using a graph Laplacian regularization network to obtain denoised high-frequency IQ data.

[0039] After denoising, low-frequency IQ data is converted into a low-frequency denoised depth map, and high-frequency IQ data is converted into a high-frequency denoised depth map.

[0040] In one embodiment, the denoised depth map is post-processed to remove regions with confidence levels below a threshold, resulting in a final depth map, including:

[0041] The low-frequency denoising depth map and the high-frequency denoising depth map are fused by phase unfolding to obtain a fused depth map;

[0042] The fused depth map is then processed by removing regions with confidence scores below a threshold to obtain the final depth map.

[0043] In a second aspect, this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the ToF depth image denoising method based on a graph Laplacian regularization network as described in the first aspect.

[0044] As can be seen from the technical solutions provided in the embodiments of this specification above, this solution can effectively remove noise from depth images and effectively alleviate the contradiction between noise reduction and detail preservation. In addition, although this solution is based on a deep learning network, it can still effectively avoid overfitting problems and achieve robust noise reduction effect in the absence of training data. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 A flowchart illustrating the ToF depth image denoising method based on a graph Laplacian regularization network provided in this application;

[0047] Figure 2 A schematic diagram illustrating the denoising of the single-frequency raw data provided in this application;

[0048] Figure 3 A schematic diagram illustrating the denoising of the multi-frequency raw data provided in this application;

[0049] Figure 4 A schematic diagram of noise removal based on a graph Laplacian regularization network for the raw data provided in this application;

[0050] Figure 5 is a schematic diagram comparing the depth maps after denoising using the method of this application and the libfreenect denoising method;

[0051] Figure 6 A schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0052] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0053] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0054] Various modifications and variations can be made to the specific embodiments described in this application without departing from the scope or spirit of this application, as will be apparent to those skilled in the art. Other embodiments derived from this application will be obvious to those skilled in the art. This application specification and embodiments are merely exemplary.

[0055] The terms “include,” “including,” “have,” “contain,” etc., used in this article are all open-ended terms, meaning that they include but are not limited to.

[0056] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0057] Reference Figure 1 It shows a flowchart of the ToF depth image denoising method based on graph Laplacian regularization network provided in the embodiments of this application.

[0058] like Figure 1 As shown, a ToF depth image denoising method based on a Graph Laplacian Regularized Network (Graph Laplacian Regularized Network) may include:

[0059] S110. Acquire raw data collected by the ToF sensor;

[0060] S120 and raw data are used to remove noise based on a graph Laplacian regularization network to obtain a denoised depth map;

[0061] S130. The denoised depth map is processed by removing regions with confidence scores below a threshold to obtain the final depth map.

[0062] Raw data can include single-frequency raw data and multi-frequency raw data. For single-frequency raw data, such as... Figure 2 As shown, a graph Laplacian regularization network is used for noise removal, followed by post-processing to obtain the final depth map. For multi-frequency raw data, such as... Figure 3 As shown, each frequency data point is denoised using a graph Laplacian regularization network, and then fused and post-processed to obtain the final depth map. The following sections describe the denoising of depth maps for single-frequency raw data and multi-frequency raw data.

[0063] In one embodiment of this application, if the raw data is single-frequency raw data, continue to refer to... Figure 2 Step S120 involves removing noise from the raw data using a graph Laplacian regularization network to obtain a denoised depth map, which may include:

[0064] Single-frequency raw data is converted into corresponding IQ data;

[0065] IQ data is denoised using a graph Laplacian regularization network to obtain denoised IQ data;

[0066] The denoised IQ data is converted into a denoised depth map.

[0067] The single-frequency raw data is converted into the corresponding IQ data using the following formula:

[0068]

[0069] Among them, V k This is the output signal corresponding to different phase shifts k = 1, 2, 3...N for single-frequency raw data. For example, N = 3.

[0070] To denoise noisy IQ data, a graph Laplacian regularization network is used, such as... Figure 4 As shown.

[0071] In one embodiment of this application, the IQ data is denoised using a graph Laplacian regularization network to obtain denoised IQ data, including:

[0072] IQ data is input into a feature extraction network to obtain feature maps;

[0073] Determine the graph adjacency matrix based on the feature map;

[0074] Determine the Laplace matrix based on the graph adjacency matrix;

[0075] IQ data is input into a pre-filtering network to obtain pre-filtered IQ data;

[0076] IQ data is input into a coefficient prediction network to obtain predicted coefficients;

[0077] The denoised IQ data is determined based on the Laplacian matrix, the pre-filtered IQ data, and the prediction coefficients.

[0078] The denoised IQ data is determined based on the Laplacian matrix, the pre-filtered IQ data, and the prediction coefficients, including:

[0079] Equations are constructed based on the Laplacian matrix, pre-filtered IQ data, prediction coefficients, and denoised IQ data.

[0080] The equations are solved using a QP solver to obtain the denoised IQ data.

[0081] For details, please refer to... Figure 4 Noisy IQ data is input into a feature extraction network to obtain feature maps F. n n = 1, ..., N, used for graph model construction. Specifically, the graph neighbor matrix A is determined based on the feature map using the following formula:

[0082]

[0083] Among them, A ij Let be the element in the i-th row and j-th column of the graph neighbor matrix A.

[0084] The Laplace matrix L is calculated using the following formula:

[0085] L = diag(A1) - A

[0086] Where 1 is a vector with all elements l, and diag represents converting the vector into a diagonal matrix, where the elements on the diagonal are vector elements.

[0087] Simultaneously, the IQ data will pass through a pre-filtering network to obtain pre-filtered IQ data I′Q′. Furthermore, the IQ data will also pass through a coefficient prediction network to obtain the prediction coefficients μ needed for the denoising process.

[0088] Finally, the following equation is solved using the QP solver to obtain the denoised IQ data I. * Q * .

[0089] (E+μL)I * =I′,(E+μL)Q * =Q′

[0090] Where E is the identity matrix.

[0091] From the above, we obtain the denoised IQ data I. * Q * .

[0092] In this application, the feature extraction network, pre-filtering network, and coefficient prediction network all employ convolutional neural networks (CNNs). The networks are trained using a simulated dataset containing Gaussian noise, and the loss function is the mean-square error (MSE).

[0093] In one embodiment of this application, converting denoised IQ data into a denoised depth map may include:

[0094] The denoised IQ data is converted into phase data, and the phase is calculated using the following formula.

[0095]

[0096] The depth d of the denoised depth map is calculated based on the phase data:

[0097]

[0098] Where f is the frequency of the signal emitted by the ToF sensor, and c is the speed of light.

[0099] In one embodiment of this application, step S130, which involves post-processing the denoised depth map to remove regions with confidence levels less than a threshold to obtain a final depth map, may include:

[0100] Determine the confidence level based on the denoised IQ data;

[0101] Regions with confidence scores below the threshold are set as invalid values ​​to obtain the final depth map.

[0102] Specifically, the confidence level C is calculated using the following formula:

[0103] C = abs(I * )+abs(Q * )

[0104] Calculate the confidence level C. The lower the value C, the lower the confidence level of that point. By setting a threshold, some points with low confidence are set as invalid values ​​and removed. That is, it should satisfy:

[0105] C ij >T

[0106] Where T is a pre-set threshold, which can be changed as needed.

[0107] In one embodiment of this application, if the raw data is multi-frequency raw data, the multi-frequency raw data includes at least low-frequency raw data and high-frequency raw data. This application uses dual-frequency data as an example, meaning the multi-frequency raw data includes low-frequency raw data and high-frequency raw data. (Continue to refer to...) Figure 3 Step S120 involves removing noise from the raw data using a graph Laplacian regularization network to obtain a denoised depth map, which may include:

[0108] Low-frequency raw data is converted into corresponding low-frequency IQ data, and high-frequency raw data is converted into corresponding high-frequency IQ data.

[0109] Low-frequency IQ data is denoised using a graph Laplacian regularization network to obtain denoised low-frequency IQ data, and high-frequency IQ data is denoised using a graph Laplacian regularization network to obtain denoised high-frequency IQ data.

[0110] After denoising, low-frequency IQ data is converted into a low-frequency denoised depth map, and high-frequency IQ data is converted into a high-frequency denoised depth map.

[0111] The corresponding step S130, denoising the depth map, involves post-processing to remove regions with confidence levels below a threshold, resulting in the final depth map. This process may include:

[0112] The low-frequency denoising depth map and the high-frequency denoising depth map are fused by phase unfolding to obtain a fused depth map;

[0113] The fused depth map is then processed by removing regions with confidence scores below a threshold to obtain the final depth map.

[0114] Specifically, unlike single-frequency raw data, dual-frequency raw data protects both high and low frequency data. This solution processes the data of different frequencies separately, and the processing method is the same as that of single-frequency data. First, the raw data collected by the ToF sensor is converted into the corresponding IQ data. Then, a graph Laplacian regularization network is used to denoise it and convert it into the corresponding denoised depth map.

[0115] To obtain the final depth map, the depth maps corresponding to different frequencies are fused by phase unwrapping to obtain a fused depth map. Finally, the fused depth map is post-processed to remove low-confidence regions (i.e. regions with confidence less than the threshold) and set them as invalid values.

[0116] Experimental comparison

[0117] As shown in Figure 5 (including...) Figures 5(a)-5(c) As shown in Figure 5(a), using data collected by the ToF sensor in the Kinectv2 camera as input, the sensor contains a large amount of noise before denoising due to sensor noise and ambient light effects (as shown in Figure 5(a)). After denoising by the libfreenect method, the noise is removed to a certain extent, but the noise that cannot be removed is still clearly observable (as shown in Figure 5(b)). After denoising by the ToF depth image denoising method based on graph Laplacian regularization network provided in this application, the noise can be removed to a greater extent, and the preservation of details is similar to that of the contrast method (as shown in Figure 5(c)).

[0118] The Time-of-Flight (ToF) depth image denoising method provided in this application utilizes a graph Laplacian regularized network to achieve better denoising results. Furthermore, it eliminates the need for actual data acquisition for training, allowing direct training using simulated Gaussian noise and exhibiting excellent generalization ability. In contrast, existing technologies suffer from limited generalization capabilities and rely heavily on actual data acquisition.

[0119] The ToF depth image denoising method based on a graph Laplacian regularized network provided in this application utilizes a coefficient prediction network of the graph Laplacian regularized network to adjust the denoising intensity, adapting to the denoising strength requirements of different frequencies. Existing technologies, however, do not possess adaptive denoising intensity adjustment.

[0120] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Figure 6The diagram shows a structural schematic of an electronic device 600 suitable for implementing embodiments of this application.

[0121] like Figure 6 As shown, the electronic device 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage section 608 into a random access memory (RAM) 603. The RAM 603 also stores various programs and data required for the operation of the device 600. The CPU 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0122] The following components are connected to I / O interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to I / O interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 610 as needed so that computer programs read from it can be installed into storage section 608 as needed.

[0123] In particular, according to embodiments of this disclosure, the above references Figure 1 The described process can be implemented as a computer software program. For example, embodiments of this disclosure include a computer program product comprising a computer program tangibly embodied on a machine-readable medium, the computer program containing program code for performing the above-described ToF depth image denoising method based on a Graph Laplacian Regularized Network. In such embodiments, the computer program can be downloaded and installed from a network via communication section 609, and / or installed from removable medium 611.

[0124] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0125] The units or modules described in the embodiments of this application can be implemented in software or hardware. The described units or modules can also be located in a processor. The names of these units or modules do not necessarily constitute a limitation on the unit or module itself.

[0126] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, a laptop computer, a mobile phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or any combination of these devices.

[0127] It should be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0128] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

Claims

1. A ToF depth image denoising method based on a Graph Laplacian Regularized Network, characterized in that, The method includes: Acquire raw data collected by the ToF sensor; If the raw data is single-frequency raw data, the single-frequency raw data is converted into the corresponding IQ data using the following formula: in, For single-frequency raw data corresponding to different phase shifts k=1,2,3…N The output signal at that time; The raw data is subjected to noise removal based on a graph Laplacian regularization network to obtain a denoised depth map, including: The single-frequency raw data is converted into corresponding IQ data; The IQ data is denoised using a graph Laplacian regularization network to obtain denoised IQ data. The denoised IQ data is converted into the denoised depth map; If the raw data is multi-frequency raw data, the multi-frequency raw data includes at least low-frequency raw data and high-frequency raw data; The raw data is subjected to noise removal based on a graph Laplacian regularization network to obtain a denoised depth map, including: The low-frequency raw data is converted into corresponding low-frequency IQ data, and the high-frequency raw data is converted into corresponding high-frequency IQ data. The low-frequency IQ data is denoised using a graph Laplacian regularization network to obtain denoised low-frequency IQ data, and the high-frequency IQ data is denoised using a graph Laplacian regularization network to obtain denoised high-frequency IQ data. The denoised low-frequency IQ data is converted into a low-frequency denoised depth map, and the denoised high-frequency IQ data is converted into a high-frequency denoised depth map. The denoised depth map is then post-processed to remove regions with confidence levels below a threshold, resulting in the final depth map.

2. The method according to claim 1, characterized in that, The IQ data is denoised using a graph Laplacian regularization network to obtain denoised IQ data, including: The IQ data is input into a feature extraction network to obtain a feature map; Based on the feature map, determine the graph adjacency matrix; Determine the Laplace matrix based on the graph adjacency matrix; The IQ data is input into a pre-filtering network to obtain pre-filtered IQ data. The IQ data is input into the coefficient prediction network to obtain the predicted coefficients; The denoised IQ data is determined based on the Laplacian matrix, the pre-filtered IQ data, and the prediction coefficients.

3. The method according to claim 2, characterized in that, The step of determining the denoised IQ data based on the Laplacian matrix, the pre-filtered IQ data, and the prediction coefficients includes: An equation is constructed based on the Laplacian matrix, the pre-filtered IQ data, the prediction coefficients, and the denoised IQ data; The equations are solved using a QP solver to obtain the denoised IQ data.

4. The method according to claim 1, characterized in that, The denoised IQ data is converted into the denoised depth map, including: The denoised IQ data is converted into phase data; The depth of the denoised depth map is calculated based on the phase data.

5. The method according to claim 1, characterized in that, The denoised depth map is post-processed to remove regions with confidence scores below a threshold, resulting in a final depth map, including: The confidence level is determined based on the denoised IQ data; The regions corresponding to confidence levels less than the threshold are set as invalid values ​​to obtain the final depth map.

6. The method according to claim 1, characterized in that, The denoised depth map is post-processed to remove regions with confidence scores below a threshold, resulting in a final depth map, including: The low-frequency denoising depth map and the high-frequency denoising depth map are fused by phase unfolding to obtain a fused depth map; The fused depth map is post-processed to remove regions with confidence levels below a threshold, resulting in the final depth map.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the ToF depth image denoising method based on a graph Laplacian regularization network as described in any one of claims 1-6.

Images