Depth image denoising smoothing optimization method, system and device and medium

Abstract

The embodiment of the invention discloses a depth image denoising smooth optimization method, system and device and a medium, and the method comprises the steps: carrying out the down-sampling processing of an obtained to-be-processed depth image pp and a corresponding guide image II, and obtaining a to-be-processed depth image p and a guide image I after the down-sampling; respectively determining constant coefficients ak and bk corresponding to all position points i in the to-be-processed depth image p and the guide image I after down-sampling in a filtering window Wk; determining a mean value # imgabs0 # according to the constant coefficients ak and bk corresponding to each position point i in the filtering window Wk, and determining ai and bi parameter tables under the original size according to the mean value # imgabs1 # and the depth image pp to be processed; determining a filtered output image q according to a linear relationship between ai and bi under each pixel in the ai and bi parameter table and a point qi of the filtered output image and a point Ii of the guide image; and combining the edge data edgefilter in the obtained depth image pp to be processed with the filtered output image q to obtain a final output image.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method, system, device and medium for deep image denoising and smoothing optimization. Background Art

[0002] In today's era of rapid technological development, 3D imaging technology has been widely used in various fields, from virtual reality to self-driving cars, and accurate depth information has become an indispensable part of these technologies. However, due to various factors, such as ambient light, surface characteristics of objects, etc., the depth map obtained from the 3D camera often has noise and discontinuity, which requires effective filtering technology to smooth the depth map to improve its quality.

[0003] Among the traditional depth map smoothing filters, bilateral filtering, least squares filtering and guided filtering are three commonly used edge-preserving filters. Among them, bilateral filtering and least squares filtering have good filtering and smoothing effects, but their calculations are complex, CPU usage is high, and the operation time is long, making them difficult to apply in real time. Summary of the invention

[0004] Based on this, it is necessary to propose a deep image denoising and smoothing optimization method, system, device and medium to address the above problems.

[0005] An embodiment of the present invention provides a deep image denoising and smoothing optimization method, which is:

[0006] Down-sampling the acquired depth image to be processed pp and the corresponding guide image II to obtain the down-sampled depth image to be processed p and the guide image I;

[0007] Determine respectively all position points i in the downsampled depth image p to be processed and the guide image I within the filter window W k The corresponding constant coefficient a k ,b k ;

[0008] According to each position point i in the filter window W k The corresponding constant coefficient a k ,b k Determine the mean

[0009] According to the mean and the depth image pp to be processed to determine a at the original size i , b i Parameter table;

[0010] According to the a i , b i a under each pixel in the parameter table i, b i and the point q of the filtered output image i and point I of the guide image i The linear relationship determines the filtered output image q;

[0011] The edge data edge_filter in the obtained depth image to be processed pp is combined with the filtered output image q to obtain the final output image.

[0012] In the above scheme, the down-sampled depth image p to be processed and the guide image I are respectively determined to be within the filtering window W k The corresponding constant coefficient a k ,b k , specifically including:

[0013] At any position i on the depth image p to be processed and the guide image I, take point p i and I i ;

[0014] At the point p i and I i As the center, take a 5*5 area as the local filtering window W k ;

[0015] The point q corresponding to the output image i and point I of the guide image i In the filter window W k There is a local linear relationship on : Among them, a k ,b k is the filter window W k Uniquely determined constant coefficient;

[0016] Construct the loss function:

[0017] Solving the loss function using the least squares method yields in, It is window W k The variance of It is window W k The mean value of the depth image p to be processed within the range, ε=0.1.

[0018] In the above scheme, the filtering window W is based on each position point i. k The corresponding constant coefficient a k ,b k Determine the mean Specifically, all the position points i are contained in 25 filtering windows W k In the equation, each location point has 25 constant coefficients a.k ,b k , 25 a k ,b k Find the average and get the a of the current position point i i ,

[0019] In the above scheme, the point q of the output image after filtering is i and point I of the guide image i In the filter window W k There is a local linear relationship on : Among them, a k ,b k is the filter window W k Uniquely determined constant coefficient.

[0020] In the above scheme, according to a i , b i a under each pixel in the parameter table i , b i and the point q of the filtered output image i and Point I i The linear relationship determines the filtered output image q, which specifically includes:

[0021] The a under each pixel i , b i Substitute the point q of the filtered output image i and point I of the guide image i The linear relationship determines all points q in the output image i , and obtain the filtered output image q.

[0022] In the above solution, the step of merging the edge data edge_filter in the depth image to be processed pp with the filtered output image q to obtain the final output image specifically includes:

[0023] Extract edge data of the depth image pp to be processed;

[0024] Perform bilateral filtering and smoothing on the edge data set edge to obtain the filtered edge_filter;

[0025] Merge the edge data edge_filter with the output image q of the fast guided filter to get the final output image.

[0026] In the above solution, the step of extracting edge data of the depth image pp to be processed specifically includes: within the filtering window, when When , the data is stored in the edge data set edge, and the pixels at the same position in the filtered output image q are set to 0;

[0027] Traverse the pixel points of the depth image p to be processed. If the depth value of the adjacent points differs by 10 pixels, classify this point as an edge point, add the pixel points of the 5*5 area centered on this pixel point to the edge data set edge, and set the same position points of the output image q to 0.

[0028] The embodiment of the present invention also provides a real-time deep image denoising and smoothing optimization system, which includes a downsampling unit, a filtering unit, an output unit

[0029] The down-sampling unit is used to perform down-sampling processing on the acquired depth image to be processed pp and the corresponding guide image II to obtain the down-sampled depth image to be processed p and the guide image I;

[0030] The filtering unit is used to determine the position points i in the downsampled depth image p to be processed and the guide image I in the filtering window W respectively. k The corresponding constant coefficient a k ,b k ; It is also used to calculate the filter window W according to each position point i k The corresponding constant coefficient a k ,b k Determine the mean a i , b i ; Also used according to the mean a i , b i and the depth image pp to be processed to determine a at the original size i , b i Parameter table;

[0031] The output unit is used to i , b i a under each pixel in the parameter table i , b i and the point q of the filtered output image i and Point I i The linear relationship of determines the filtered output image q; and is also used to merge the edge data edge_filter in the obtained depth image pp to be processed with the filtered output image q to obtain the final output image.

[0032] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor executes the following steps:

[0033] Down-sampling the acquired depth image to be processed pp and the corresponding guide image II to obtain the down-sampled depth image to be processed p and the guide image I;

[0034] Determine respectively all position points i in the downsampled depth image p to be processed and the guide image I within the filter window W k The corresponding constant coefficient a k ,b k ;

[0035] According to each position point i in the filter window W k The corresponding constant coefficient a k ,b k Determine the mean a i , b i ;

[0036] According to the mean a i , b i and the depth image pp to be processed to determine a at the original size i , b i Parameter table;

[0037] According to the a i , b i a under each pixel in the parameter table i , b i and the point q of the filtered output image i and point I of the guide image i The linear relationship determines the filtered output image q;

[0038] The edge data edge_filter in the obtained depth image to be processed pp is combined with the filtered output image q to obtain the final output image.

[0039] An embodiment of the present invention further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the following steps:

[0040] Down-sampling the acquired depth image to be processed pp and the corresponding guide image II to obtain the down-sampled depth image to be processed p and the guide image I;

[0041] Determine respectively all position points i in the downsampled depth image p to be processed and the guide image I within the filter window W k The corresponding constant coefficient a k ,b k ;

[0042] According to each position point i in the filter window W k The corresponding constant coefficient a k ,b k Determine the mean a i , b i ;

[0043] According to the mean a i , b i and the depth image pp to be processed to determine a at the original size i , b i Parameter table;

[0044] According to the a i , b i a under each pixel in the parameter table i , b i and the point q of the filtered output image i and point I of the guide image i The linear relationship determines the filtered output image q;

[0045] The edge data edge_filter in the obtained depth image to be processed pp is combined with the filtered output image q to obtain the final output image.

[0046] The embodiments of the present invention have the following beneficial effects:

[0047] The present invention adopts a method of extracting the edge of the object separately for secondary processing. The extraction of the edge part is divided into two parts: one is to extract the edge of the object separately by determining the variance value of the filter; the second is to determine that the pixel values ​​of adjacent positions differ by 10 pixels, and then determine it as an edge area. After the edge area is extracted, the edge area is smoothed using bilateral filtering, and then merged with the image after fast guided filtering to obtain the final output image q. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0049] in:

[0050] Figure 1 A flowchart of a deep image denoising and smoothing optimization method is provided for an embodiment of the present invention;

[0051] Figure 2 A qualitative comparison diagram of a deep image denoising and smoothing optimization method before and after optimization is provided for an embodiment of the present invention. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] The embodiment of the present invention provides a deep image denoising and smoothing optimization method, such as Figure 1 As shown, the method includes:

[0054] Step 101: down-sampling the acquired depth image to be processed pp and the corresponding guide image II to obtain the down-sampled depth image to be processed p and the guide image I;

[0055] Step 102: Determine the position of all the positions i in the downsampled depth image p to be processed and the guide image I in the filter window W. k The corresponding constant coefficient a k ,b k ;

[0056] Specifically, at any position i on the depth image to be processed p and the guide image I, take point p i and I i ;

[0057] At the point p i and I i As the center, take a 5*5 area as the local filtering window W k ;

[0058] The point q corresponding to the output image i and point I of the guide image i In the filter window W k There is a local linear relationship on : Among them, a k ,b k is the filter window W k Uniquely determined constant coefficient;

[0059] The linear relationship between the output image and the guidance image ensures that in each local window W k In the above example, if there is an edge in the guidance image I, the output image q will keep the edge unchanged.

[0060] At the same time, the output image q should be as similar as possible to the depth image p to be processed to reduce the information loss caused by filtering. The least squares representation of the algorithm is:

[0061] Introducing a regularization parameter ε to prevent a kToo large, and the loss function is:

[0062] Solving the loss function using the least squares method yields in, It is window W k The variance of It is window W k The mean value of the depth image p to be processed within the range, ε=0.1.

[0063] At the same time, the non-edge area and non-smooth area in the depth image to be processed p is a noise point n i , in the filter window W k The output image q after filtering i and the depth image p to be processed i There are the following relationships:

[0064] Step 103: Based on each position point i in the filtering window W k The corresponding constant coefficient a k ,b k Determine the mean

[0065] Specifically, all the position points i are contained in 25 filtering windows W k In the equation, each location point has 25 constant coefficients a. k ,b k , 25 a k ,b k Find the average and get the current position point i

[0066] Step 104: According to the mean and the depth image pp to be processed to determine a at the original size i , b i Parameter table;

[0067] Step 105: According to the a i , b i a under each pixel in the parameter table i , b i and the point q of the filtered output image i and point I of the guide image i The linear relationship determines the filtered output image q;

[0068] Specifically, the point q of the filtered output image i and point I of the guide image i In the filter window W k There is a local linear relationship on : Among them, a k ,b k is the filter window W k Uniquely determined constant coefficient.

[0069] The a under each pixel i , b i Substitute the point q of the filtered output image i and point I of the guide image i The linear relationship determines all points q in the output image i , and obtain the filtered output image q.

[0070] Step 106: Merge the edge data edge_filter in the depth image to be processed pp with the filtered output image q to obtain a final output image.

[0071] Specifically, edge data of the depth image pp to be processed is extracted;

[0072] Perform bilateral filtering and smoothing on the edge data set edge to obtain the filtered edge_filter;

[0073] Merge the edge data edge_filter with the output image q of the fast guided filter to get the final output image.

[0074] The step of extracting edge data of the depth image pp to be processed specifically includes: in a fast guided filtering algorithm, if the area within a certain filtering window is relatively smooth, the variance will be much smaller than ε. Thus a k ≈0, On the contrary, if the region is a marginal region, the variance is large. will be much larger than ε, so a k ≈1, b k ≈0. Therefore, when When , the data is stored in the edge data set edge, and the pixels at the same position in the filtered output image q are set to 0;

[0075] In addition, since some edge parts in the depth cannot be well identified in some cases during calculation, in the present invention, the pixels of the input image are traversed. If the depth values ​​of adjacent points differ by 10 pixels, this point is classified as an edge point, and the pixels in a 5*5 area centered on this pixel point are added to the edge data set edge, and the points at the same position of q are set to 0.

[0076] In the present invention, the depth image pp to be processed and the guide image II are set to be the same image. At this time, the fast guide filter is an edge-preserving filter, and the edge part of the depth image will be kept as much as possible. In order to speed up the processing speed, the fast guide filter uses the method of downsampling to calculate the parameters and then upsampling, which leads to the edge part of the object in the image. There are large areas that are not smoothed or the smoothing effect is relatively poor. In order to make the edge part of the object also be well processed, the present invention adopts the method of extracting the edge of the object separately and performing secondary processing; the extraction of the edge part is divided into two parts: one is to extract the edge of the object separately by determining the variance value of the filter; the second is to determine that the pixel values ​​of adjacent positions differ by 10 pixels, and then determine it as an edge area; after the edge area is extracted, the edge area is smoothed using bilateral filtering, and then merged with the image after the fast guide filter to obtain the final output image q.

[0077] In order to verify the effect of the present invention, the present invention uses qualitative and quantitative methods to analyze the effects before and after optimization respectively.

[0078] The experimental method is: use a 3D camera to collect plane depth maps at a distance of 1000mm / 1500mm / 2000mm / 2500mm / 3000mm, conduct simulation experiments on the collected plane data, and compare them with the data before optimization. Among them, the quantitative analysis adopts the calculation of the point rate, which is specifically implemented as follows: fit the point cloud plane, set the point cloud with an error of 10mm as the internal point, and calculate the percentage of the number of internal points in the total valid points.

[0079] Table 1

[0080]

[0081]

[0082] It can be seen from Table 1 that the point-earning rate after optimization of the present invention is greatly improved compared with the point-earning rate before optimization.

[0083] Depend on Figure 2 It can be seen that the plane fluctuation amplitude after optimization of the present invention is significantly smaller than the fluctuation amplitude before optimization.

[0084] The embodiment of the present invention also provides a real-time deep image denoising and smoothing optimization system, which includes a downsampling unit, a filtering unit, an output unit

[0085] The down-sampling unit is used to perform down-sampling processing on the acquired depth image to be processed pp and the corresponding guide image II to obtain the down-sampled depth image to be processed p and the guide image I;

[0086] The filtering unit is used to determine the position points i in the downsampled depth image p to be processed and the guide image I in the filtering window W respectively. k The corresponding constant coefficient a k ,b k ; It is also used to calculate the filter window W according to each position point i k The corresponding constant coefficient a k ,b k Determine the mean Also used according to the mean and the depth image pp to be processed to determine a at the original size i , b i Parameter table;

[0087] The output unit is used to i , b i a under each pixel in the parameter table i , b i and the point q of the filtered output image i and point I of the guide image i The linear relationship determines the filtered output image q; it is also used to merge the edge data edge_filter in the obtained depth map with the filtered output image q to obtain the final output image.

[0088] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor performs the following steps:

[0089] Step 101: down-sampling the acquired depth image to be processed pp and the corresponding guide image II to obtain the down-sampled depth image to be processed p and the guide image I;

[0090] Step 102: Determine the position of all the positions i in the downsampled depth image p to be processed and the guide image I in the filter window W. k The corresponding constant coefficient a k ,b k ;

[0091] Step 103: Based on each position point i in the filtering window W k The corresponding constant coefficient a k ,b k Determine the mean

[0092] Step 104: According to the mean and the depth image pp to be processed to determine a at the original size i , b i Parameter table;

[0093] Step 105: According to the a i , b i a under each pixel in the parameter table i , b i and the point q of the filtered output image i and point I of the guide image i The linear relationship determines the filtered output image q;

[0094] Step 106: Merge the edge data edge_filter in the obtained depth map with the filtered output image q to obtain a final output image.

[0095] An embodiment of the present invention further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the following steps:

[0096] Step 101: down-sampling the acquired depth image to be processed pp and the corresponding guide image II to obtain the down-sampled depth image to be processed p and the guide image I;

[0097] Step 102: Determine the position of all the positions i in the downsampled depth image p to be processed and the guide image I in the filter window W. k The corresponding constant coefficient a k ,b k ;

[0098] Step 103: Based on each position point i in the filtering window W k The corresponding constant coefficient a k ,b k Determine the mean

[0099] Step 104: According to the mean and the depth image pp to be processed to determine a at the original size i , b i Parameter table;

[0100] Step 105: According to the a i , b i a under each pixel in the parameter table i , b i and the point q of the filtered output image i and point I of the guide image i The linear relationship determines the filtered output image q;

[0101] Step 106: Merge the edge data edge_filter in the obtained depth map with the filtered output image q to obtain a final output image.

[0102] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0103] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0104] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A deep image denoising and smoothing optimization method, characterized in that: The method is: Down-sampling the acquired depth image to be processed pp and the corresponding guide image II to obtain the down-sampled depth image to be processed p and the guide image I; Determine respectively all position points i in the downsampled depth image p to be processed and the guide image I within the filter window W k The corresponding constant coefficient a k ,b k ; According to each position point i in the filter window W k The corresponding constant coefficient a k ,b k Determine the mean According to the mean and the depth image pp to be processed to determine a at the original size i , b i Parameter table; According to the a i , b i a under each pixel in the parameter table i , b i and the point q of the filtered output image i and point I of the guide image i The linear relationship determines the filtered output image q; The edge data edge_filter in the obtained depth image to be processed pp is combined with the filtered output image q to obtain the final output image.

2. The depth image denoising and smoothing optimization method according to claim 1, characterized in that: The method of respectively determining the positions of all the position points i in the downsampled depth image p to be processed and the guide image I within the filtering window W k The corresponding constant coefficient a k ,b k , specifically including: At any position i on the depth image p to be processed and the guide image I, take point p i and I i ; At the point p i and I i As the center, take a 5*5 area as the local filtering window W k ; The point q corresponding to the output image i and point I of the guide image i In the filter window W k There is a local linear relationship on q i =a k I i +b k , Among them, a k ,b k is the filter window W k Uniquely determined constant coefficient; Construct the loss function: Solving the loss function using the least squares method yields in, It is window W k The variance of It is window W k The mean value of the depth image p to be processed within the range, ε=0.

1.

3. The depth image denoising and smoothing optimization method according to claim 1 or 2, characterized in that: According to each position point i in the filtering window W k The corresponding constant coefficient a k ,b k Determine the mean Specifically include: All the position points i are contained in the 25 filter windows W k In the equation, each location point has 25 constant coefficients a. k ,b k , 25 a k ,b k Find the average and get the current position point i 4. The depth image denoising and smoothing optimization method according to claim 3, characterized in that: The point q of the filtered output image i and point I of the guide image i In the filter window W k There is a local linear relationship on q i =a k I i +b k , Among them, a k ,b k is the filter window W k Uniquely determined constant coefficient.

5. The depth image denoising and smoothing optimization method according to claim 4, characterized in that: According to the a i , b i a under each pixel in the parameter table i , b i and the point q of the filtered output image i and Point I i The linear relationship determines the filtered output image q, which specifically includes: The a under each pixel i , b i Substitute the point q of the filtered output image i and Point I i The linear relationship determines all points q in the output image i , and obtain the filtered output image q.

6. The depth image denoising and smoothing optimization method according to claim 5, characterized in that: The step of merging the edge data edge_filter in the depth image to be processed pp with the filtered output image q to obtain the final output image specifically includes: Extract edge data of the depth image pp to be processed; Perform bilateral filtering and smoothing on the edge data set edge to obtain the filtered edge_filter; Merge the edge data edge_filter with the output image q of the fast guided filter to get the final output image.

7. The depth image denoising and smoothing optimization method according to claim 6, characterized in that: The step of extracting edge data of the depth image pp to be processed specifically includes: within the filtering window, when When , the data is stored in the edge data set edge, and the pixels at the same position in the filtered output image q are set to 0; Traverse the pixel points of the depth image p to be processed. If the depth value of the adjacent points differs by 10 pixels, classify this point as an edge point, add the pixel points of the 5*5 area centered on this pixel point to the edge data set edge, and set the same position points of the output image q to 0.

8. A real-time deep image denoising and smoothing optimization system, characterized in that: The system includes a downsampling unit, a filtering unit, and an output unit; The down-sampling unit is used to perform down-sampling processing on the acquired depth image to be processed pp and the corresponding guide image II to obtain the down-sampled depth image to be processed p and the guide image I; The filtering unit is used to determine the position points i in the downsampled depth image p to be processed and the guide image I in the filtering window W respectively. k The corresponding constant coefficient a k ,b k ; It is also used to calculate the filter window W according to each position point i k The corresponding constant coefficient a k ,b k Determine the mean Also used according to the mean and the depth image pp to be processed to determine a at the original size i , b i Parameter table; The output unit is used to i , b i a under each pixel in the parameter table i , b i and the point q of the filtered output image i and Point I i The linear relationship determines the filtered output image q; It is also used to merge the edge data edge_filter in the obtained depth image to be processed pp with the filtered output image q to obtain the final output image.

9. A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 7.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 7.

Images