A depth map denoising method and system based on gradient correction

Abstract

The application discloses a depth map denoising method and system based on gradient correction, and belongs to the field of image denoising. A complete section of a foreground in a local image is selected, a control point set is selected on the complete section, a section model is constructed, and section gradient information is further calculated; then, the rest of the sections are traversed, the section gradient information of the last section and a dynamic programming algorithm are used to gradually correct the section of the current image, and the current interface gradient information is calculated for the calculation of the next section. The application is used for repairing error areas, and does not process non-error areas. The application can fill the holes in the image, and can also find abnormal points in the image caused by noise, and the abnormal points are repaired according to the gradient information. The section model is calculated only once, and is used for generating initial gradient information. In the process of traversing the image, the gradient information is continuously updated according to the minimum principle of corresponding items of adjacent sections of errors. The local image is only traversed once, and the effect can be achieved, so that the application is a high-efficiency denoising algorithm.

Description

Technical Field

[0001] This invention belongs to the field of image denoising, and more specifically, relates to a depth map denoising method and system based on gradient correction. Background Technology

[0002] With the development of computing demands and devices, the acquisition of object information has gradually shifted from two-dimensional to three-dimensional. In applications such as object recognition and grasping, and autonomous driving, simple RGB images are no longer sufficient. More information is needed, and depth maps can precisely capture the depth information of objects, working together with RGB images to describe the true three-dimensional information of objects. However, depth maps themselves contain noise and errors, which significantly affect the effectiveness of subsequent processing. Therefore, noise reduction of depth images is a necessary and crucial step.

[0003] For scene understanding applications, depth maps are required to be free of missing pixels or holes, with no further requirements on the precision of each pixel. However, for object recognition and grasping, it relies on fitting the target object's point cloud information to a corresponding model to calculate the object's position in 3D space, thus requiring higher precision in the depth map pixels. In terms of error, it can be divided into systematic errors and random errors. Random errors affect the correlation calculations between data, such as data variance, while systematic errors are a global error affecting the accuracy of data indication. For object recognition, it is necessary to perform calculations between data to generate object models, such as the RANSAC algorithm generating a cylindrical model. Therefore, object recognition and grasping require suppressing random errors.

[0004] To address random errors, existing methods often employ filtering techniques, specifically temporal filtering and spatial filtering. Temporal filtering uses information from consecutive frames for filtering, while spatial filtering uses information from the surrounding pixels to correct errors, sometimes also borrowing information from RGB images for joint filtering. However, the distribution characteristics of random errors lie in random time and random location. The aforementioned filtering methods focus on the entire image, and cannot specifically repair errors at their locations, while modifying normal locations and thus generating new errors. Summary of the Invention

[0005] In view of the shortcomings of the prior art, the purpose of this invention is to provide a depth map denoising method and system based on gradient correction, which aims to solve the problem that existing methods for suppressing random errors cannot perform targeted repair.

[0006] To achieve the above objectives, in a first aspect, the present invention provides a depth map denoising method based on gradient correction, the method comprising:

[0007] S1. Obtain a local depth image as the image to be processed, wherein the local depth image corresponds to a target candidate region, and the target is a curved surface with a consistent cross-section;

[0008] S2. Calculate the foreground mask of the image to be processed, multiply the foreground mask by the image to be processed to obtain the foreground of the image to be processed;

[0009] S3. Extract actual data of a complete cross section in the foreground, use the type of surface of the target as prior information, and construct a theoretical cross section model, wherein the cross section is the depth data of a certain row or column of the image;

[0010] S4. Obtain the theoretical data of the cross section based on the theoretical cross section model, repair the actual data of the cross section based on the theoretical data of the cross section, and calculate the gradient information of the cross section;

[0011] S5. Iteratively repair the remaining sections until all sections are repaired: Using the gradient information of the previous section, repair the current section according to the principle of minimizing the positional error between the two sections, and then calculate the gradient information of the current section after repair.

[0012] Preferably, in step S2, a specific search algorithm is used to calculate the foreground mask of the image to be processed. The search algorithm specifically includes:

[0013] Construct a binary image of the same size as the original image as the mask to be calculated, where 0 represents the background and 1 represents the foreground. Starting from the center pixel of the image, traverse outwards. For a given neighboring pixel, if the difference between its depth value and the depth value of the center pixel is less than a given threshold, then the neighboring pixel is considered to be continuous with the center pixel, and the corresponding position of the mask to be calculated is set to 1. The search continues with the neighboring pixel as the new center point. If the difference exceeds the threshold, then the neighboring pixel is considered to be discontinuous with the center pixel, and the corresponding position of the mask to be calculated is set to 0. The above algorithm guarantees that each pixel is traversed only once and terminates after the traversal is completed.

[0014] Preferably, in step S3, the extraction of actual data for a complete cross-section in the foreground is specifically as follows:

[0015] (1) Calculate the angle between the target's orientation in the image and the X-axis based on the foreground mask;

[0016] (2) If the included angle is within the range of [45°, 135°], select a row in the foreground as the cross section of the image; otherwise, select a column in the foreground as the longitudinal section of the image.

[0017] Preferably, in step S3, the step of using the type of surface of the target as prior information to construct a theoretical cross-sectional model is as follows:

[0018] 1) Determine the shape of the selected section based on prior information;

[0019] 2) Based on the cross-sectional shape, select several key points from the selected cross-section to construct a theoretical cross-sectional model containing unknown parameters;

[0020] 3) Solve the theoretical cross-sectional model containing unknown parameters based on the actual data of all key points to obtain the theoretical cross-sectional model.

[0021] Preferably, step S4 includes the following sub-steps:

[0022] S41. Calculate the difference between the theoretical value and the actual value of each pixel in the cross section. If the difference exceeds the set threshold, the pixel is considered an error pixel, and the actual value is updated using the theoretical value of the pixel. Otherwise, no processing is performed.

[0023] S42. Calculate the difference between adjacent pixels of the repaired section to obtain the gradient array of the section.

[0024] Preferably, in step S5, the gradient information from the previous section is used to repair the current section according to the principle of minimizing the positional error between the two sections, as follows:

[0025]

[0026] Among them, X cur [i] represents the value of the current section at pixel i, X pre [i] represents the value of the previous section at pixel i, dis represents the difference between the current section at pixel i and pixel i-1, and dp[i-1] represents the gradient value of the previous section at pixel i-1.

[0027] To achieve the above objectives, in a second aspect, the present invention provides a depth map denoising system based on gradient correction, comprising: a processor and a memory; the memory for storing computer execution instructions; and the processor for executing the computer execution instructions, causing the method described in the first aspect to be executed.

[0028] In summary, the technical solutions conceived by this invention have the following beneficial effects compared with the prior art:

[0029] This invention proposes a depth map denoising method and system based on gradient correction. It selects arbitrary cross-sectional data from the foreground of a local image, chooses a set of control points, constructs a cross-sectional model, and further calculates the cross-sectional gradient information. Then, it traverses the remaining cross-sections, using the gradient information of the previous cross-section and a dynamic programming algorithm to progressively correct the cross-sections of the current image, while simultaneously calculating the gradient information of the current interface for the calculation of the next cross-section. This invention specifically repairs error regions, leaving non-error regions unprocessed. It can fill holes in the image and also detect anomalies caused by noise, repairing them based on gradient information. The cross-sectional model is calculated only once to generate initial gradient information; during image traversal, the gradient information is continuously updated according to the principle of minimizing corresponding terms of adjacent error cross-sections. Achieving the desired effect with only one traversal of the local image is a highly efficient denoising algorithm. Attached Figure Description

[0030] Figure 1 This is a flowchart of a depth map denoising method based on gradient correction provided by the present invention.

[0031] Figure 2 This is a schematic diagram of the aligned RGB and D images provided by the present invention.

[0032] Figure 3 This is a schematic diagram of the depth map before and after repair and the calculated mask provided by the present invention.

[0033] Figure 4 These are the fitted scatter plot and corrected data plot provided by this invention. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0035] like Figure 1 As shown, this invention provides a depth map denoising method based on gradient correction, the method comprising:

[0036] Step S1. Obtain a local depth image as the image to be processed. The local depth image corresponds to the target candidate region, and the target is a curved surface with a consistent cross-section.

[0037] like Figure 2 As shown, based on the intrinsic and extrinsic parameters of the color camera and the depth camera, the depth image is aligned with the color image, and based on the target candidate box in the color image, the corresponding position in the depth image is cropped to generate a local image to be processed, with a size of m×n.

[0038] Step S2. Calculate the foreground mask of the image to be processed, multiply the foreground mask by the image to be processed to obtain the foreground of the image to be processed.

[0039] Calculate a foreground mask to exclude distracting pixel sets.

[0040] Preferably, in step S2, a specific search algorithm is used to calculate the foreground mask of the image to be processed. The search algorithm specifically includes:

[0041] Construct a binary image of the same size as the original image as the mask to be calculated, where 0 represents the background and 1 represents the foreground. Starting from the center pixel of the image, traverse outwards. For a given neighboring pixel, if the difference between its depth value and the center pixel's depth value is less than a given threshold, the neighboring pixel is considered continuous with the center pixel, and the corresponding position in the mask to be calculated is set to 1. The search continues using this neighboring pixel as the new center point. If the difference exceeds the threshold, the neighboring pixel is considered discontinuous with the center pixel, and the corresponding position in the mask to be calculated is set to 0. The algorithm guarantees that each pixel is traversed only once and terminates after the traversal is complete. The obtained mask is shown below. Figure 3 As shown on the right.

[0042] The specific process is as follows:

[0043] (1) Select the center of the image and build a queue to store the coordinates of the center of the image.

[0044] (2) Simultaneously construct two arrays, mask and visited, both of size m×n. mask is the mask to be calculated. A value of 0 indicates the background, and a value of 1 indicates the foreground. All values ​​are initialized to 0. visited records the access status of each pixel to avoid repeated access. A value of 0 indicates that the pixel has not been accessed, and a value of 1 indicates that the pixel has been accessed. Pixels at the edge positions are set to 1, and pixels at other positions are set to 0.

[0045] (3) Traverse the queue, take out the head element in turn, and then traverse the pixels of its eight neighbors, calculating the difference between its depth value and the depth value of the center pixel:

[0046] For a pixel position p in the eight-neighborhood, if p has been visited, determine whether the difference exceeds the threshold. If it is lower than the threshold, set the corresponding position of the mask to 1.

[0047] For a pixel position p in the eight-neighborhood, if p has not been visited, it is also determined whether the difference exceeds the threshold. If it is lower than the threshold, the corresponding position of the mask is set to 1, the corresponding position of the visited is marked, and its coordinates are enqueued into the queue.

[0048] Repeat the above operation until the queue is empty to obtain the mask of the corresponding foreground object.

[0049] This operation aims to avoid the influence of pixel traversal order. As long as a pixel is continuous with any surrounding pixel, it can be included in the mask area, thus avoiding the generation of a large number of small holes.

[0050] Step S3. Extract actual data of a complete cross section in the foreground, use the type of surface of the target as prior information, and construct a theoretical cross section model. The cross section is the depth data of a certain row or column of the image.

[0051] Preferably, in step S3, the complete cross-sectional data should not be missing.

[0052] Preferably, in step S3, the extraction of actual data from a certain cross-section in the foreground is specifically as follows:

[0053] (1) Calculate the angle between the target's orientation in the image and the X-axis based on the foreground mask;

[0054] (2) If the included angle is within the range of [45°, 135°], select a row in the foreground as the cross section of the image; otherwise, select a column in the foreground as the longitudinal section of the image.

[0055] If 45°≤α≤135°, it indicates that the object is close to a "vertical" state. In this case, selecting a row of the image as a cross-section can reflect the cross-sectional properties of the object. Taking a cylinder as an example, the cross-section should be an ellipse; taking a plane as an example, the cross-section should be a straight line. If α is not within the above range, then a column of the image should be selected as a longitudinal section. The following content takes obtaining the image cross-section Xpre as an example.

[0056] Preferably, the formula for calculating the angle between the target's orientation in the image and the X-axis is as follows:

[0057]

[0058] Where α represents the angle between the target's orientation in the image and the X-axis, m represents the height of the foreground mask, and s1 and s2 represent the total number of pixels in the traversal process from the left 1 / 4 and 3 / 4 positions of the foreground mask array to the right, stopping when the corresponding position is 1.

[0059] Preferably, in step S3, the step of using the type of surface of the target as prior information to construct a theoretical cross-sectional model is as follows:

[0060] 1) Determine the shape of the selected section based on prior information;

[0061] 2) Based on the cross-sectional shape, select several key points from the selected cross-section to construct a theoretical cross-sectional model containing unknown parameters;

[0062] 3) Solve the theoretical cross-sectional model containing unknown parameters based on the actual data of all key points to obtain the theoretical cross-sectional model.

[0063] Taking a cylinder as an example, its cross-section is an ellipse. First, select a cross-section Xpre of the foreground object, and then select a set of control points. Since the data terminal is close to the vertex of the cylinder, its gradient is small, and the depth camera has a quantization error, which is generally 1mm. Therefore, the central data segment contains a lot of quantization error and is generally unusable. However, its vertex information is very important. Therefore, the left 1 / 3 data segment, the central data point, and the right 1 / 3 data segment are selected as the set of control points.

[0064] Further, the equation is fitted based on the control point set, and the specific process is as follows:

[0065] First, the general equation of an ellipse is:

[0066] Ax 2 +Bxy+Cy 2 +Dx+Ey+F=0

[0067] The elliptic parameters can be solved by constructing equations using the least squares method.

[0068] Let W = [A, B, C, D, E, F] T X = [x 2 ,xy,y 2 [x,y,1] T The problem can then be solved by constructing the Lagrange function, with the optimization objective being:

[0069] min||W T X|| 2 =W T XX T W

[0070] stW T HW=1

[0071] in

[0072]

[0073] The parameter constraint for the ellipse equation is 4AC-B. 2 >0, here expressed as W T Since HW>0, and the optimization objective is ||W T X|| 2 =0, W can be scaled, therefore the constraint condition is defined as W T HW=1.

[0074] Furthermore, the Lagrangian function can be constructed to solve for W:

[0075] L(W,λ)=WT XX T W-λ(W T HW-1).

[0076] Scatter points of cross-section model fitting Figure 4 As shown on the left.

[0077] Step S4. Obtain the theoretical data of the cross section based on the theoretical cross section model, repair the actual data of the cross section based on the theoretical data of the cross section, and calculate the gradient information of the cross section.

[0078] Preferably, step S4 includes the following sub-steps:

[0079] S41. Calculate the difference between the theoretical value and the actual value of each pixel in the cross section. If the difference exceeds the set threshold, the pixel is considered an error pixel, and the actual value is updated using the theoretical value of the pixel. Otherwise, no processing is performed.

[0080] S42. Calculate the difference between adjacent pixels of the repaired section to obtain the gradient array of the section. Step S5. Iteratively repair the remaining sections until all sections are repaired: using the gradient information of the previous section, repair the current section according to the principle of minimizing the error between corresponding positions of the two sections, and then calculate the gradient information of the current section after repair.

[0081] Preferably, in step S5, the gradient information from the previous section is used to repair the current section according to the principle of minimizing the positional error between the two sections, as follows:

[0082]

[0083] Among them, X cur [i] represents the value of the current section at pixel i, X pre [d] represents the value of the previous section at pixel i, dis represents the difference between the current section at pixel i and pixel i-1, and dp[i-1] represents the gradient value of the previous section at pixel i-1. The correction result is as follows: Figure 4 As shown on the right.

[0084] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A depth map denoising method based on gradient correction, characterized in that, The method includes: S1. Obtain a local depth image as the image to be processed, wherein the local depth image corresponds to a target candidate region, and the target is a curved surface with a consistent cross-section; S2. Calculate the foreground mask of the image to be processed, multiply the foreground mask by the image to be processed to obtain the foreground of the image to be processed; S3. Extract actual data of a complete cross section in the foreground, use the type of surface of the target as prior information, and construct a theoretical cross section model, wherein the cross section is the depth data of a certain row or column of the image; S4. Obtain the theoretical data of the cross section based on the theoretical cross section model, repair the actual data of the cross section based on the theoretical data of the cross section, and calculate the gradient information of the cross section; S5. Iteratively repair the remaining sections until all sections are repaired: Using the gradient information of the previous section, repair the current section according to the principle of minimizing the positional error between the two sections, and then calculate the gradient information of the current section after repair. In step S3, the method of using the type of surface of the target as prior information to construct a theoretical cross-section model is as follows: 1) Determine the shape of the selected cross-section based on the prior information; 2) Select several key points from the selected cross-section based on the cross-section shape to construct a theoretical cross-section model containing unknown parameters; 3) Solve the theoretical cross-section model containing unknown parameters based on the actual data of all key points to obtain the theoretical cross-section model.

2. The method as described in claim 1, characterized in that, In step S2, a specific search algorithm is used to calculate the foreground mask of the image to be processed. The search algorithm is as follows: Construct a binary image of the same size as the original image as the mask to be calculated, where 0 represents the background and 1 represents the foreground. Starting from the center pixel of the image, traverse outwards. For a given neighboring pixel, if the difference between its depth value and the depth value of the center pixel is less than a given threshold, then the neighboring pixel is considered to be continuous with the center pixel, and the corresponding position of the mask to be calculated is set to 1. The search continues with the neighboring pixel as the new center point. If the difference exceeds the threshold, then the neighboring pixel is considered to be discontinuous with the center pixel, and the corresponding position of the mask to be calculated is set to 0. The above algorithm guarantees that each pixel is traversed only once and terminates after the traversal is completed.

3. The method as described in claim 1, characterized in that, In step S3, the extraction of actual data for a complete cross-section in the foreground is specifically as follows: (1) Calculate the angle between the target's orientation in the image and the X-axis based on the foreground mask; (2) If the included angle is at Within the specified range, select a row in the foreground as the cross-section of the image; otherwise, select a column in the foreground as the longitudinal section of the image.

4. The method as described in claim 1, characterized in that, Step S4 includes the following sub-steps: S41. Calculate the difference between the theoretical value and the actual value of each pixel in the cross section. If the difference exceeds the set threshold, the pixel is considered an error pixel, and the actual value is updated using the theoretical value of the pixel. Otherwise, no processing is performed. S42. Calculate the difference between adjacent pixels of the repaired section to obtain the gradient array of the section.

5. The method as described in claim 1, characterized in that, In step S5, the gradient information from the previous section is used to repair the current section according to the principle of minimizing the positional error between the two sections, as follows: in, Indicates the current section in pixels The value, Indicates the previous section at pixel The value, Indicates the current section in pixels and pixels The difference, Indicates the pixel of the previous section. The gradient value.

6. A depth map denoising system based on gradient correction, characterized in that, include: Processor and memory; The memory is used to store computer-executed instructions; The processor is configured to execute the computer execution instructions, causing the method described in any one of claims 1 to 5 to be executed.

Images