Method for denoising a depth image, image processing system and image processing device

Abstract

The application provides a depth image denoising method, an image processing system and an image processing device, and relates to the technical field of image processing. The depth image denoising method greatly reduces the calculation resources consumed by denoising by taking the number of mutations of a single pixel in its adjacent direction as a confidence weight. Meanwhile, the depth image denoising method does not need to be adjusted, and greatly improves the robustness of the denoising method. In addition, noise and edges are both mutations, and the number of mutations in the adjacent direction of the pixel is taken as the confidence weight, so that better noise filtering effect is achieved while more texture details and other information are retained. The method reduces the calculation resources consumed, enhances the robustness of the denoising method, preserves the texture and edges of the image and other detail information, and improves the denoising effect.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method for denoising depth images, an image processing system, and an image processing apparatus. Background Technology

[0002] With the development of portable and affordable depth cameras, depth images are playing an increasingly important role in fundamental research and applications in the field of image processing. By utilizing information from depth images, the performance of related research and applications in the field of machine vision can be improved, such as image segmentation, object tracking, image recognition, and image reconstruction.

[0003] However, due to limitations in the principles of existing depth camera technology, the quality of the depth images obtained from them is inferior to that of visual images, containing significant noise interference. This noise typically includes random noise and "black holes" of varying shapes at object edges and on dark surfaces—regions where depth information has been lost. These issues hinder the application of depth information from depth images. Therefore, to obtain more accurate information, depth enhancement processing is needed to remove noise from the depth images.

[0004] Currently, depth image denoising methods are mainly divided into two categories: traditional image denoising methods and deep learning-based denoising methods. Deep learning-based denoising methods are highly complex and demanding on hardware. Traditional image denoising methods mainly include filtering methods and transform domain denoising methods. Filtering methods utilize the local spatial correlation of an image to achieve weighted summation filtering, such as nonlocal mean filtering, bilateral filtering, and guided filtering. While these methods can effectively remove noise, they require selecting appropriate filter window sizes and weight parameters based on different noisy images. Furthermore, to preserve some edge texture information while improving denoising performance, the algorithms are often highly complex and require significant computational resources. Unlike filtering methods, transform domain denoising often uses wavelet transform or sparse representation for preprocessing, and then utilizes the characteristics of the transform domain or the redundancy of the representation for further processing. Examples include the K-SVD algorithm and wavelet denoising. However, transform domain denoising methods also face the challenge of selecting the transform domain and wavelet coefficients based on the noisy image, and are often also highly complex. The urgent technical problem to be solved is how to address the issues of high computational resource consumption, low robustness, and loss of detail information such as texture and image edges in existing depth image denoising techniques. Summary of the Invention

[0005] This application provides a denoising method, image processing system, and image processing apparatus for depth images, which solves the problems of long processing time, high complexity, low robustness, and loss of detail information such as texture and image edges in existing denoising methods.

[0006] To achieve the above objectives, in a first aspect, embodiments of this application provide a method for denoising depth images, characterized by comprising the following steps:

[0007] Step 1: Fill the edges with pixels from the depth image of the acquired real scene;

[0008] Step 2: Calculate the confidence score for each pixel in the depth image;

[0009] Step 3: Calculate the confidence weight matrix for each pixel based on the confidence level, and calculate the denoised pixel value based on the confidence weight matrix.

[0010] Optionally, the confidence level is the difference between the number of abruptly changing pixels in the neighboring pixels of the current pixel and the number of neighboring pixels.

[0011] Optionally, the absolute value of the difference between the pixel value of the adjacent pixel and the pixel value of the current pixel is used as the mutation amount of the adjacent pixel.

[0012] Optionally, a mutation threshold is preset, and the mutation amount is compared with the mutation threshold to determine whether the adjacent pixels have mutated.

[0013] Optionally, the threshold is set by the number of neighboring pixels of the current pixel and the abrupt change of the neighboring pixels.

[0014] Optionally, the element values ​​of the confidence weight matrix are the confidence scores of the element when it is used as the current pixel.

[0015] Optionally, the confidence weight matrix is ​​normalized so that the sum of all element values ​​in the confidence weight matrix is ​​1.

[0016] Optionally, the denoised pixel values ​​are obtained from the element values ​​in the weight matrix, the corresponding pixel values ​​of the depth image, and the number of elements in the weight matrix.

[0017] Secondly, embodiments of this application provide an image processing system, characterized in that the image processing system includes an image acquisition module and an image processing module, wherein the image processing module uses the depth image denoising method described in the first aspect.

[0018] Thirdly, embodiments of this application provide an image processing apparatus, characterized in that the image processing apparatus includes: a memory and a processor, the memory being used to store an original image, and the processor being used to execute the depth image denoising method described in the first aspect.

[0019] This application provides a depth image denoising method that significantly reduces the computational resources required for denoising by using the number of mutations in a single pixel along its adjacent directions as the confidence weight. Simultaneously, this denoising method does not require parameter adjustments, greatly improving its robustness. Furthermore, both noise and edges are treated as mutations, with the number of mutations in the pixel's adjacent directions used as the confidence weight. This achieves good noise filtering while preserving more texture details and other information. This results in reduced computational resource consumption, enhanced robustness of the denoising method, and improved denoising performance while preserving texture and edge details. Attached Figure Description

[0020] Figures 1A to 1F These are all schematic diagrams of a pixel unit provided in the embodiments of this application;

[0021] Figure 2A A schematic flowchart illustrating a depth image denoising method provided in an embodiment of this application;

[0022] Figure 2B A flowchart illustrating a method for obtaining pixel confidence scores provided in an embodiment of this application;

[0023] Figure 2C A flowchart illustrating a method for calculating denoised pixel values ​​provided in an embodiment of this application;

[0024] Figure 3 A comparative schematic diagram showing the existing denoising method provided in the embodiments of this application and the denoising method provided in this application;

[0025] Figure 4 This is a schematic diagram of an image processing system provided in an embodiment of this application;

[0026] Figure 5 This is a schematic diagram illustrating the implementation of an image processing apparatus provided in an embodiment of this application. Detailed Implementation

[0027] In the following description, specific details such as particular system structures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application can also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of the working principles of well-known image acquisition methods, image filtering methods, and image processing methods are omitted to avoid unnecessary detail from hindering the description of this application.

[0028] The terminology used in the following embodiments is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “the,” “the,” and “the” are intended to also include expressions such as “one or more,” unless the context clearly indicates otherwise.

[0029] With the rapid development of computer vision technology, the importance of scene depth information is becoming increasingly prominent, such as obstacle recognition in autonomous driving scenarios, measurement of the 3D dimensions of objects, and 3D reconstruction of natural scenes. Currently, depth image acquisition methods mainly include Time-of-Flight (TOF) cameras, binocular cameras, structured light cameras, and LiDAR. However, limitations imposed by ambient light sources and the material of the objects being photographed result in the final output depth images exhibiting a wide variety of noise types and significant noise amplitudes. This noise severely impacts the quality of depth maps, preventing them from meeting practical application requirements. Therefore, improving depth map quality is of significant research importance. Furthermore, due to the hardware limitations of the cameras themselves, the complexity of depth map denoising methods is also crucial.

[0030] Therefore, this application proposes a depth image denoising method that significantly reduces the computational resources required for denoising by using the number of mutations in a single pixel along its adjacent directions as the confidence weight. Simultaneously, this depth image denoising method does not require parameter adjustments, greatly improving its robustness. Furthermore, noise and edges are both treated as mutations, with the number of mutations in the pixel's adjacent directions used as the confidence weight, achieving good noise filtering while preserving more texture details. This achieves reduced computational resource consumption, enhanced robustness of the denoising method, and improved denoising performance while preserving image texture and edge details.

[0031] The following is a schematic diagram of pixel unit division provided in the embodiments of this application. See also Figure 1A , Figure 1A A schematic diagram of a pixel unit provided in an embodiment of this application is shown below. Figure 1A As shown, the pixel unit may include: the current pixel, and the adjacent pixels 1, 2, 3, 4, 5, 6, 7, and 8.

[0032] It should be further noted that, see [link / reference] Figure 1B , Figure 1B A schematic diagram of another pixel unit provided in an embodiment of this application, as shown below. Figure 1BAs shown, the pixel unit may include: the current pixel, and pixel 1, pixel 2, pixel 3, and pixel 4 adjacent to the current pixel.

[0033] It should be further noted that, see [link / reference] Figure 1C , Figure 1C A schematic diagram of another pixel unit provided in an embodiment of this application, as shown below. Figure 1C As shown, the pixel unit may include: the current pixel, and the adjacent pixels 1, 2, 3, 4, 5, and 6.

[0034] It should be further noted that, see [link / reference] Figure 1D , Figure 1D A schematic diagram of another pixel unit provided in an embodiment of this application, as shown below. Figure 1D As shown, the pixel unit may include: the current pixel, and pixel 1, pixel 2, and pixel 3 adjacent to the current pixel.

[0035] It should be further noted that, see [link / reference] Figure 1E , Figure 1E A schematic diagram of another pixel unit provided in an embodiment of this application, as shown below. Figure 1E As shown, the pixel unit may include: the current pixel, and pixel 1, pixel 2, and pixel 3 adjacent to the current pixel.

[0036] It should be further noted that, see [link / reference] Figure 1F , Figure 1F A schematic diagram of another pixel unit provided in an embodiment of this application, as shown below. Figure 1F As shown, the pixel unit may include: the current pixel, and the adjacent pixels 1, 2, 3, 4, 5, and 6.

[0037] It should be noted that, Figures 1A-1C This application only provides a few ways to divide the current pixel and its neighboring pixels. In practical applications, the pixels can be re-divided according to the requirements. This application does not impose any specific restrictions.

[0038] The following section provides a detailed introduction to denoising methods for depth imaging.

[0039] See Figure 2A , Figure 2A This is a flowchart illustrating a depth image denoising method provided in an embodiment of this application, as shown below. Figure 2A As shown, the process of the depth image denoising method includes the following steps:

[0040] S10. Fill the edges of the acquired depth image of the real scene with pixels. In this step, the pixel values ​​can be normalized to the range of 0-255. Normalization of pixel values ​​is optional; this application does not impose specific restrictions.

[0041] S20, Participation Figure 2B Calculate the confidence score for each pixel in the depth image.

[0042] Figure 2B This is a flowchart illustrating a method for obtaining pixel confidence scores provided in an embodiment of this application, as shown below. Figure 2B As shown, the method for obtaining pixel confidence includes the following steps:

[0043] S201, Determine the neighboring pixels of this pixel. This can be achieved by using... Figure 1A-1F The adjacent pixel embodiment shown determines adjacent pixels, but the adjacent pixels of this pixel can also be determined according to actual needs. This application does not impose specific limitations.

[0044] S202. The difference between the value of the adjacent pixel and the value of the current pixel is used as the mutation amount. In this step, the absolute value of the difference between the pixel value of the adjacent pixel determined in S201 and the pixel value of the current pixel is used as the mutation amount of the adjacent pixel.

[0045] S203. Determine whether adjacent pixels have undergone a mutation. The difference between the number of mutated pixels and the number of adjacent pixels is used as the confidence level of this pixel. In this step, a mutation threshold is preset. If the mutation amount of the adjacent pixel is greater than the preset threshold, the adjacent pixel is considered to have mutated; otherwise, the adjacent pixel is considered not to have mutated. The threshold can be preset using formula (1):

[0046]

[0047] In formula (1), a is the threshold, b i Let N be the abrupt change of the i-th neighboring pixel of this pixel, and N be the total number of neighboring pixels of this pixel. For example, in... Figure 1A In the illustrated embodiment, N is 8. Formula (1) only provides an illustrative method for setting the threshold, but it is not limited to the method provided by formula (1). For example, the maximum value, minimum value, difference, and other methods can also be used to set the threshold.

[0048] S30, see also Figure 2C The confidence weight matrix for each pixel is calculated based on the confidence level, and the denoised pixel value is calculated based on the confidence weight matrix. Figure 2C This is a flowchart illustrating a method for calculating denoised pixel values ​​provided in an embodiment of this application, as shown below. Figure 2CThe method shown includes the following steps:

[0049] S301: Create an M*K weight matrix, where the values ​​of M and K are determined by the number of neighboring pixels of the current pixel. For example, in... Figure 1A In the illustrated embodiment, both M and K are 3. Figure 1B In the example shown, the values ​​of M and K are both 2. Figure 1C In the illustrated embodiment, the value of M is 3 and the value of K is 2. Figure 1D In the illustrated embodiment, the value of M is 1 and the value of K is 3. Figure 1E In the illustrated embodiment, the value of M is 3 and the value of K is 1. Figure 1F In the illustrated embodiment, the value of M is 3 and the value of K is 2. However, the values ​​of M and K can also be 2 and 3, or M and K can be 6 and 1, or M and K can be 1 and 6. This application does not impose any specific limitations. The values ​​of the elements in the M*K weight matrix created in step S301 represent the confidence level when that element in the matrix is ​​used as the current pixel.

[0050] S302. Normalize the above weight matrix. Normalization in this step ensures that the sum of all elements in the weight matrix is ​​1. Of course, this step can be omitted, and this application does not impose specific restrictions.

[0051] S303. Calculate the denoised pixel values. In this step, the denoised pixel values ​​can be defined as those in formula (2).

[0052]

[0053] Where d is the denoised pixel value, q i Let be the value of the i-th element in the weight matrix. i For q i The corresponding depth image pixel values, M and K are the values ​​determined in step S301.

[0054] Table (1) compares the quantitative indicators of the depth image denoising method provided in this embodiment with those of bilateral filtering and guided filtering. The denoising method provided in this embodiment was used to process noise, and its denoising effect was compared with that of bilateral filtering and guided filtering. After five experiments, the average value was taken, and the experimental results are shown in Table (1). The PSNR (Peak Signal-to-Noise Ratio) of the denoising method provided in this embodiment is slightly lower than that of guided filtering, but its SSIM (Structural Similarity) value and computation time are better than those of bilateral filtering and guided filtering, demonstrating the effectiveness of the depth image denoising method provided in this embodiment.

[0055] Table (1)

[0056]

[0057] See Figure 3 , Figure 3 A comparative schematic diagram of the existing denoising method and the denoising method provided in this application is shown in the embodiments of this application. Figure 3 Figure 301 shows the overall image, from left to right: the original image, the overall image after bilateral filtering, the overall image after guided filtering, and the overall image after denoising using the denoising method provided in this application; Figure 302 shows a magnified view of a portion within the box at the upper right of the overall image, from left to right: the original image, the image after bilateral filtering, the image after guided filtering, and the image after denoising using the denoising method provided in this application; Figure 303 shows a magnified view of a portion within the box at the lower left of the overall image, from left to right: the original image, the image after bilateral filtering, the image after guided filtering, and the image after denoising using the denoising method provided in this application. Figure 3 As shown in Figure 302, in the flat areas of the image, bilateral filtering, guided filtering, and the denoising method provided in this application all achieved good denoising results, with guided filtering showing the best effect, followed by the denoising method provided in this application and bilateral filtering. As shown in Figure 303, in the edge areas of the image, the denoising method provided in this application significantly better preserves image edge and texture information than bilateral filtering and guided filtering.

[0058] This application embodiment significantly reduces the computational resources required by the algorithm by using the number of mutations of a single pixel on its neighboring pixels as the confidence weight. Simultaneously, the denoising method provided by this application embodiment does not require parameter adjustment, greatly improving the robustness of the algorithm. Furthermore, since noise and edges are both treated as mutations, this application uses the number of mutations on neighboring pixels as the confidence weight, achieving good noise filtering results while preserving more texture details and other information. This achieves reduced computational resource consumption, enhanced algorithm robustness, and improved denoising performance while preserving image texture and edge details.

[0059] See Figure 4 , Figure 4 This is a schematic diagram of an image processing system provided in an embodiment of this application, such as... Figure 4The image processing system shown includes an image acquisition module 401, an image processing module 402, and an image output module 403. The image acquisition module acquires depth images, and its acquisition methods mainly include TOF cameras, binocular cameras, structured light cameras, and LiDAR, etc., which are not specifically limited in this application. The image acquisition module 401 inputs the acquired image to the image processing module 402 for processing. The image processing module 402 can perform filtering, smoothing, and other processing on the acquired image, and can use the denoising method provided in the embodiments of this application to denoise the acquired image; the image output module 403 outputs the image processed by the image processing module 402.

[0060] See Figure 5 , Figure 5 This is a schematic diagram illustrating the implementation of an image processing apparatus provided in an embodiment of this application, such as... Figure 5 The image includes a memory 501 and a processor 502. The memory 501 stores the acquired raw images; the processor 502 executes image processing algorithms or computer programs. The memory can be a data storage area capable of storing related raw images, etc. Furthermore, the memory can be a high-speed random access memory, or a non-volatile memory, such as a plug-in hard disk, a smart media card (SMC), a secure digital card (SD), or a flash card, or other volatile solid-state storage devices. The processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, or the processor can be any conventional processor.

[0061] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0062] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0063] In the embodiments provided in this application, it should be understood that the disclosed apparatus / devices and methods can be implemented in other ways. For example, the apparatus / device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0064] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0065] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0066] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0067] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0068] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0069] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A method for denoising depth images, characterized in that, Including: the following Steps: Step 1: Fill the edges with pixels from the depth image of the acquired real scene; Step 2: Calculate the confidence score for each pixel in the depth image; Step 3: Calculate the confidence weight matrix for each pixel based on the confidence level, and calculate the denoised pixel value based on the confidence weight matrix; The confidence level is the difference between the number of abruptly changed pixels in the neighboring pixels of this pixel and the number of neighboring pixels.

2. The method for denoising depth images according to claim 1, characterized in that, The absolute value of the difference between the pixel value of the adjacent pixel and the pixel value of the current pixel is used as the mutation amount of the adjacent pixel.

3. The method for denoising depth images according to claim 2, characterized in that, A preset mutation threshold is used to compare the mutation amount with the mutation threshold to determine whether the adjacent pixels have mutated.

4. The method for denoising depth images according to claim 3, characterized in that, The threshold is set by the number of neighboring pixels of the current pixel and the abrupt change of the neighboring pixels.

5. The method for denoising depth images according to claim 1, characterized in that, The element value of the confidence weight matrix is ​​the confidence level of that element when it is used as the current pixel.

6. The method for denoising depth images according to claim 1, characterized in that, The confidence weight matrix is ​​normalized so that the sum of all element values ​​in the confidence weight matrix is ​​1.

7. The method for denoising depth images according to claim 1, characterized in that, The denoised pixel values ​​are obtained by the element values ​​in the weight matrix, the corresponding pixel values ​​of the depth image, and the number of elements in the weight matrix.

8. An image processing system, characterized in that, The image processing system includes an image acquisition module and an image processing module, wherein the image processing module uses the denoising method for depth images as described in any one of claims 1 to 7.

9. An image processing apparatus, characterized in that, The image processing apparatus includes a memory and a processor, the memory being used to store an original image, and the processor being used to execute a denoising method for a depth image as described in any one of claims 1 to 7.

Images