Denoising methods, systems, and image processing devices for flying point noise in depth images
By projecting the point cloud data of the depth image onto the xz and yz planes for sparsity detection, removing flying point noise, and back-projecting it back into the depth image, the problem of long removal time and information loss in removing flying point noise in the depth image is solved, achieving efficient and accurate noise removal.
Patent Information
- Application Number
- CN202410812904.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2023-12-01
- Filing Date
- 2024-06-22
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2044-06-22
AI Technical Summary
Existing methods for removing flying point noise from depth images are time-consuming and prone to losing useful information, making it difficult to meet the requirements of real-time performance and accuracy.
Sparsity detection is performed by projecting the point cloud data of the depth image onto the xz and yz planes, removing points that meet the sparsity condition as flying point noise, and then back-projecting back to the depth image to retain the effective pixels.
It effectively removes flying point noise, avoids the shortcomings of traditional methods that mistakenly remove effective pixels and point cloud domain methods that consume a lot of computational resources, and improves detection speed and accuracy.
Smart Images

Figure CN118552437B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image processing technology, and in particular to a method, system and image processing apparatus for denoising flying point noise in depth images. Background Technology
[0002] Depth images have become an important data source in the field of computer vision, widely used in 3D reconstruction, industrial measurement, object tracking, and face recognition. However, in practical applications, depth images are often affected by various noises, such as flying spots and dead pixels. These noises can seriously affect subsequent processing. Therefore, removing noise from depth images is crucial for improving their quality and usability.
[0003] However, due to limitations in the principles and hardware of current depth cameras, the quality of acquired depth images is relatively low and easily affected by noise. Flying-spot noise, in particular, arises because each pixel on an image sensor has a specific physical size. When measuring object edges, a single pixel receives light reflected from both the foreground and background simultaneously. The superposition of these energies causes the sensor to obtain an incorrect depth value, which visually appears as an invalid point floating in the air. Flying-spot noise severely interferes with depth information at object edges, preventing depth cameras from acquiring effective depth edge information and significantly limiting the further development and application of depth images.
[0004] Currently, there are two main methods for removing flying point noise from depth images: traditional methods and deep learning methods. Deep learning methods often require large datasets and have high algorithm complexity, as well as certain hardware requirements. Traditional algorithms mainly include edge detection methods and point cloud domain filtering methods. While edge detection methods are computationally simple, they are prone to mistakenly removing valid pixels, resulting in significant loss of edge information. Although point cloud domain filtering can effectively remove flying point noise, it typically requires substantial computational resources, is slow, and struggles to meet real-time requirements. Therefore, finding a convenient, fast, and effective way to remove flying point noise from depth images is a pressing technical problem that needs to be solved. Summary of the Invention
[0005] This application provides a method, system, and image processing apparatus for denoising flying point noise in depth maps, which solves the problems of long time consumption and easy loss of effective information in existing flying point denoising methods.
[0006] To achieve the above objectives, in a first aspect, embodiments of this application provide a method for denoising flying point noise in depth images, characterized by comprising the following steps:
[0007] Step 1: Acquire image data of the depth image;
[0008] Step 2: Project the point cloud data obtained from the depth image calculation onto the xz and yz planes;
[0009] Step 3: Perform sparsity testing on every point in the xz and yz planes;
[0010] Step 4: If the sparsity test is satisfied, then the point is flying noise;
[0011] Step 5: Backproject the projection data of the xz and yz planes after removing flying point noise back to the depth image, and retain the same values in the depth images of the xz and yz planes after removing flying point noise.
[0012] Optionally, projecting onto the xz plane involves normalizing the x-coordinate of the point cloud data to 0 to the height of the depth image, normalizing the z-coordinate to 0 to the width of the depth image, and keeping the y-coordinate unchanged. In this case, the x-coordinate of the point cloud data becomes the x-coordinate of the projected image, the z-coordinate of the point cloud data becomes the y-coordinate of the projected image, and the y-coordinate of the point cloud data becomes the pixel value of the projected image. Projecting onto the yz plane is similar.
[0013] Optionally, the number of pixels with a value of 0 in this pixel and its neighboring pixels is considered sparsity.
[0014] Optionally, a sparsity threshold is preset, and the sparsity is compared with the sparsity threshold to determine whether the pixel satisfies the sparsity requirement.
[0015] Optionally, the threshold is set by the number of neighboring pixels of the current pixel and the number of neighboring pixels with a value of 0.
[0016] Optionally, removing flying point noise involves setting the pixel value of the point to 0 on the xz plane or yz plane.
[0017] Optionally, retaining the same values in the depth images transformed back from the xz and yz planes after removing flying point noise involves comparing the depth images transformed from the xz plane and the yz plane. If the pixel values at the same location are the same, the pixel is retained; otherwise, it is considered as flying point noise and is removed.
[0018] In a second aspect, embodiments of this application provide a depth image flying point noise processing system, characterized in that the depth image flying point noise processing system includes an image acquisition module, a calculation module, a judgment module, and an image output module, wherein the calculation module and the judgment module use the depth image flying point noise denoising method described in the first aspect.
[0019] Thirdly, embodiments of this application provide an image processing apparatus, characterized in that the image processing apparatus includes: a collector, a memory, and a processor, wherein the collector is used to collect depth images, the memory is used to store the depth images collected by the collector, and the processor is used to execute the denoising method for flying point noise in depth images as described in the first aspect.
[0020] This application provides a method, system, and image processing apparatus for denoising flying point noise in depth images, relating to the field of image processing technology. The method involves projecting point cloud data calculated from the depth image onto the xz and yz planes, performing sparsity detection on the projected point cloud data, and identifying points that meet sparsity conditions as flying point noise. The method retains the common values in the depth images obtained from the inverse transformation of the xz and yz planes, resulting in the depth image after removing the flying point noise. This invention avoids the drawbacks of traditional edge detection methods that easily mistakenly remove valid pixels when removing flying point noise, and also avoids the high computational resource requirements of point cloud domain flying point noise removal. It can improve detection speed and accuracy while retaining valid pixels. Attached Figure Description
[0021] Figure 1 A flowchart illustrating a depth image flying point denoising method provided in an embodiment of this application;
[0022] Figure 2 A schematic diagram illustrating a projection onto the xz and yz planes, provided for an embodiment of this application;
[0023] Figure 3 A schematic flowchart of a sparsity detection method provided in an embodiment of this application;
[0024] Figure 4 A schematic diagram of a depth image flying point noise processing system provided in an embodiment of this application;
[0025] Figure 5 This is a schematic diagram illustrating the implementation of a depth image flying point noise image processing device provided in an embodiment of this application. Detailed Implementation
[0026] In the following description, specific details such as particular system structures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application can also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of the working principles of well-known image acquisition methods, image filtering methods, and image processing methods are omitted to avoid unnecessary detail from hindering the description of this application.
[0027] The terminology used in the following embodiments is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “the,” “the,” and “the” are intended to also include expressions such as “one or more,” unless the context clearly indicates otherwise.
[0028] With the widespread application of 3D image processing technology in 3D reconstruction, industrial measurement, autonomous driving, and other fields, the acquisition and quality improvement of depth images have become increasingly important. Currently, depth image acquisition methods mainly include ToF cameras, binocular cameras, structured light cameras, and LiDAR. However, due to the influence of imaging principles, shooting environment, and hardware conditions, the imaging quality of depth images is often low and easily affected by noise. Flying point noise, in particular, severely impacts the extraction of object edge information by depth cameras, making it difficult to meet practical application requirements. Therefore, removing flying point noise and improving depth image quality is of great significance. Furthermore, considering the real-time requirements in practical application scenarios, the complexity of algorithms for removing flying point noise in depth images also needs to be appropriately increased.
[0029] Therefore, this application proposes a method for denoising flying point noise in depth images. This method transforms the depth image to the point cloud domain and projects it onto the xz and yz planes. Sparsity detection is then performed on the xz and yz planes, and finally, the image is back-projected back to the depth image. This method avoids the drawback of traditional edge detection methods that easily mistakenly remove valid pixels, effectively preserving edge information in the depth image. It also avoids the high computational resource requirements of point cloud domain noise removal, thus meeting the real-time and reliability requirements of practical applications.
[0030] The following section will first provide a detailed introduction to methods for denoising flying point noise in depth images.
[0031] See Figure 1 , Figure 1 This is a flowchart illustrating a depth image denoising method provided in an embodiment of this application, as shown below. Figure 1 As shown, the process of the depth image denoising method includes the following steps:
[0032] S10. Image data of the depth image of the real scene.
[0033] S20, see also Figure 2 The point cloud data calculated from the depth image is projected onto the xz and yz planes.
[0034] Figure 2 This is a schematic diagram illustrating a process for obtaining projection data of the xz and yz planes, provided as an embodiment of this application. Figure 2As shown, obtaining the projection data of the xz plane and yz plane includes the following steps:
[0035] The process begins by reading depth image data. It should be noted that the depth data read is the depth image captured by the depth camera.
[0036] The depth data is then converted into a point cloud. It should be noted that the conversion of depth data into a point cloud can be calculated according to formula (1):
[0037]
[0038] In formula (1), , Represents any coordinate point in the image coordinate system. and This indicates the center coordinates of the image. , , Represents a three-dimensional coordinate point in the world coordinate system. This represents the depth value of the depth image. , , These are camera parameters.
[0039] After converting to point cloud data, the projection onto the xz plane first requires normalizing the x-axis data in the point cloud data to the image height. It should be noted that the image height refers to the height of the read depth image.
[0040] Next, the z-axis data in the point cloud data is normalized to the image width. It should be noted that the normalization in the above steps is only to ensure that the projection data and the depth image data are the same size, which facilitates subsequent processing.
[0041] Finally, the y-axis data is used as the pixel values of the projected image. It should be noted that the x and y-axis coordinates of the projected image are the x-axis coordinates and z-axis coordinates of the point cloud data after the above normalization process, respectively.
[0042] Projection onto the yz plane is similar to projection onto the xz plane, except that projection onto the yz plane requires normalizing the y-axis data in the point cloud data to the image height, and using the x-axis data as the pixel values of the projected image.
[0043] S30, see also Figure 3 Sparsity detection is performed on every point in the xz plane and yz plane.
[0044] Figure 3This is a schematic flowchart illustrating a process for sparsity detection of each point on the xz and yz planes, provided as an embodiment of this application. Figure 3 As shown, sparsity detection is performed on each point in the xz and yz planes, including the following steps:
[0045] S301. Read the projection data of the xz plane and yz plane.
[0046] S302. Iterate through each point in the projection data of the xz plane and yz plane.
[0047] S303. Determine the neighboring pixels of this pixel. It should be noted that neighboring pixels generally refer to all pixels within the 3*3 neighborhood of this pixel. Alternatively, the neighboring pixels of this pixel can be determined according to actual needs. This application does not impose any specific restrictions.
[0048] S304. Count the number of non-zero values in this pixel and its neighboring pixels, and use it as the sparsity value.
[0049] S40. Determine whether the current pixel satisfies the sparsity detection. If it does, the pixel is considered flying noise. In this step, a sparsity threshold is preset. If the sparsity of the pixel is less than the preset threshold, the pixel is considered to satisfy the sparsity detection; otherwise, it is considered not to satisfy the sparsity detection. The threshold can be set based on the number of the current pixel and its neighboring pixels. Generally, the threshold needs to be less than half the number of the current pixel and its neighboring pixels.
[0050] S50. Transform the projection data after removing flying point noise back into a depth image. It should be noted that after removing flying point noise from the projection data of the xz plane and yz plane, the projection data are transformed back into depth images respectively. Then, the same pixels in the two depth images are retained as the depth image after removing flying point noise.
[0051] See Figure 4 , Figure 4 This is a schematic diagram of an image processing system provided in an embodiment of this application, such as... Figure 4The image processing system shown includes: an image acquisition module 401, a first calculation module 402, a second calculation module 403, a third calculation module 404, a first judgment module 405, a second judgment module 406, and an image output module 407. The image acquisition module acquires depth images, and its acquisition methods mainly include TOF cameras, binocular cameras, structured light cameras, and LiDAR, etc., which are not specifically limited in this application. The image acquisition module 401 inputs the acquired image to the first calculation module 402, which performs point cloud conversion and projection onto the xz and yz planes of the acquired depth image. Then, the projection data of the xz and yz planes are input to the second calculation module 403 and the third calculation module 404, respectively, which perform sparsity detection on the projection data of the xz and yz planes. The first judgment module 405 and the second judgment module 406 determine whether the sparsity values output by the second calculation module 403 and the third calculation module 404 satisfy the sparsity detection. The image output module 407 is used to transform the projection data after removing flying point noise back into a depth image, and retain the same pixels in the two depth images as the depth image after removing flying point noise.
[0052] See Figure 5 , Figure 5 This is a schematic diagram illustrating the implementation of a depth image flying point noise processing device provided in an embodiment of this application, as shown below. Figure 5 The device includes a data acquisition unit 501, a memory 502, and a processor 503. The data acquisition unit 501 is used to acquire depth images, the memory 502 is used to store the depth images acquired by the data acquisition unit 501, and the processor 503 is used to execute algorithms or computer programs for processing flying point noise in the depth images.
[0053] The denoising device for flying point noise in depth images according to this invention collects image data of a depth image and converts it into point cloud data. The point cloud data is then projected onto the xz and yz planes respectively. Sparsity detection is then performed on each point of the projection data in the xz and yz planes according to a preset threshold. If a point satisfies the sparsity detection, it is considered flying point noise. Finally, the projection data of the xz and yz planes after removing flying point noise are back-projected back into the depth image, and identical pixels from both depth images are retained as the depth image after removing flying point noise. This effectively avoids the shortcomings of traditional edge detection algorithms that easily mistakenly remove valid pixels when removing flying point noise, and the drawback of point cloud domain flying point noise removal requiring a large amount of computational resources. It can improve detection speed and accuracy while retaining valid pixels.
[0054] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0055] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0056] In the embodiments provided in this application, it should be understood that the disclosed apparatus / devices and methods can be implemented in other ways. For example, the apparatus / device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.
[0057] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.
[0058] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.
[0059] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."
[0060] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.
[0061] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.
[0062] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.
Claims
1. A method for denoising flying point noise in depth images, characterized in that, Includes the following steps: Step 1: Acquire image data of the depth image; Step 2: Calculate point cloud data based on the image data, and project the point cloud data onto the xz plane and yz plane respectively; Step 3: Perform sparsity detection on each point in the point cloud projection data of the xz plane and yz plane, where sparsity is the number of pixels with a value of 0 among the neighboring pixels of the current pixel; Step 4: If the sparsity detection is satisfied, then the point is determined to be flying noise; Step 5: Independently backproject the point cloud projection data of the xz plane and yz plane after removing flying point noise back to the depth image to obtain the first reconstructed depth image and the second reconstructed depth image. Compare the pixel values at the same position in the first reconstructed depth image and the second reconstructed depth image, and retain only the pixels with the same pixel value as the depth image after removing flying point noise.
2. The method for denoising flying point noise in depth images according to claim 1, characterized in that, The projection onto the xz plane involves normalizing the x-coordinate of the point cloud data to 0 to the height of the depth image, normalizing the z-coordinate to 0 to the width of the depth image, and keeping the y-coordinate unchanged. In this case, the x-coordinate of the point cloud data becomes the x-coordinate of the projected image, the z-coordinate of the point cloud data becomes the y-coordinate of the projected image, and the y-coordinate of the point cloud data becomes the pixel value of the projected image. The projection onto the yz plane is similar.
3. The method for denoising flying point noise in depth images according to claim 2, characterized in that, A preset sparsity threshold is used to compare the sparsity with the sparsity threshold to determine whether the pixel satisfies the sparsity requirement.
4. The method for denoising flying point noise in depth images according to claim 3, characterized in that, The threshold is set by the number of neighboring pixels of the current pixel and the number of neighboring pixels with a value of 0.
5. The method for denoising flying point noise in depth images according to claim 4, characterized in that, The removal of flying point noise involves setting the pixel value of the point to 0 on the xz or yz plane, that is, setting the corresponding x or y value in the original point cloud data to 0, and then back-projecting it back to the depth image.
6. An image processing system, characterized in that, The image processing system includes an image acquisition module, a calculation module, a judgment module, and an image output module. The calculation module and the judgment module use the denoising method for flying point noise in depth images as described in any one of claims 1 to 5.
7. An image processing apparatus, characterized in that, The image processing device includes: a collector, a memory, and a processor. The collector is used to acquire depth images, the memory is used to store the depth images acquired by the collector, and the processor is used to execute the denoising method for flying point noise in depth images as described in any one of claims 1 to 5.
Citation Information
Patent Citations
Point cloud data de-noising method and system
CN106548520A
Depth map denoising method and device, medium and equipment
CN115115535A
