Point cloud hole repairing method and system based on original depth image

Abstract

The application relates to a point cloud hole repairing method and system based on an original depth image, which comprises the following steps: S1, acquiring an original depth image, a filter kernel size and an iteration number; S2, distinguishing the original depth image into an effective area and a to-be-processed area; S3, distinguishing the to-be-processed area into an invalid area and a to-be-filled area; S4, obtaining a normalized image through normalization processing; S5, multiplying the normalized image with the original depth image to obtain an effective image; S6, counting the sum of pixel values in a filter kernel of the normalized image and recording the sum into an effective point number list; counting the sum of pixel values in a filter kernel of the effective image and recording the sum into an effective point depth value list; S7, dividing the effective point depth value list by the effective point number list to obtain a filling depth value of the to-be-filled area; and S8, iterating the steps S2 to S7. The application can fill image areas of any shape, and the pixels on the boundary of the hole area are not affected by the areas outside the detection area and the invalid area.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of industrial automation machine vision, and in particular to a point cloud hole repairing method and system based on original depth images. BACKGROUND

[0002] In the field of industrial automation machine vision, point cloud loss problems often occur during point cloud image acquisition, splicing, coordinate system transformation and other processing, and repairing the lost point cloud becomes an important step in image processing. The mean filling method is widely used in such problems, but many current methods process the entire image or a rectangular region of the image, and the processing effect is not good when processing the image boundary. For example, the mean filtering method in the commonly used image processing open source library opencv in the field of machine vision takes the mean value in the neighborhood of each pixel point as the filling value of the hole. The disadvantages of this processing are as follows: first, the processing region can only be rectangular or the entire image, and the effect is not good for processing complex region images of any shape. Second, since the image boundary is processed by filling pixels, when processing the image boundary pixels, it will be affected by the image outside or invalid region, resulting in a large difference between the hole repair value and the actual value on the boundary, causing large errors or even errors in subsequent processing. SUMMARY

[0003] The technical problem to be solved by the present application is to provide a point cloud hole repairing method and system based on original depth images.

[0004] The technical scheme adopted by the present application to solve the technical problem is: a point cloud hole repairing method based on original depth images, comprising the following steps:

[0005] S1: obtaining an original depth image, a filter kernel width, a filter kernel height and an iteration number;

[0006] S2: dividing the original depth image into an effective region and a to-be-processed region by binarization;

[0007] S3: dividing the to-be-processed region into an invalid region and a to-be-filled region by area judgment;

[0008] S4: setting the pixel value of the effective region to 1 and the pixel value of the to-be-processed region to 0 to obtain a normalized image;

[0009] S5: multiplying the normalized image by the original depth image to obtain an effective image;

[0010] S6: taking the pixel point belonging to the to-be-filled region in the normalized image as an anchor point, establishing the filter kernel, counting the sum of pixel values in the filter kernel, and recording into an effective point number list; taking the pixel point belonging to the to-be-filled region in the effective image as an anchor point, establishing the filter kernel, counting the sum of pixel values in the filter kernel, and recording into an effective point depth value list;

[0011] S7: dividing the effective point depth value list by the effective point number list to obtain a filling depth value of the to-be-filled region, and outputting a depth image after repairing a hole edge.

[0012] S8: iteratively performing the operations of S2 to S7 to obtain a depth image after a hole is completely repaired.

[0013] Preferably, the step S1 further comprises:

[0014] The area of the to-be-filled region, the area of the invalid region, and a set value are obtained, the set value is greater than the area of the to-be-filled region, and the set value is less than the area of the invalid region.

[0015] Preferably, the step S2 further comprises:

[0016] The pixel gray value of the effective region is converted into a preset value, and the pixel gray value of the to-be-processed region is converted into 0.

[0017] Preferably, the preset value is 255.

[0018] Preferably, the step S3 further comprises:

[0019] It is judged whether the current region area is greater than the set value, if yes, the invalid region is judged, and if not, the to-be-filled region is judged.

[0020] Preferably, the filter kernel width w and the filter kernel height h are both odd numbers.

[0021] Preferably, the iteration number is set according to the shape and area of the to-be-filled region, and the filter kernel width w and the filter kernel height h, so as to cover all pixel points in the to-be-filled region.

[0022] Preferably, when the original depth image does not contain the invalid region, the step S3 is omitted.

[0023] The application also constructs a point cloud hole repair system based on an original depth image, comprising:

[0024] An acquisition module is configured to acquire an original depth image, a filter kernel width, a filter kernel height, and an iteration number;

[0025] The first distinguishing module is configured to distinguish the original depth image into an effective area and a to-be-processed area through binarization.

[0026] The second distinguishing module is configured to distinguish the to-be-processed area into an invalid area and a to-be-filled area through area judgment.

[0027] The normalization module is configured to set the pixel value of the effective area as 1 and the pixel value of the to-be-processed area as 0 to obtain a normalized image.

[0028] The first calculation module is configured to multiply the normalized image by the original depth image to obtain an effective image.

[0029] The list recording module is configured to take the pixel point belonging to the to-be-filled area in the normalized image as an anchor point to establish the filter kernel, count the sum of the pixel values in the filter kernel, and record in an effective point number list; and take the pixel point belonging to the to-be-filled area in the effective image as an anchor point to establish the filter kernel, count the sum of the pixel values in the filter kernel, and record in an effective point depth value list.

[0030] The second calculation module is configured to divide the effective point depth value list by the effective point number list to obtain the filling depth value of the to-be-filled area, and output a depth image after repairing a hole edge.

[0031] The iteration module is configured to iterate the operations of the first distinguishing module, the second distinguishing module, the normalization module, the first calculation module, the list recording module and the second calculation module to obtain a depth image after completely repairing a hole.

[0032] The present application has the following beneficial effects: the present application can set a repair area of any position and any shape according to requirements, unlike the traditional detection method which can only repair a rectangular area or the whole image; the pixels on the boundary of the detection area are not affected by the pixels outside the detection area and the invalid area, the boundary hole can be accurately repaired, and the distortion of the boundary point cloud is avoided; the point cloud repair speed is fast, and the real-time requirement can be better met. BRIEF DESCRIPTION OF DRAWINGS

[0033] The present application will be further described below with reference to the drawings and embodiments, and the drawings are as follows:

[0034] Figure 1 is a processing flowchart of the point cloud hole repair method based on the original depth image of the present application;

[0035] Figure 2a is an original point cloud image provided by the embodiment of the present application;

[0036] Figure 2b is a point cloud image after point cloud repair by the method;

[0037] Figure 2c is a raw depth image provided by the embodiment of the present application;

[0038] Figure 2d is a depth image after point cloud repair by the method;

[0039] Figure 3 is a module block diagram of the point cloud hole repair system based on the raw depth image of the present application. DETAILED DESCRIPTION

[0040] In order to have a clearer understanding of the technical features, objectives and effects of the present application, the specific embodiments of the present application will be described in detail below. In the following description, it should be understood that the directions or positional relationships indicated by "front", "back", "up", "down", "left", "right", "vertical", "horizontal", "vertical", "horizontal", "top", "bottom", "inner", "outer", "head", "tail" and the like are based on the directions or positional relationships shown in the drawings, constructed and operated in a particular direction, and are only for the convenience of describing the technical solutions, and cannot be understood as indicating that the devices or elements indicated must have a particular direction, therefore, it cannot be understood as a limitation on the present application.

[0041] It should also be noted that, unless otherwise explicitly specified and limited, the terms "mounting", "connecting", "connecting", "fixing", "setting" and the like should be understood broadly, for example, it can be fixedly connected, or it can be detachably connected, or it can be integrated; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two elements or the interaction relationship between two elements. When an element is referred to as "on" or "under" another element, the element can be "directly" or "indirectly" located on the other element, or there can be one or more intervening elements. The terms "first", "second", "third" and the like are only for the convenience of describing the technical solutions, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features, therefore, the features with "first", "second", "third" and the like can be explicitly or implicitly included one or more of the features. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0042] In the following description, specific details such as specific system structures, techniques, etc. are presented for the purpose of explanation, not for the purpose of limitation, so as to thoroughly understand the embodiments of the present application. However, it should be clear to those skilled in the art that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits and methods are omitted to avoid unnecessary details that hinder the description of the present application.

[0043] In the field of industrial automation machine vision, point cloud image repair is used more in image processing and has relatively high requirements, both repair accuracy and real-time requirements. The detection area may be arbitrary or variable, and the filling area supports optional, so it is far from enough to only use a rectangular detection area for processing. Therefore, a detection area of any position and any shape needs to be considered as a processing area. As shown in Figure 2a and Figure 2c , a product with obvious height difference is a product that is prone to point cloud loss and difficult to repair. Because in the imaging process, there is a large area of loss in the backlit area, and there is also a partial area loss at the intersection of two different height planes, there is an obvious hole in the point cloud image.

[0044] First, the area of the point cloud loss region is used to determine the hole area to be filled. When the area is large, the region is considered invalid and does not need to be repaired, and does not affect other hole repairs. When the area is small, it is filled according to the neighborhood information. Therefore, when the image needs to be repaired in any position and any shape region, only the other regions outside the region of interest need to be set as invalid regions.

[0045] In order to realize the influence of invalid information in the neighborhood, a mask image needs to be generated to count the number of valid points in the neighborhood. In combination with the valid point information obtained from the original image, the corresponding relationship of the valid points in the neighborhood can be obtained. Therefore, the filling method is reliable and fast to implement.

[0046] In addition, as shown in Figure 2b and Figure 2d , after processing, the repair area is relatively complete, and the point cloud repair at the junction tends to be smooth. Considering the above factors, a point cloud hole filling method based on a depth image is designed. This method only needs to process the actual detection area and will not be affected by the pixels outside the image and the actual detection area. It can better fill the hole area.

[0047] Specifically, as shown in Figure 1 , the application discloses a point cloud hole repair method based on an original depth image, which can be used to restore the true value of the missing area of the point cloud image. It can not only process images of any shape, but also will not be affected by the image outside or invalid area when processing the image boundary pixels. The method comprises the following steps:

[0048] S1: obtaining an original depth image, a filter kernel width w, a filter kernel height h, and an iteration number;

[0049] S2: distinguishing the original depth image into an effective area and a to-be-processed area by binarization;

[0050] S3: judging the to-be-processed region to be invalid region or to-be-filled region by area;

[0051] S4: setting the pixel value of the valid region to be 1 and the pixel value of the to-be-processed region to be 0 to obtain a normalized image;

[0052] S5: multiplying the normalized image with the original depth image to obtain an effective image;

[0053] S6: taking the pixel point belonging to the to-be-filled region in the normalized image as an anchor point to establish a filter kernel, counting the sum of the pixel values in the filter kernel and recording in the effective point number list; taking the pixel point belonging to the to-be-filled region in the effective image as an anchor point to establish a filter kernel, counting the sum of the pixel values in the filter kernel and recording in the effective point depth value list;

[0054] S7: dividing the effective point depth value list by the effective point number list to obtain the filling depth value of the to-be-filled region, and outputting the depth image after repairing the hole edge.

[0055] S8: iterating the operations of S2 to S7 to obtain a depth image with holes completely repaired.

[0056] Further, in the embodiment, step S1 further comprises:

[0057] obtaining the area of the to-be-filled region, the area of the invalid region and a set value, the set value being greater than the area of the to-be-filled region and less than the area of the invalid region.

[0058] In the embodiment, step S2 further comprises:

[0059] converting the pixel gray value of the valid region to a preset value and converting the pixel gray value of the to-be-processed region to 0. The valid region of any position and any shape can be determined manually by a user or automatically recognized by a computer. In some embodiments, the preset value is 255, and in other embodiments, the preset value can not be limited to 255.

[0060] In the embodiment, step S3 further comprises:

[0061] judging whether the current region area is greater than the set value, if yes, judging to be the invalid region, and if not, judging to be the to-be-filled region.

[0062] In some embodiments, the filter kernel width w and the filter kernel height h are both odd numbers. Preferably, the iteration number is set according to the shape and area of the to-be-filled region and the filter kernel width w and the filter kernel height h to cover all pixel points in the to-be-filled region. Understandably, when the to-be-filled region of the original depth image appears at the junction of different height planes, the filter kernel width w and the filter kernel height h need to be as small as possible, such as for Figure 2aThe original point cloud image and Figure 2c The original depth image is shown, the filter kernel width w and the filter kernel height h can be set to 3, and the iteration number is 6. It can be understood that when the to-be-filled region of the original depth image is a relatively smooth surface, the filter kernel width w and the filter kernel height h can be increased and the iteration number can be reduced, so that the processing time can be shortened and a better processing effect can be ensured, or only the operation of step S6 can be iterated, and finally the entire to-be-filled region is filled.

[0063] Preferably, when the local region of the image needs to be filled, that is, the original depth image does not contain invalid regions, and all regions except the valid region are regarded as to-be-filled regions, step S3 can be omitted, and the time for counting the area of each region in the depth image and the area judgment is saved.

[0064] As shown in Figure 3 The application discloses a point cloud hole repair system based on an original depth image, which comprises:

[0065] An acquisition module is configured to acquire an original depth image, a filter kernel width w, a filter kernel height h, and an iteration number;

[0066] A first distinguishing module is configured to distinguish the original depth image into a valid region and a to-be-processed region through binarization;

[0067] A second distinguishing module is configured to distinguish the to-be-processed region into an invalid region and a to-be-filled region through area judgment;

[0068] A normalization module is configured to set the pixel value of the valid region to 1 and the pixel value of the to-be-processed region to 0 to obtain a normalized image;

[0069] A first calculation module is configured to multiply the normalized image and the original depth image to obtain an effective image;

[0070] A list recording module is configured to take the pixel points belonging to the to-be-filled region in the normalized image as anchor points, establish a filter kernel, count the sum of the pixel values in the filter kernel, and record in an effective point number list; take the pixel points belonging to the to-be-filled region in the effective image as anchor points, establish a filter kernel, count the sum of the pixel values in the filter kernel, and record in an effective point depth value list;

[0071] A second calculation module is configured to divide the effective point depth value list by the effective point number list to obtain the filling depth value of the to-be-filled region, and output a depth image after the hole edge is repaired;

[0072] An iteration module is configured to iterate the operations of the first distinguishing module, the second distinguishing module, the normalization module, the first calculation module, the list recording module, and the second calculation module to obtain a depth image with a completely repaired hole.

[0073] Further, in the embodiment, the acquisition module further comprises:

[0074] An area of the to-be-filled region, an area of the invalid region and a set value are acquired, the set value being greater than the area of the to-be-filled region and less than the area of the invalid region.

[0075] In the embodiment, the first distinguishing module further comprises:

[0076] The pixel gray value of the effective region is converted into a preset value, and the pixel gray value of the to-be-processed region is converted into 0. The effective region of any position and any shape can be determined manually by a user or recognized automatically by a computer. In some embodiments, the preset value is 255, and in other embodiments, the preset value can not be limited to 255.

[0077] In the embodiment, the second distinguishing module further comprises:

[0078] It is determined whether the current region area is greater than the set value, if yes, the current region is determined as the invalid region, and if no, the current region is determined as the to-be-filled region.

[0079] The present application has the following advantages: the present application can set a repair region of any position and any shape according to requirements, unlike the traditional detection method which can only repair a rectangular region or the whole image; the pixels on the boundary of the detection region are not affected by the pixels outside the detection region and the pixels in the invalid region, the boundary holes can be repaired accurately, and the distortion of the boundary point cloud is avoided; the point cloud is repaired quickly, and the real-time requirement can be met well.

[0080] It can be understood that the above embodiments only express the preferred embodiments of the present application, the description is more specific and detailed, but it cannot be understood as the limitation of the patent scope of the present application; it should be pointed out that for the ordinary skilled in the art, the above technical features can be freely combined without departing from the concept of the present application, and some modifications and improvements can be made, which belong to the protection scope of the present application; therefore, any equivalent transformation and modification within the scope of the claims of the present application should belong to the coverage of the claims of the present application.

Claims

1. A method for hole filling of point cloud based on original depth image, characterized in that, The method comprises the following steps: S1: obtaining an original depth image, a filter kernel width w, a filter kernel height h, and an iteration number; The filter kernel width w and the filter kernel height h are both odd numbers; S2: distinguishing the original depth image into an effective region and a to-be-processed region through binarization; S3: distinguishing the to-be-processed region into an invalid region and a to-be-filled region through area judgment; setting the iteration number according to the shape and area of the to-be-filled region and the filter kernel width w and the filter kernel height h, so as to cover all pixel points in the to-be-filled region; S4: setting the pixel value of the effective region as 1 and the pixel value of the to-be-processed region as 0 to obtain a normalized image; S5: multiplying the normalized image with the original depth image to obtain an effective image; S6: taking the pixel points belonging to the to-be-filled region in the normalized image as anchor points to establish the filter kernel, counting the sum of pixel values in the filter kernel, and recording into an effective point number list; Taking the pixel points belonging to the to-be-filled region in the effective image as anchor points to establish the filter kernel, counting the sum of pixel values in the filter kernel, and recording into an effective point depth value list; S7: dividing the effective point depth value list by the effective point number list to obtain the filling depth value of the to-be-filled region, and outputting a depth image after repairing the hollow edge; S8: iterating the operations of S2 to S7 to obtain a depth image with completely repaired hollows.

2. The method of claim 1, wherein, The step S1 further comprises: obtaining the area of the to-be-filled region, the area of the invalid region, and a setting value, the setting value being greater than the area of the to-be-filled region and smaller than the area of the invalid region.

3. The method of claim 2, wherein, The step S2 further comprises: converting the pixel gray value of the effective region into a preset value and converting the pixel gray value of the to-be-processed region into 0.

4. The method of claim 3, wherein, The preset value is 255.

5. The method of claim 3, wherein the method further comprises: The step S3 further comprises: judging whether the current region area is greater than the setting value, if yes, judging as the invalid region, and if not, judging as the to-be-filled region.

6. The method of claim 1, wherein, When the original depth image does not contain the invalid region, the step S3 is omitted.

7. A system for original depth image based point cloud hole filling, which implements the method of any one of claims 1 to 6. The point cloud hollow repairing system based on an original depth image comprises: an obtaining module, configured to obtain an original depth image, a filter kernel width w, a filter kernel height h, and an iteration number; the filter kernel width w and the filter kernel height h are both odd numbers; a first distinguishing module, configured to distinguish the original depth image into an effective region and a to-be-processed region through binarization; a second distinguishing module, configured to distinguish the to-be-processed region into an invalid region and a to-be-filled region through area judgment; setting the iteration number according to the shape and area of the to-be-filled region and the filter kernel width w and the filter kernel height h, so as to cover all pixel points in the to-be-filled region; a normalization module, configured to set the pixel value of the effective region as 1 and the pixel value of the to-be-processed region as 0 to obtain a normalized image; a first calculating module, configured to multiply the normalized image with the original depth image to obtain an effective image; a second calculating module, configured to take the pixel points belonging to the to-be-filled region in the normalized image as anchor points to establish the filter kernel, count the sum of pixel values in the filter kernel, and record into an effective point number list; The list recording module is configured to take a pixel point belonging to the region to be filled in the normalized image as an anchor point, establish the filter kernel, count a sum of pixel values in the filter kernel, and record in an effective point number list; take a pixel point belonging to the region to be filled in the effective image as an anchor point, establish the filter kernel, count a sum of pixel values in the filter kernel, and record in an effective point depth value list; The second calculation module is configured to divide the effective point depth value list by the effective point number list to obtain a filling depth value of the region to be filled, and output a depth image after a hole edge is repaired. The iteration module is configured to iterate operations of the first division module, the second division module, the normalization module, the first calculation module, the list recording module, and the second calculation module to obtain a depth image after a hole is completely repaired.

Images