Three-dimensional point cloud invalid value filling method, system and storage medium

Abstract

The present invention provides a method, system, and storage medium for filling invalid values ​​in a three-dimensional point cloud, comprising the following steps: S1: obtaining a point cloud image S and a valid value range of the point cloud, and mapping the point cloud image S to obtain a corresponding depth map; S2: preprocessing the depth map according to the valid value range of the point cloud to obtain a weighted image W, and calculating a gain image C based on the weighted image W; S3: invalidating the weights corresponding to the invalid values ​​in the depth map to obtain an image G; S4: multiplying the pixels corresponding to image G and gain image C to obtain a filled image D; S5: determining whether the filled image D has reached the number of iterations; if not, replacing the depth map with the filled image D and returning to step S2; if so, outputting the filled image D as the final result image. The present invention can fill holes based on trend information of the neighborhood, and the filled value still maintains the original trend. The workpiece does not need to be flat, and can automatically fill with an adaptive value without setting a fill value.

Description

Technical Field

[0001] The present invention relates to the field of industrial automation machine vision technology, and in particular to a method, system and storage medium for filling invalid values ​​in a three-dimensional point cloud. Background Art

[0002] In the field of industrial automation machine vision, particularly 3D defect detection, the goal is to detect defects in designated areas on a workpiece. However, due to the limitations of 3D laser sensors, when scanning the workpiece, out-of-range height, reflections, or occlusions can cause point cloud loss. Lost points are typically replaced with invalid values. These invalid values ​​affect defect detection. Common 3D laser sensors typically use very small values ​​to represent invalid values, which can be detected as deep pits, resulting in false positives. Traditional vision methods, which fill in fixed values, only work when the workpiece is flat. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a method for filling invalid values ​​in a three-dimensional point cloud in response to the above-mentioned defects of the prior art, which is characterized by comprising the following steps:

[0004] S1: Obtain a point cloud image S and a valid value range of the point cloud, and map the point cloud image S to obtain a corresponding depth map;

[0005] S2: Preprocessing the depth map according to the valid value range of the point cloud to obtain a weight and image W, and calculating a gain image C based on the weight and image W;

[0006] S3: Invalidate the weights corresponding to the invalid values ​​of the depth map to obtain an image G;

[0007] S4: Multiplying the pixels corresponding to the image G and the gain image C to obtain an image D;

[0008] S5: Determine whether the filling image D reaches the number of iterations. If not, replace the depth map with the filling image D and return to step S2; if so, output the filling image D and use it as the final result image.

[0009] Preferably, the valid value range of the point cloud is the valid height value range of the device used to obtain the point cloud image S.

[0010] Preferably, in the three-dimensional point cloud invalid value filling method of the present invention, step S2 includes:

[0011] S21: Binarizing the depth map according to the valid value range of the point cloud to obtain a valid binary image R;

[0012] S22: Filter the image R to obtain a weighted image W;

[0013] S23: 1 is divided by the weight sum image W to obtain the gain image C.

[0014] Preferably, in the three-dimensional point cloud invalid value filling method of the present invention, step S21 includes:

[0015] The height value range corresponding to the pixel value of the valid binary image R is determined according to the valid value range of the point cloud. If the pixel value is 1, it means that the corresponding height value is a valid value; if the pixel value is 0, it means that the corresponding height value is an invalid value.

[0016] Preferably, in the three-dimensional point cloud invalid value filling method of the present invention, step S3 includes:

[0017] S31: Setting invalid values ​​of the depth map to zero to obtain an image F;

[0018] S32: Filter the image F to obtain the image G.

[0019] Preferably, in the three-dimensional point cloud invalid value filling method of the present invention, step S31 includes:

[0020] A valid pixel value range of the depth map is preset, and invalid values ​​of the depth map are determined according to the valid pixel value range.

[0021] Preferably, the filtering process includes performing filtering process using a Gaussian filtering function or a mean filtering function.

[0022] The present invention also constructs a three-dimensional point cloud invalid value filling system, comprising:

[0023] A depth map acquisition unit is used to acquire a point cloud image S and a valid value range of the point cloud and map the point cloud image S to obtain a corresponding depth map;

[0024] a gain image acquisition unit, configured to pre-process the depth map according to a valid value range of the point cloud and calculate a position weight of a center point of the depth map to obtain a gain image C;

[0025] an invalidation processing unit, configured to invalidate weights corresponding to invalid values ​​of the depth map to obtain an image G;

[0026] a filling unit, configured to multiply the pixel points corresponding to the image G and the gain image C to obtain a filled image D;

[0027] The iteration judgment unit is used to judge whether the filling image D reaches the iteration number. If not, the depth map is replaced by the filling image D and the process returns to step S2. If so, the filling image D is output as the final result map.

[0028] Preferably, the present invention further includes an electronic device comprising a processor and a memory storing computer-readable instructions, wherein the processor is configured to execute any one of the above methods when executing the computer-readable instructions.

[0029] Preferably, the present invention further includes a computer-readable storage medium having a computer program stored thereon, which implements any of the above methods when executed by a processor.

[0030] The beneficial effects of the present invention are as follows: the present invention can fill holes according to the trend information of the neighborhood, the filling value still maintains the original trend and the workpiece does not need to be laid flat, and the adaptive value can be automatically filled without setting the filling value. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0032] Figure 1 This is a flow chart of a method for filling invalid values ​​in a three-dimensional point cloud according to the present invention;

[0033] Figure 2(a) is the point cloud image S, Figure 2(b) is the result image processed by the traditional visual method, and Figure 2(c) is the final processing result image of this method;

[0034] Figure 3(a) is the effective binary image R obtained by binarizing the point cloud image, Figure 3(b) is a local enlarged view of Figure 3(a), Figure 3(c) is the weighted sum image W obtained by filtering 3(b), Figure 3(d) is a local enlarged view of Figure 2(a), Figure 3(e) is the image F obtained by setting the invalid value area to 0, Figure 3(f) is the image G obtained by filtering the image in Figure 3(e), Figure 3(g) is the final filling result image (i.e., image D), Figure 3(h) is the gain image C, and Figure 3(I) is the Gaussian filter kernel weight map;

[0035] Figure 4 It is a block diagram of a three-dimensional point cloud invalid value filling system of the present invention. DETAILED DESCRIPTION

[0036] In order to have a clearer understanding of the technical features, purposes and effects of the present invention, specific embodiments of the present invention are now described in detail with reference to the accompanying drawings.

[0037] It should be noted that the flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all content and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, while others may be combined or partially combined. Therefore, the actual execution order may vary depending on the actual situation.

[0038] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically separate entities. That is, these functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0039] In the field of industrial automation machine vision, the demand for product defect detection is relatively high, and the scenarios are relatively complex. Due to the limitations of 3D laser sensors, the generated point cloud images often contain invalid values. Traditional methods detect invalid value areas and then fill them with a fixed value, which is generally obtained by calculating the mean of the effective values ​​of the entire image. As shown in Figure 2(a), a common point cloud image, the entire image is filled with black holes. These black holes are filled with invalid values. This is due to the reflection of the 3D laser sensor on the metal surface. The traditional vision method fills these black holes with fixed values, which are selected based on the mean of the effective values ​​of the entire image. As shown in Figure 2(b), even though the holes are filled, the holes are uneven, and there is a clear boundary between the filled value and the outer value. This is because the workpiece is not flat, and the calculated mean value cannot adapt to this situation. Taking these factors into consideration, the present invention provides a method for filling invalid values ​​in 3D point clouds. As shown in Figure 2(c), this method uses a Gaussian filter to mask the influence of invalid values ​​and extracts the value near the filter kernel as the filling value of the current pixel. This method has the following advantages:

[0040] 1. Fill in the values ​​that match the trend based on the high and low trends of nearby valid values.

[0041] 2. The workpiece does not need to be placed flat, and the adaptive value can be automatically filled without setting the filling value.

[0042] Specifically, if Figure 1 As shown, the present invention discloses a method for filling invalid values ​​in a 3D point cloud. As a preprocessing algorithm for 3D defect detection, it is widely used in the field of defect detection, such as power battery weld detection, battery electrode defect detection, FPCB defect detection, cable defect detection, etc. The method includes the following steps:

[0043] S1: Obtain the point cloud image S as shown in Figure 2(a) and the valid value range of the point cloud, and map the point cloud image S to obtain the corresponding depth map;

[0044] Figure 3(d) is a partial enlarged view of Figure 2(a). As can be seen from Figures 2(a) and 3(d), the effective value range of the point cloud is the effective height range of the device used to obtain the point cloud image S. In this embodiment, the device is a three-dimensional laser sensor, so the effective value range of the point cloud is the effective height range of the three-dimensional laser sensor.

[0045] S2: Preprocess the depth map according to the valid value range of the point cloud to obtain the weight and image W as shown in Figure 3(c), and calculate the gain image C based on the weight and image W as shown in Figure 3(h);

[0046] S3: Invalidate the weights corresponding to the invalid values ​​of the depth map to obtain the image G as shown in Figure 3(f);

[0047] S4: Multiply the corresponding pixels of image G and gain image C to obtain the filled image D as shown in Figure 3(g);

[0048] S5: Determine whether the filling image D reaches the number of iterations. If not, replace the depth map with the filling image D and return to step S2; if so, output the filling image D and use it as the final result image.

[0049] In this embodiment, step S2: preprocessing the depth map according to the valid value range of the point cloud to obtain a weight and image W, and calculating the gain image C based on the weight and image W; further, step S2 includes:

[0050] S21: Binarize the depth map according to the valid value range of the point cloud to obtain a valid binary image R as shown in Figure 3(a);

[0051] S22: Filter the image R to obtain a weighted image W;

[0052] S23: 1 is divided by the weight sum image W to obtain the gain image C.

[0053] In some embodiments, the filtering process includes filtering using a Gaussian filter function or a mean filter function. The filtering process uses a Gaussian filter kernel as the basic filter kernel. FIG3(I) is a Gaussian filter kernel weight map, where the weight at the center of the Gaussian filter kernel is high, and the weight away from the center is low. When Gaussian filtering is performed on image F to obtain image G, the values ​​at the boundary between valid values ​​and invalid values ​​will be pulled down. In order to bring the boundary values ​​back to be close to the values ​​at the corresponding positions in image S, it is necessary to multiply the pixel values ​​at the corresponding positions in image G by a gain coefficient. Step S23 is to calculate this gain coefficient.

[0054] Furthermore, step S21 includes:

[0055] The height value range corresponding to the valid binary image R pixel value is determined according to the valid value range of the point cloud. If the pixel value is 1, it means that the corresponding height value is a valid value; if the pixel value is 0, it means that the corresponding height value is an invalid value.

[0056] In this embodiment, step S3: invalidating the weights corresponding to the invalid values ​​of the depth map S to obtain the image G; further, step S3 includes:

[0057] S31: Setting the invalid values ​​of the depth map to zero to obtain an image F as shown in FIG3(e);

[0058] S32: Filter image F to obtain image G.

[0059] The filtering process includes using a Gaussian filter function or a mean filter function for filtering. Steps S31 and S32 are for considering only valid values ​​during filtering, while invalid values ​​are not involved in the calculation, that is, shielding the influence of invalid values.

[0060] Furthermore, step S31 includes:

[0061] A valid pixel value range of the depth map is preset, and invalid values ​​of the depth map are determined according to the valid pixel value range.

[0062] like Figure 4 As shown, the present invention discloses a three-dimensional point cloud invalid value filling system, comprising:

[0063] The depth map acquisition unit is used to obtain the point cloud image S and the valid value range of the point cloud, and obtain the corresponding depth map according to the mapping of the point cloud image S;

[0064] The effective value range of the point cloud is the effective height value range of the device used to obtain the point cloud image S. In this embodiment, the device is a three-dimensional laser sensor, so the effective value range of the point cloud is the effective height value range of the three-dimensional laser sensor.

[0065] A gain image acquisition unit is used to pre-process the depth map according to the valid value range of the point cloud to obtain a weight and image W, and calculate the gain image C based on the weight and image W;

[0066] an invalidation processing unit, configured to invalidate weights corresponding to invalid values ​​of the depth map to obtain an image G;

[0067] A filling unit, configured to multiply corresponding pixels of the image G and the gain image C to obtain a filled image D;

[0068] The iteration judgment unit is used to judge whether the filling image D reaches the iteration number. If not, the depth map is replaced by the filling image D and the process returns to step S2. If so, the filling image D is output as the final result map.

[0069] In this embodiment, the gain image acquisition unit is configured to preprocess the depth map according to the valid value range of the point cloud to obtain a weight sum image W, and calculate a gain image C based on the weight sum image W. Accordingly, obtaining the gain image C includes:

[0070] The depth map is binarized according to the valid value range of the point cloud to obtain a valid binary image R; further, it also includes judging the height value range corresponding to the pixel value of the valid binary image R according to the valid value range of the point cloud, if the pixel value is 1, it indicates that the corresponding height value is a valid value; if the pixel value is 0, it indicates that the corresponding height value is an invalid value.

[0071] The image R is filtered to obtain a weighted sum image W; 1 is divided by the weighted sum image W to obtain a gain image C.

[0072] Furthermore, the filtering process includes filtering using a Gaussian filter function or a mean filter function. The filtering process uses a Gaussian filter kernel as the basic filter kernel, with high weights at the center of the Gaussian filter kernel and low weights away from the center. When Gaussian filtering is performed on image F to obtain image G, the values ​​at the boundary between valid and invalid values ​​are lowered. In order to bring the values ​​at the boundary back to be close to the values ​​at the corresponding positions in image S, the pixel values ​​at the corresponding positions in image G are multiplied by a gain coefficient. The result is a gain image C obtained by dividing 1 by the weight and image W.

[0073] In this embodiment, the invalidation processing unit is used to invalidate the weights corresponding to the invalid values ​​of the depth map to obtain the image G; the acquisition of the image G includes:

[0074] The invalid values ​​of the depth map are set to zero to obtain an image F; and the image F is filtered to obtain the image G. The filtering process includes using a Gaussian filter function or a mean filter function to perform filtering. The purpose is to only consider valid values ​​during the filtering process, and invalid values ​​are not involved in the calculation, that is, to shield the influence of invalid values.

[0075] Furthermore, setting the invalid value of the depth map to zero to obtain the image F includes: presetting a valid pixel value range of the depth map, and determining the invalid value of the depth map according to the valid pixel value range.

[0076] The present invention also includes an electronic device, which includes a processor and a memory storing computer-readable instructions. The processor is configured to execute the three-dimensional point cloud invalid value filling method disclosed in an embodiment of the present invention when executing the computer-readable instructions.

[0077] The present invention also provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the method for filling invalid values ​​of a three-dimensional point cloud disclosed in an embodiment of the present invention is implemented.

[0078] The implementation of the three-dimensional point cloud invalid value filling method of the present invention has at least the following beneficial effects:

[0079] 1. Fill in the values ​​that match the trend based on the high and low trends of nearby valid values.

[0080] 2. The workpiece does not need to be placed flat, and the adaptive value can be automatically filled without setting the filling value.

[0081] It is understandable that the above embodiments only express the preferred implementation modes of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the patent scope of the present invention. It should be pointed out that for ordinary technicians in this field, without departing from the concept of the present invention, the above technical features can be freely combined, and several deformations and improvements can be made, all of which fall within the scope of protection of the present invention. Therefore, all equivalent changes and modifications made to the scope of the claims of the present invention should fall within the scope of coverage of the claims of the present invention.

Claims

1. A method for filling invalid values ​​in a three-dimensional point cloud, characterized in that: The following steps are involved: S1: Obtain a point cloud image S and a valid value range of the point cloud, and map the point cloud image S to obtain a corresponding depth map; S2: Preprocessing the depth map according to the valid value range of the point cloud to obtain a weight and image W, and calculating a gain image C based on the weight and image W; The step S2 comprises: S21: Binarizing the depth map according to the valid value range of the point cloud to obtain a valid binary image R; S22: Filter the image R to obtain a weighted image W; S23: 1 is divided by the weight and image W to obtain the gain image C; S3: Invalidate the weights corresponding to the invalid values ​​of the depth map to obtain an image G; The step S3 comprises: S31: Setting invalid values ​​of the depth map to zero to obtain an image F; S32: Filter the image F to obtain the image G; S4: Multiplying the pixel points corresponding to the image G and the gain image C to obtain a filled image D; S5: Determine whether the filling image D reaches the number of iterations. If not, replace the depth map with the filling image D and return to step S2; if so, output the filling image D and use it as the final result image.

2. The method for filling invalid values ​​of a three-dimensional point cloud according to claim 1, characterized in that: The valid value range of the point cloud is the valid height value range of the device used to obtain the point cloud image S.

3. The method for filling invalid values ​​of a three-dimensional point cloud according to claim 1, characterized in that: The step S21 includes: The height value range corresponding to the pixel value of the valid binary image R is determined according to the valid value range of the point cloud. If the pixel value is 1, it means that the corresponding height value is a valid value; if the pixel value is 0, it means that the corresponding height value is an invalid value.

4. The method for filling invalid values ​​of a three-dimensional point cloud according to claim 1, characterized in that: The step S31 includes: A valid pixel value range of the depth map is preset, and invalid values ​​of the depth map are determined according to the valid pixel value range.

5. The method for filling invalid values ​​of a three-dimensional point cloud according to claim 1, characterized in that: The filtering process includes using a Gaussian filtering function or a mean filtering function to perform filtering process.

6. A three-dimensional point cloud invalid value filling system, characterized in that: include: A depth map acquisition unit is used to acquire a point cloud image S and a valid value range of the point cloud, and obtain a corresponding depth map according to the mapping of the point cloud image S; a gain image acquisition unit, configured to preprocess the depth map according to a valid value range of the point cloud to obtain a weight sum image W, and calculate a gain image C based on the weight sum image W; Obtaining the gain image C includes: binarizing the depth map according to the valid value range of the point cloud to obtain a valid binary image R; filtering the image R to obtain a weighted sum image W; dividing 1 by the weighted sum image W to obtain the gain image C; an invalidation processing unit, configured to invalidate weights corresponding to invalid values ​​of the depth map to obtain an image G; The acquisition of the image G includes: setting the invalid values ​​of the depth map to zero to obtain the image F; performing filtering processing on the image F to obtain the image G; a filling unit, configured to multiply the pixel points corresponding to the image G and the gain image C to obtain a filled image D; The iteration judgment unit is used to judge whether the filling image D reaches the iteration number. If not, the depth map is replaced by the filling image D and returned to the gain image acquisition unit; if so, the filling image D is output and used as the final result image.

7. An electronic device, characterized in that: The invention comprises a processor and a memory storing computer-readable instructions, wherein the processor is configured to perform the method according to any one of claims 1 to 5 when executing the computer-readable instructions.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images