A method and system for removing invalid points in structured light three-dimensional measurement

Abstract

The invalid point removal method and system for structured light three-dimensional measurement provided by the present application obtain the wrapping phase, background intensity and modulation intensity of the fringe pattern; obtain the discreteness between the sequence points of each fringe pattern coordinate and the ideal cosine curve based on the wrapping phase, background intensity and modulation intensity; correct point by point and establish a new error energy function based on the discreteness; and remove invalid points based on the new error energy function. The invalid point removal method and system for structured light three-dimensional measurement provided by the present application are easy to implement and can be used for invalid point removal tasks of complex objects in different scenarios. They are not only insensitive to the reflectivity or modulation of the object, but also effectively solve the problem of difficult edge segmentation of the measured object.

Description

Technical Field

[0001] The present application relates to the field of digital image processing technology, and in particular to a method and system for removing invalid points in structured light three-dimensional measurement. Background Art

[0002] Structured light 3D measurement is currently the most widely used 3D visual measurement technology due to its advantages such as simplicity, speed, high accuracy, and applicability to targets with weak textures. However, due to the limitations of the optical imaging mechanisms of projectors and cameras and the influence of image noise, the captured fringe pattern inevitably contains invalid points such as shadows and background, which seriously affects the post-processing of the reconstructed point cloud.

[0003] In order to ensure the quality of three-dimensional reconstruction data, these invalid points must be identified and removed. In recent years, researchers have proposed some methods for invalid point identification and removal in fringe projection profilometry. Existing invalid point removal methods are mainly divided into two categories: those based on the inherent characteristics of the unfolded phase and those based on the threshold design of the modulated image. Among them, the threshold design based on the modulated image only uses a suitable threshold to regard pixels with a low-key system as invalid points. The method has the characteristics of simplicity of implementation and does not require additional fringe frequencies. However, in optical measurement, there are complex optical phenomena such as diffraction or interference at the edges of objects, and the intensity modulation at the edges will vary greatly. In existing methods, when the intensity modulation of the object surface is small or the modulation intensity is generated by a low-quality fringe pattern, some effective modulation and invalid modulation will be intertwined with each other, so it is difficult to achieve an ideal segmentation effect with only a single modulation threshold. Summary of the Invention

[0004] In view of this, it is necessary to provide a method and system for removing invalid points in structured light three-dimensional measurement with ideal segmentation effect to address the defects in the prior art.

[0005] To solve the above problems, this application adopts the following technical solutions:

[0006] One of the purposes of this application is to provide a method for removing invalid points in structured light 3D measurement, comprising the following steps:

[0007] Obtain the wrapped phase, background intensity, and modulation intensity of the fringe pattern;

[0008] Obtaining the discreteness between the sequence points of each fringe pattern coordinate and the ideal cosine curve according to the wrapping phase, background intensity and modulation intensity;

[0009] Correcting point by point and establishing a new error energy function according to the discreteness;

[0010] Invalid points are removed according to the new error energy function.

[0011] In some embodiments, the step of obtaining the wrapping phase, background intensity, and modulation intensity of the fringe pattern specifically includes the following steps:

[0012] Project a series of standard cosine fringe images onto the object to be measured;

[0013] The height of the surface of the object being measured modulates the cosine fringes;

[0014] The camera captures a sequence of fringe patterns in the order of projection, and expresses the collected deformation fringes as the following formula:

[0015]

[0016] Use the phase shift method to calculate the wrapped phase Background intensity and modulation intensity

[0017]

[0018]

[0019]

[0020] in, is any coordinate point (x C ,y C ), k=0,1,...,N-1, N is the total number of phase shifts, the wrapped phase The inverse tangent function is wrapped in the range [-π,π) with a 2π discontinuity, and the wrapped phase is unfolded by the phase unwrapping formula, the mathematical formula of which is:

[0021]

[0022] Among them, K(x C ,y C ) represents the fringe order diagram.

[0023] In some embodiments, in the step of projecting a series of standard cosine fringe images onto the object to be measured, the standard cosine fringe images are acquired using a 4-step phase shift method.

[0024] In some embodiments, in the step of obtaining the discreteness between the serial points of each fringe pattern coordinate and the ideal cosine curve according to the package phase, background intensity and intensity modulation, specifically: using weighted Gaussian Euclidean distance to obtain the discreteness between the serial points of each fringe pattern coordinate and the ideal cosine curve according to the package phase, background intensity and modulation intensity.

[0025] In some embodiments, the weighted Gaussian Euclidean distance is used to obtain the discreteness between the sequence points of each fringe pattern coordinate and the ideal cosine curve according to the package phase, background intensity and modulation intensity, specifically including the following steps:

[0026] The background intensity and the modulation intensity are used to normalize the pixel sequence points, and the relationship between the normalized sequence value and the ideal cosine curve in a single cycle is obtained;

[0027] The Gaussian weighted Euclidean distance is used to quantify the discreteness between the sequence point and the ideal cosine curve as error energy, which is used to represent the cosine property of the same frequency pixel sequence point. The formula is as follows:

[0028]

[0029]

[0030]

[0031] Among them, Error(x C ,y C ) is the error energy, w i is the Gaussian weight, W is the normalization factor, error k (x C ,y C ) is the deviation of the pixel from the ideal cosine curve.

[0032] In some embodiments, the step of correcting and establishing a new error energy function point by point according to the discreteness specifically includes correcting and establishing a new error energy function point by point using an improved Gaussian filtering method based on modulation intensity.

[0033] In some embodiments, the new error energy function is as follows:

[0034] E(x C ,y C )=EG(x C ,y C )×M(x C ,y C )

[0035]

[0036]

[0037] Among them, E(x C ,y C ) is the new error energy, Gau is a two-dimensional Gaussian window, and λ determines the weighting factor M(x C ,y C ) intensity.

[0038] In some embodiments, the step of removing invalid points according to the new error energy function specifically includes the following steps:

[0039] Choose a suitable error threshold T error ;

[0040] By setting all the values ​​less than the threshold T error The points are marked as 1 and the remaining points are marked as 0, thus obtaining a mask template MASK(x C ,y C ), as follows:

[0041] The mask template is used to remove invalid points in the background and shadow, where the pixel is in the mask template MASK(x C ,y C ) are considered as valid points.

[0042] A second object of this application is to provide an invalid point removal system for structured light 3D measurement, comprising:

[0043] Fringe pattern acquisition unit: used to obtain the wrapped phase, background intensity and modulation intensity of the fringe pattern;

[0044] A discreteness construction unit is used to obtain the discreteness between the sequence points of each fringe pattern coordinate and the ideal cosine curve according to the package phase, background intensity and modulation intensity;

[0045] An error energy function construction unit is used to correct and establish a new error energy function point by point according to the discreteness;

[0046] Invalid point removal unit: used for removing invalid points according to the new error energy function.

[0047] This application adopts the above technical solution, and its beneficial effects are as follows:

[0048] The invalid point removal method and system for structured light three-dimensional measurement provided by the present application obtain the wrapping phase, background intensity and modulation intensity of the fringe pattern; obtain the discreteness between the sequence points of each fringe pattern coordinate and the ideal cosine curve based on the wrapping phase, background intensity and modulation intensity; correct point by point and establish a new error energy function based on the discreteness; and remove invalid points based on the new error energy function. The invalid point removal method and system for structured light three-dimensional measurement provided by the present application are easy to implement and can be used for invalid point removal tasks of complex objects in different scenarios. They are not only insensitive to the reflectivity or modulation of the object, but also effectively solve the problem of difficult edge segmentation of the measured object. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments of the present application or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0050] Figure 1 This is a flowchart of the steps of the method for removing invalid points in structured light three-dimensional measurement provided in Example 1 of the present application.

[0051] Figure 2 A schematic diagram of a common type of a group of sequential points on the captured stripe image provided in Application Example 1;

[0052] Figure 3 A schematic diagram of the error energy calculated by the Gaussian weighted Euclidean distance provided in Application Example 1;

[0053] Figure 4 The new error energy map and its segmentation results provided in application embodiment 1.

[0054] Figure 5 This is a schematic diagram of the structure of the invalid point removal system for structured light three-dimensional measurement provided in Example 2 of the present application. DETAILED DESCRIPTION

[0055] The following describes in detail embodiments of the present application, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0056] In the description of this application, it should be understood that the terms "upper", "lower", "horizontal", "inside", "outside", etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing this application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on this application.

[0057] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. Throughout the description of this application, "plurality" means two or more, unless otherwise specifically defined.

[0058] In order to make the purpose, technical solutions and advantages of this application more clear, this application is further described in detail below with reference to the accompanying drawings and embodiments.

[0059] Example 1

[0060] See also Figure 1 , which is a flowchart of a method for removing invalid points in structured light 3D measurement provided in Example 1, including the following steps:

[0061] Step S110: Obtaining the wrapping phase, background intensity, and modulation intensity of the fringe pattern.

[0062] In some embodiments, the step of obtaining the wrapping phase, background intensity, and modulation intensity of the fringe pattern specifically includes the following steps:

[0063] Step S111: Projecting a series of standard cosine fringe images onto the object to be measured.

[0064] In this embodiment, a digital light processing (DLP) projector projects a standard cosine fringe pattern onto the surface of an object. The standard cosine fringe pattern is acquired using a four-step phase shift method.

[0065] Step S112: modulating the cosine fringes according to the height of the surface of the object being measured;

[0066] Step S113: The camera captures a set of fringe pattern sequences in the order of projection, and expresses the collected deformation fringes as the following formula:

[0067]

[0068] It is understandable that in structured light 3D measurement, the projector is usually the main light source in the measurement environment. Therefore, the intensity variation of the pixels on the camera (CMOS) image plane is mainly affected by the structured light reflected from various areas within the camera field of view.

[0069] Step S114: Calculate and obtain the package phase using the phase shift method Background intensity and modulation intensity

[0070]

[0071]

[0072]

[0073] in, is any coordinate point (x C ,y C), k=0,1,...,N-1, N is the total number of phase shifts, the wrapped phase The inverse tangent function is wrapped in the range [-π,π) with a 2π discontinuity, and the wrapped phase is unfolded by the phase unwrapping formula, the mathematical formula of which is:

[0074]

[0075] Among them, K(x C ,y C ) represents the fringe order diagram.

[0076] See also Figure 2 (a) shows four fringe patterns acquired by a CMOS camera using the 4-step phase shift method as an example. There are four main types of sequence points on the pattern.

[0077] Step S120: obtaining the discreteness between the sequence points of each fringe pattern coordinate and the ideal cosine curve according to the wrapping phase, background intensity and modulation intensity.

[0078] It is understandable that since the background intensity and modulation intensity in different areas of the camera field of view are different due to the reflectivity of the object being measured and the ambient light, the sequence points in different areas need to be converted to the same scale for analysis.

[0079] In this embodiment, in the step of obtaining the discreteness between the serial points of each fringe pattern coordinate and the ideal cosine curve according to the package phase, background intensity and intensity modulation, specifically: using weighted Gaussian Euclidean distance to obtain the discreteness between the serial points of each fringe pattern coordinate and the ideal cosine curve according to the package phase, background intensity and modulation intensity.

[0080] Furthermore, the weighted Gaussian Euclidean distance is used to obtain the discreteness between the sequence points of each fringe pattern coordinate and the ideal cosine curve according to the package phase, background intensity and modulation intensity, which specifically includes the following steps:

[0081] Step S121: the background intensity and the modulation intensity are used to normalize the pixel sequence points, and the relationship between the normalized sequence value and the ideal cosine curve in a single cycle is obtained.

[0082] See also Figure 2 As shown in (b), the background intensity and modulation intensity normalize the pixel sequence points, and the relationship between the normalized sequence value and the ideal cosine curve in a single cycle is obtained.

[0083] Step S122: using Gaussian weighted Euclidean distance to quantify the discreteness between the sequence point and the ideal cosine curve as error energy, which is used to represent the cosine property of the same-frequency pixel sequence point. The formula is as follows:

[0084]

[0085]

[0086]

[0087] Among them, Error(x C ,y C ) is the error energy, w i is the Gaussian weight, W is the normalization factor, error k (x C ,y C ) is the deviation of the pixel from the ideal cosine curve.

[0088] Step S130: Correcting point by point according to the discreteness and establishing a new error energy function.

[0089] In some embodiments, the step of correcting and establishing a new error energy function point by point according to the discreteness specifically includes correcting and establishing a new error energy function point by point using an improved Gaussian filtering method based on modulation intensity.

[0090] Furthermore, the new error energy function is as follows:

[0091] E(x C ,y C )=EG(x C ,y C )×M(x C ,y C )

[0092]

[0093]

[0094] Among them, E(x C ,y C ) is the new error energy, Gau is a two-dimensional Gaussian window, and λ determines the weighting factor M(x C ,y C ) intensity.

[0095] See also Figure 3 (a) and (b) show that the error energy between valid points and invalid points has a good degree of distinction. In actual measurement, when the changes of some invalid sequence points are also similar to some cosine changes, such as Figure 3 The error energy obtained by quantizing the Gaussian weighted Euclidean distance of these points shown in (c) and (d) is similar to the error energy obtained by valid sequence points. Based on the prior knowledge that invalid points are usually concentrated and often have low-intensity modulation, an improved Gaussian filtering method based on modulation intensity is used to establish a new error energy function, such as Figure 4 (a).

[0096] It can be understood that the improved Gaussian filtering method uses neighborhood error energy to compensate for its own error energy. While the error energy of invalid points in the background / shadow area can be significantly affected by the neighborhood error energy, this function does not change the error energy distinction between invalid and valid points in the valid area. A weighting factor based on modulation intensity is used to correct the calculated error energy function based on the prior knowledge that invalid pixels often have low modulation information on low-quality stripes in structured light measurement technology.

[0097] Step S140: removing invalid points according to the new error energy function.

[0098] In some embodiments, the step of removing invalid points according to the new error energy function specifically includes the following steps:

[0099] Step S141: Select a suitable error threshold T error ;

[0100] Step S142: By error The points are marked as 1 and the remaining points are marked as 0, thus obtaining a mask template MASK(x C ,y C ), as follows:

[0101] Step S143: Remove invalid points in the background and shadow by using the mask template, where the pixel is in the mask template MASK(x C ,y C ) are considered as valid points.

[0102] See also Figure 4 (b) The mask template is used to remove invalid points in the background and shadow, where the pixel is in the mask template MASK(x C ,y C ) are considered as valid points. Use the mask template MASK(x C ,y C ) removes a large number of invalid points in the final generated 3D point cloud model, such as Figure 4 (c).

[0103] The invalid point removal method for structured light three-dimensional measurement provided in Example 1 of the present application is easy to implement and can be used for invalid point removal tasks of complex objects in different scenarios. It is not only insensitive to the reflectivity or modulation of the object, but also effectively solves the problem of difficult edge segmentation of the measured object.

[0104] Example 2

[0105] See also Figure 5 Embodiment 2 of the present application provides an invalid point removal system for structured light three-dimensional measurement, including: a fringe pattern acquisition unit 110: used to obtain the wrapping phase, background intensity and modulation intensity of the fringe pattern; a discreteness construction unit 120: used to obtain the discreteness between the sequence points of each fringe pattern coordinate and the ideal cosine curve according to the wrapping phase, background intensity and modulation intensity; an error energy function construction unit 130: used to correct point by point according to the discreteness and establish a new error energy function; an invalid point removal unit 140: used to remove invalid points according to the new error energy function.

[0106] The invalid point removal system for structured light three-dimensional measurement provided in Example 2 of the present application can be implemented in detail by referring to Example 1 and will not be described in detail here.

[0107] The invalid point removal system for structured light three-dimensional measurement provided in Example 2 of the present application is easy to implement and can be used for invalid point removal tasks of complex objects in different scenarios. It is not only insensitive to the reflectivity or modulation of the object, but also effectively solves the problem of difficult edge segmentation of the measured object.

[0108] It can be understood that the various technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the various technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0109] The above are merely preferred embodiments of the present application and only specifically describe the technical principles of the present application. These descriptions are intended only to explain the principles of the present application and should not be construed in any way as limiting the scope of protection of the present application. Based on the explanations herein, any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present application, as well as other specific implementations of the present application that can be conceived by those skilled in the art without inventive effort, shall be included within the scope of protection of the present application.

Claims

1. A method for removing invalid points in structured light three-dimensional measurement, characterized in that: The steps include: Obtain the wrapped phase, background intensity, and modulation intensity of the fringe pattern; Obtaining the discreteness between the sequence points of each fringe pattern coordinate and the ideal cosine curve according to the wrapping phase, background intensity and modulation intensity; Correcting point by point and establishing a new error energy function according to the discreteness; Invalid points are removed according to the new error energy function.

2. The method for removing invalid points in structured light 3D measurement according to claim 1, wherein: The step of obtaining the wrapping phase, background intensity, and modulation intensity of the fringe pattern specifically includes the following steps: Project a series of standard cosine fringe images onto the object to be measured; The height of the surface of the object being measured modulates the cosine fringes; The camera captures a sequence of fringe patterns in the order of projection, and expresses the collected deformation fringes as the following formula: Use the phase shift method to calculate the wrapped phase , background intensity and modulation intensity ; in, is any coordinate point in the kth fringe image The intensity value, , is the total phase shift, the wrapped phase Wrapped in the inverse tangent function with Discontinuous range Here, the wrapped phase is expanded by the phase unwrapping formula, and its mathematical formula is: in, Represents a fringe order diagram.

3. The method for removing invalid points in structured light 3D measurement according to claim 2, wherein: In the step of projecting a series of standard cosine fringe images onto the object to be measured, the standard cosine fringe images are acquired using a 4-step phase shift method.

4. The method for removing invalid points in structured light 3D measurement according to claim 3, wherein: In the step of obtaining the discreteness between the serial points of each stripe pattern coordinate and the ideal cosine curve according to the package phase, background intensity and intensity modulation, specifically: using weighted Gaussian Euclidean distance to obtain the discreteness between the serial points of each stripe pattern coordinate and the ideal cosine curve according to the package phase, background intensity and modulation intensity.

5. The method for removing invalid points in structured light 3D measurement according to claim 4, wherein: The weighted Gaussian Euclidean distance is used to obtain the discreteness between the sequence points of each fringe pattern coordinate and the ideal cosine curve according to the package phase, background intensity and modulation intensity, specifically comprising the following steps: The background intensity and the modulation intensity are used to normalize the pixel sequence points, and the relationship between the normalized sequence value and the ideal cosine curve in a single cycle is obtained; The Gaussian weighted Euclidean distance is used to quantify the discreteness between the sequence point and the ideal cosine curve as error energy, which is used to represent the cosine property of the same frequency pixel sequence point. The formula is as follows: in, is the error energy, are Gaussian weights, is the normalization factor, It is the deviation of the pixel from the ideal cosine curve.

6. The method for removing invalid points in structured light 3D measurement according to claim 5, wherein: In the step of correcting and establishing a new error energy function point by point according to the discreteness, specifically: correcting and establishing a new error energy function point by point using an improved Gaussian filtering method based on modulation intensity.

7. The method for removing invalid points in structured light 3D measurement according to claim 6, wherein: The new error energy function is as follows: in, is the new error energy, is a two-dimensional Gaussian window, Determines the weighting factor strength.

8. The method for removing invalid points in structured light 3D measurement according to claim 6, wherein: The step of removing invalid points according to the new error energy function specifically includes the following steps: Choose an appropriate error threshold ; By setting all The points are marked as 1 and the remaining points are marked as 0, thus obtaining a mask template , as follows: ； The invalid points in the background and shadow are removed by the mask template, where the pixels are in the mask template. The points with a value of 1 are considered valid points.

9. A system for removing invalid points in structured light 3D measurement, characterized in that: include: Fringe pattern acquisition unit: used to obtain the wrapped phase, background intensity and modulation intensity of the fringe pattern; A discreteness construction unit is used to obtain the discreteness between the sequence points of each fringe pattern coordinate and the ideal cosine curve according to the package phase, background intensity and modulation intensity; An error energy function construction unit is used to correct and establish a new error energy function point by point according to the discreteness; Invalid point removal unit: used for removing invalid points according to the new error energy function.

Images