Phase invalid point removal method based on multiple features
Through the multi-feature combination with the K-means clustering algorithm, the problem of phase invalid point removal in the existing technology requires resetting the threshold, and efficient removal of complex modulation objects is achieved, and the adaptive ability of the method is improved. Accuracy.
Patent Information
- Application Number
- CN202411121322.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-15
- Publication Date
- 2024-11-26
AI Technical Summary
The existing phase invalid point removal method requires resetting the threshold according to the new system or the object to be measured, resulting in limited application range and affecting efficiency.
A multi-feature-based method is adopted, including data modulation, error energy and continuity characteristics of phase maps, and the segmentation threshold is automatically set in combination with the K-means clustering algorithm to achieve the removal of adaptive invalid points for different systems and objects to be measured.
Effectively removing phase invalid points of complex modulation objects ensures the scope of application and accuracy of the method, avoids additional work in threshold setting, and improves the system's adaptability.
Smart Images

Figure CN119027349A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing, and in particular to a method for removing phase invalid points based on multiple features. Background Art
[0002] In recent years, fast, high-precision, non-contact three-dimensional (3D) reconstruction technology has been increasingly used in the fields of 3D measurement, reverse engineering, biometrics, and fruit picking in industry, medicine, and agriculture to reconstruct various types of objects. Phase profilometry (PSP) has become one of the most widely used methods due to its high accuracy.
[0003] In practical applications, phase profile measurement is usually used in combination with point cloud technology. Since there are various noises, backgrounds, shadows and other parts in the camera field of view that do not contain effective object surface information, these will result in a large number of phase invalid points when calculating the phase, which will cause the generated point cloud to contain a large number of corresponding invalid points. These invalid points not only interfere with the measurement, but also make it difficult to observe the object, and will cause errors in the post-processing of some point clouds, such as registration. Therefore, it is very necessary to remove these invalid points when generating point clouds.
[0004] At present, the methods for removing invalid points are mainly divided into two categories. One is to use modulation to separate the background and shadow, and the other is to remove or correct the abnormal phase according to the characteristics of the phase. In the study of modulation, Zhang et al. improved the OTSU method by adding a weight to obtain a better threshold to distinguish objects from background shadows, but this method is only applicable to high-modulation objects. For low-modulation objects, it will cause excessive removal, resulting in incomplete point clouds. In the study of phase correction, Zhuang et al. fit the plane of the unfolded phase and judge whether it is an invalid point according to whether the error of the fitting plane at different points is within the standard deviation threshold. However, this method can only remove some invalid points. The existing methods can achieve the effect of removing invalid phase points to a certain extent, but their effects are closely related to the thresholds set by themselves. For each new system or object to be measured, most of them need to reset the threshold to achieve good results. This leads to the need for technicians to invest additional work in the actual application process, which affects the scope of application of the method. Summary of the invention
[0005] In order to solve the problem that most of the existing invalid point removal methods need to reset the threshold when applied to new systems or objects under test, the present invention provides a phase invalid point removal method based on multiple features, which can automatically set the data modulation related thresholds and has the ability to adapt to different systems and objects under test.
[0006] The technical solution of the present invention is as follows: a method for removing phase invalid points based on multiple features, characterized in that it comprises the following steps:
[0007] S1: Collect fringe images of the object to be reconstructed at the specified frequency f, collect N images for each frequency f, and calculate the absolute phase;
[0008] S2: Calculate the data modulation map, determine the appropriate segmentation threshold based on the K-means clustering method, and segment the background and shadow;
[0009] The specific steps include:
[0010] S21: Divide the modulation by the average intensity and use the data modulation As the segmentation standard, the data modulation diagram corresponding to the frequency is obtained based on the data modulation formula;
[0011]
[0012] Where a(x,y) is the modulation degree and b(x,y) is the average intensity;
[0013] S22: Setting segmentation threshold To separate the background and shadow points;
[0014] S23: Automatically set segmentation threshold using K-means clustering method The corresponding value of
[0015] S24: Set the segmentation threshold with a specific value Act on the data modulation diagram to obtain the data modulation mask;
[0016] S3: Based on the error energy map corresponding to all frequency fringe images, a fixed invalid point screening threshold T is set Error , get the error energy mask;
[0017] S4: Obtain the phase from the phase credibility standard map of the absolute phase, obtain the common mask from all the masks, and remove invalid points in the absolute phase.
[0018] It is further characterized by:
[0019] In step S1, three-dimensional reconstruction is performed based on grating projection profilometry, and phase is calculated based on phase shift method, which specifically includes the following steps:
[0020] S11: The projector projects the pre-designed horizontal and vertical sinusoidal fringe images onto the object to be processed, and the camera collects the deformed fringe images; its intensity I n (x,y) distribution calculation formula I n (x,y) is:
[0021]
[0022] In the formula, a(x,y) is the background light intensity, b(x,y) is the modulation intensity, is the phase of the fringe, N is the number of phase shift steps; n represents the nth image of the fringe corresponding to frequency f, and x and y are the pixel coordinates on the image;
[0023] S12: The phase fringes projected onto the object to be processed include: phase fringes of 0, π / 2, π, and 3π / 2; the phase of the fringe image is calculated based on the following formula to obtain a wrapped phase map;
[0024]
[0025] in, For the wrapping phase;
[0026] S13: Use the multi-frequency heterodyne method to perform phase unwrapping to obtain a continuously distributed absolute phase. The unwrapping process is:
[0027]
[0028] Among them, Φ(x,y) is the absolute phase after unwrapping, and k(x,y) is the phase order corresponding to the current pixel;
[0029] S14: completing three-dimensional reconstruction based on the intrinsic and extrinsic parameters and absolute phase map obtained by system calibration;
[0030] Step S3 specifically includes the following contents:
[0031] S31: Construct an error function Error f , get the error energy diagram corresponding to each frequency;
[0032] Error function f as follows:
[0033]
[0034] Error f is the error energy of the fringe of frequency f, is the wrapped phase of the fringe with frequency f, N is the total number of phase shift steps, I n (x,y) is the gray value of the nth image of the frequency f stripe, a f (x,y) is the average intensity of the fringe with frequency t, b f (x,y) is the modulation intensity of the frequency f fringe;
[0035] S32: Setting invalid point screening threshold T Error , based on TError Screen out invalid points on the error energy graph;
[0036] S33: Calculate a mask for each frequency, combine the masks of all frequencies to obtain a total error energy mask;
[0037] Step S4 includes the following contents:
[0038] S41: define the absolute maximum value of the difference between the four areas of the pixel and the pixel itself as the credibility standard W, and obtain a phase credibility standard map;
[0039] W(i,j)=max[△ x (i,j),△ y (i,j)];
[0040] Where i and j are the row and sequence numbers of the absolute phase diagram, △ x (i,j) and ∆ y (i, j) are the absolute values of the maximum partial derivatives of the absolute phase Φ(i, j) or the depth Z(i, j) in the x and y directions, respectively;
[0041] △ x (i,j) and ∆ y The expression of (i,j) is:
[0042]
[0043] S42: Set the effective point threshold Limit the range of phase difference;
[0044] In absolute phase, the threshold Points below the threshold are considered valid points, while those above the threshold The point is judged as an invalid point, and the phase credibility standard map mask corresponding to the absolute phase credibility is obtained;
[0045] S43: Obtain a common mask from the total error energy mask, the phase credibility standard map mask and the data modulation mask, and remove invalid points in the absolute phase.
[0046] The present application provides a multi-feature based phase invalid point removal method, which uses multiple features such as data modulation, error energy, and continuity of phase diagram, introduces the K-means clustering algorithm, completes the automatic setting of relevant thresholds, and realizes the invalid point removal of complex modulation objects, while ensuring that the method has the ability to adapt to different systems and objects under test. Experimental results show that the proposed method solves the problem of phase invalid point removal in the reconstruction of objects with complex modulation. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] Figure 1 It is a diagram of the structured light system configuration;
[0048] Figure 2 It is an example of a stripe pattern corresponding to a plaster sculpture;
[0049] Figure 3 for Figure 2 Data modulation results corresponding to the fringe diagram;
[0050] Figure 4 It is a real picture of plaster sculpture;
[0051] Figure 5 This is a comparison of the plaster sculpture phase invalid point before and after correction;
[0052] Figure 6 This is a comparison of the invalid points in the plaster sculpture point cloud before and after correction. DETAILED DESCRIPTION
[0053] like Figure 1 to Figure 6 As shown, the present application includes a multi-feature based phase invalid point removal method, which includes the following steps.
[0054] S1: According to the specified frequency f, collect the fringe image of the object to be reconstructed, collect N fringe images for each frequency f, and calculate an absolute phase based on all fringe images. In this embodiment, 3 frequency fringes and 4 phases are used, namely: frequencies f1, f2 and f3, N=4.
[0055] In this method, 3D reconstruction is performed based on Fringe Projection Profilometry (FPP), and its structure is as follows: Figure 1 As shown, the phase is calculated based on the phase shift method, which specifically includes the following steps.
[0056] S11: Figure 1 As shown in the figure, the object Object is placed on the reference plane Reference plane, and the projector Projector projects the pre-designed horizontal and vertical sinusoidal fringe images onto the object to be reconstructed Object, and the camera Camera collects the deformed fringe images; the specific projector height d and the distance l between the projector and the camera are adaptively adjusted according to the height h of the object.
[0057] In the four-step phase shift method, the projector projects pre-designed horizontal and vertical sinusoidal fringe images onto the object, and the camera collects the deformed fringe images; the intensity distribution of each fringe image is calculated by formula I n (x,y) is:
[0058]
[0059] In the formula, a(x,y) is the background light intensity, b(x,y) is the modulation intensity, is the phase of the stripes, N is the number of phase shift steps; n represents the nth image of the stripes corresponding to the frequency f, and x and y are the pixel coordinates on the image; in this embodiment, the four-step phase shift method used needs to project four phase stripes with phases of 0, π / 2, π, and 3π / 2 onto the object.
[0060] S12: Calculate the phase of the fringe image according to the following formula based on the fringe image to obtain a wrapped phase map;
[0061]
[0062] Since the phase range of the inverse tangent function is (-π,π], the phase It is called the wrapping phase.
[0063] S13: In order to calculate the continuous absolute phase of the whole field, it is necessary to use the multi-frequency heterodyne method to perform phase unwrapping to obtain the continuously distributed absolute phase. The unwrapping process can be described as:
[0064]
[0065] Among them, Φ(x,y) is the absolute phase after unwrapping, and k(x,y) is the phase order corresponding to the current pixel;
[0066] S14: The phase order is calculated by assisting the multi-frequency heterodyne method. This method uses a three-frequency heterodyne method for unwrapping calculation. Then, the three-dimensional reconstruction is completed based on the intrinsic and extrinsic parameters and the absolute phase map obtained by system calibration.
[0067] For objects with low reflectivity, such as black objects, their calculated modulation is larger than that of background points and shadow points, but very close, and also quite different from the modulation of some surface points of objects with high reflectivity. Therefore, using only modulation, it is impossible to find a suitable threshold to separate the effective surface points with low reflectivity from the background points and shadow points.
[0068] S2: When calculating the data modulation map, an appropriate segmentation threshold is determined based on the K-means clustering method to segment the background and shadow. Specifically, the following steps are included.
[0069] S21: In order to have a segmentation threshold that can distinguish both high and low reflectivity points from background points and shadow points, this method divides the modulation by the average intensity and uses the data modulation. As the segmentation standard, the data modulation diagram corresponding to the frequency is obtained based on the data modulation formula;
[0070]
[0071] Where a(x,y) is the modulation degree and b(x,y) is the average intensity;
[0072] For points with higher reflectivity, the average intensity is larger, and for points with lower reflectivity, the average intensity is smaller, so the data modulation of the two will be similar. The data modulation of the background and shadow points is still smaller, and relative to the valid points, a threshold can be set at this time. Based on the fringe pattern, the data modulation of each pixel is The data modulation diagram corresponding to each fringe diagram was obtained through calculation.
[0073] S22: Setting segmentation threshold To separate the background and shadow points. Figure 2 and Figure 3 As shown, the fringe map and data modulation map of the sculpture are displayed. It can be clearly seen that the shadow part of the data modulation map is darker than other parts.
[0074] In practical applications, when the system changes, this segmentation threshold Need to be reset, so this method uses K-means clustering method to automatically set the threshold Ensure that this method has the ability to adapt to different systems and objects. After that, you can get the data modulation mask
[0075] S23: Automatically set segmentation threshold using K-means clustering method The corresponding value of , the specific clustering model is implemented based on the existing technology;
[0076] S24: Set the segmentation threshold with a specific value Act on the data modulation diagram to obtain the data modulation mask
[0077] S3: Based on the error energy map corresponding to all frequency fringe images, a fixed invalid point screening threshold T is set Error , the error energy mask is obtained, which specifically includes the following contents.
[0078] The gray value of a valid point that is correctly encoded should conform to the cosine form of formula (1). By changing the form of formula (1), the following formula can be obtained:
[0079]
[0080] S31: When the surface of the object is correctly encoded by the structured light, the wrapped phase calculated from formula (2) is brought into formula (5), and formula (5) should be valid. However, for the surface points of the object with strong noise and the background points and shadow points that are not encoded by the structured light, formula (5) is not valid, so an error function can be constructed as follows:
[0081]
[0082] Error f is the error energy of the fringe of frequency f, is the wrapping phase of the frequency f fringe, N is the total number of phase shift steps, in this embodiment, three frequencies and four phases, that is, N = 4, I n (x,y) is the gray value of the nth image of the frequency f stripe, a f (x,y) is the average intensity of the fringe with frequency t, b f (x, y) is the modulation intensity of the frequency f fringe. Based on the error function, the error energy of each pixel can be obtained, and then the error energy map corresponding to the fringe map of each frequency can be obtained. This application includes 3 frequencies, so there are 3 error energy maps.
[0083] In an ideal noise-free situation, for valid points correctly encoded by structured light, Error f The value should be 0, but in actual acquisition, it is impossible to have no noise, so the effective point Error f The value is not 0, but close to 0. For invalid points such as strong noise, shadow points and background points, their Error f It is random, but most of them are more effective than Error f It is usually necessary to filter out invalid points.
[0084] S32: Setting invalid point screening threshold T Error , based on T Error Screen out invalid points;
[0085] In specific applications, the invalid point screening threshold T Error is set based on historical data; in this embodiment, T Error A fixed threshold of 0.5 was used.
[0086] On the error energy graph, the value greater than the invalid point screening threshold T Error If the error is too large, it will be identified as an invalid point and all invalid points will be screened out.
[0087] S33: Calculate a mask for each frequency, combine the masks of all frequencies, and obtain a total error energy mask.
[0088] Because three frequency stripes are used in this embodiment, a mask can be obtained for each frequency, and finally a total error energy mask is combined. The formula is as follows
[0089]
[0090] Mask Error is the total error energy mask, and Then are the error energy masks for frequencies f1, f2 and f3 respectively.
[0091] S4: Obtain the phase from the phase credibility standard map of the absolute phase, obtain the common mask from all the masks, and remove invalid points, which specifically includes the following contents.
[0092] In order to effectively remove invalid points, this method uses the absolute phase image Φ continuity for processing.
[0093] S41: define the absolute maximum value of the difference between the four areas of the pixel and the pixel itself as the credibility standard W;
[0094] W(i,j)=max[△ x (i,j),△ y (i,j)] (8);
[0095] Where i and j are the row and sequence numbers of the absolute phase diagram, △ x (i,j) and ∆ y (i, j) are the absolute values of the maximum partial derivatives of the absolute phase Φ(i, j) or the depth Z(i, j) in the x and y directions, respectively;
[0096] The credibility criterion for the absolute phase image is W Φ ,△ x (i,j) and ∆ y The expression of (i,j) is:
[0097]
[0098] Then the phase credibility standard map is obtained through formulas (8) and (9).
[0099] S42: Set the effective point threshold Limit the range of phase difference.
[0100] The larger the value of the credibility standard W, the lower the credibility of the pixel. This method sets a valid point threshold Threshold on absolute phase Points below the threshold are considered valid points, while those above the threshold The point is judged as an invalid point, and the phase credibility standard map mask corresponding to the absolute phase credibility can be obtained.
[0101] In practical applications, the effective point threshold Based on historical data, in this embodiment, this threshold It will be set to 3pi / 4 to limit the range of phase difference and ensure the effectiveness and reliability of the obtained absolute phase map in point cloud reconstruction.
[0102] S43: Obtain a common mask from the total error energy mask, the phase credibility standard map mask and the data modulation mask, and remove invalid points in the absolute phase. Therefore, in this method, based on the three features of error energy, phase credibility and data modulation, the phase invalid point removal is realized, and the invalid point removal is performed from multiple features, ensuring that this method is particularly suitable for objects with complex modulation.
[0103] In order to evaluate the performance of the compensation algorithm proposed in this application, a monocular 3D scanner was used to measure the plaster sculpture to remove invalid points. Figure 3 As shown. The monocular 3D scanner consists of a MER-1070-14U3M with a resolution of 3840×2748, a computer lens with a focal length of 35mm, and a digital light processing (DLP) projector with a resolution of 1280×720. There is a baseline distance of 73mm between the center of the camera and the projector lens. The scanner has a working distance of 450mm and a field of view of 190mm×135mm. It is installed in a dark room to eliminate the influence of ambient light. The computer configuration is Intel(R) Core(TM) i7-13700, 2.10GHz, and 32GB memory. This paper uses Matlab as the programming language of the algorithm, and the compilation environment is MATLAB R2023a.
[0104] Figure 4 The corresponding fringe pattern for a certain frequency is as follows Figure 2 As shown, Figure 2 The data modulation results corresponding to the fringe diagram are as follows Figure 3 As shown. First, the fringe pattern is read, and then the absolute phase map, data modulation map, three frequency error energy maps and the phase credibility standard map of the absolute phase are calculated. Subsequently, according to the data modulation map, the K-means clustering method is used to determine the appropriate threshold value to segment the background and shadow. At the same time, based on the three error energy maps and the phase credibility standard map of the absolute phase, the corresponding fixed threshold values are set. In this way, the phase effective area composed of the data modulation, error energy and phase credibility standard can be obtained, and the point cloud data is calculated based on the obtained effective area. The phase before and after correction is shown in Figure 5As shown in Figure 1, (a) is the phase diagram before correction, and (b) is the phase diagram after correction. Figure 6 As shown, (a) is the point cloud before correction, and (b) is the point cloud after correction. It can be clearly seen that after correction, the invalid phase points of the jump are removed and there are no noises in the point cloud.
[0105] After using the technical solution of the present invention, based on the phase invalid point identification method in the fringe projection profilometry (FPP) technology, the K-means clustering algorithm is introduced on the basis of utilizing multiple features such as data modulation, error energy, and continuity of the phase diagram, and the automatic setting of the relevant threshold is completed to achieve the invalid point removal of complex modulation objects. Finally, the continuity feature is used to repair the misjudged valid points, thereby avoiding the erroneous removal of the valid point cloud. The experimental results show that this method has better solved the problem of phase invalid point removal in the reconstruction of objects with complex modulation.
Claims
1. A multi-feature based phase invalid point removal method, characterized in that: It includes the following steps: S1: Collect fringe images of the object to be reconstructed at the specified frequency f, collect N images for each frequency f, and calculate the absolute phase; S2: Calculate the data modulation map, determine the appropriate segmentation threshold based on the K-means clustering method, and segment the background and shadow; The specific steps include: S21: Divide the modulation by the average intensity and use the data modulation As the segmentation standard, the data modulation diagram corresponding to the frequency is obtained based on the data modulation formula; Where a(x,y) is the modulation degree and b(x,y) is the average intensity; S22: Setting segmentation threshold To separate the background and shadow points; S23: Automatically set segmentation threshold using K-means clustering method The corresponding value of S24: Set the segmentation threshold with a specific value Act on the data modulation diagram to obtain the data modulation mask; S3: Based on the error energy map corresponding to all frequency fringe images, a fixed invalid point screening threshold T is set Error , get the error energy mask; S4: Obtain the phase from the phase credibility standard map of the absolute phase, obtain the common mask from all the masks, and remove invalid points in the absolute phase.
2. The method for removing phase invalid points based on multiple features according to claim 1, characterized in that: In step S1, three-dimensional reconstruction is performed based on grating projection profilometry, and phase is calculated based on phase shift method, which specifically includes the following steps: S11: The projector projects the pre-designed horizontal and vertical sinusoidal fringe images onto the object to be processed, and the camera collects the deformed fringe images; its intensity I n (x,y) distribution calculation formula I n (x,y) is: In the formula, a(x,y) is the background light intensity, b(x,y) is the modulation intensity, is the phase of the fringe, N is the number of phase shift steps; n represents the nth image of the fringe corresponding to frequency f, and x and y are the pixel coordinates on the image; S12: The phase fringes projected onto the object to be processed include: phase fringes of 0, π / 2, π, and 3π / 2; the phase of the fringe image is calculated based on the following formula to obtain a wrapped phase map; in, For the wrapping phase; S13: Use the multi-frequency heterodyne method to perform phase unwrapping to obtain a continuously distributed absolute phase. The unwrapping process is: Among them, Φ(x,y) is the absolute phase after unwrapping, and k(x,y) is the phase order corresponding to the current pixel; S14: Complete three-dimensional reconstruction based on the intrinsic and extrinsic parameters and absolute phase map obtained by system calibration.
3. The method for removing phase invalid points based on multiple features according to claim 1, characterized in that: Step S3 specifically includes the following contents: S31: Construct an error function Error f , get the error energy diagram corresponding to each frequency; Error function f as follows: Error f is the error energy of the fringe of frequency f, is the wrapped phase of the fringe with frequency f, N is the total number of phase shift steps, I n (x,y) is the gray value of the nth image of the frequency f stripe, a f (x,y) is the average intensity of the fringe with frequency t, b f (x,y) is the modulation intensity of the frequency f fringe; S32: Setting invalid point screening threshold T Error , based on T Error Screen out invalid points on the error energy graph; S33: Calculate a mask for each frequency, combine the masks of all frequencies, and obtain a total error energy mask.
4. The method for removing phase invalid points based on multiple features according to claim 3, characterized in that: Step S4 includes the following contents: S41: define the absolute maximum value of the difference between the four areas of the pixel and the pixel itself as the credibility standard W, and obtain a phase credibility standard map; W(i,j)=max[△ x (i,j),△ y (i,j)]; Where i and j are the row and sequence numbers of the absolute phase diagram, △ x (i,j) and ∆ y (i, j) are the absolute values of the maximum partial derivatives of the absolute phase Φ(i, j) or the depth Z(i, j) in the x and y directions, respectively; △ x (i,j) and ∆ y The expression of (i,j) is: S42: Set the effective point threshold Limit the range of phase difference; In absolute phase, the threshold Points below the threshold are considered valid points, while those above the threshold The point is judged as an invalid point, and the phase credibility standard map mask corresponding to the absolute phase credibility is obtained; S43: Obtain a common mask from the total error energy mask, the phase credibility standard map mask and the data modulation mask, and remove invalid points in the absolute phase.
