Priori guidance restoration method for strong highlight area in fringe projection profilometry image
Through a priori guided repair method, the point cloud loss and morphological distortion caused by highlights in striped projection contour were solved, and the highlight area was realized Accurate repair improves measurement accuracy.
Patent Information
- Application Number
- CN202411274861.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-12
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-09-12
AI Technical Summary
In extreme optical environments, high reflectivity and external extreme light in the target surface during stripe projection contours lead to a large number of highlight components in the collected stripe images. The existing highlight removal methods cannot meet the precise repair requirements of stripe images, affecting the measurement accuracy.
By using a priori guided repair method, the phase modulation parameters of sinusoidal fringes are fitted line by line by identifying the highlight region, and the fringes information of the highlight region are repaired using segmented cubic Hermit interpolation and weighted nonlinear Levenberg-Marquardt fitting algorithm. , ensure the continuity and consistency of the repaired point cloud map.
The morphological distortion and point cloud cavity caused by highlights are effectively repaired, the measurement accuracy is improved, and the repaired point cloud map has no obvious faults in the visual and data structure.
Smart Images

Figure CN119067889A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the field of optical three-dimensional measurement, and in particular to the removal of highlights of projection images collected by fringe projection profilometry and the restoration of fringe information. Background Art
[0002] Highlight restoration technology is widely used in the field of visual inspection. It can accurately capture the tiny details and shapes of the object surface and provide important geometric features and color information support for the final inspection. In extreme optical environments, when using fringe projection profilometry to measure the target, the high reflectivity of the target surface and the external extreme lighting environment often cause a large number of highlight components to appear in the collected target fringe image, resulting in errors in fringe information, which in turn affects the measurement accuracy of the target. Therefore, highlight removal and fringe information restoration are crucial in the field of optical three-dimensional measurement. For highlight removal and fringe information restoration, the main difficulty lies in how to use the limited information of a single image to suppress highlights and restore fringe information. Traditional pure image processing-based methods are usually only applicable to images with weak highlight components, and the accuracy requirements of the color and texture information they restore are not high. They often only need to meet basic feature recognition requirements. However, the measurement results of the target in fringe projection profilometry are deeply dependent on the phase information of the fringe image. Therefore, the existing traditional highlight removal methods cannot meet the precise restoration requirements of fringe images and have great limitations. It is urgent to develop a set of accurate highlight restoration methods for fringe images. Summary of the invention
[0003] The present invention proposes a priori guided restoration method for strong highlights in fringe projection profilometry images, processes the restored multi-frame phase-shifted fringe images, and overcomes the problems of point cloud loss and morphological distortion caused by highlights.
[0004] In order to achieve the above object, the technical solution of the present invention is implemented as follows: a priori guided restoration method for strong highlights in a fringe projection profilometry image comprises the following steps: S1: Identify and locate the highlight area of the fringe projection profilometry image; S2: By analyzing the stripe information of the left and right adjacent areas of the highlight area, the background intensity and modulation amplitude of the sinusoidal stripes in the highlight area are determined row by row using the piecewise cubic Hermite interpolation polynomial algorithm; S3: By analyzing the fringe information of the upper and lower adjacent areas of the highlight area, the nonlinear Levenberg-Marquardt fitting algorithm is used to fit the initial values of the phase modulation parameters of the sinusoidal fringes row by row; S4: Based on the fringe information of the left and right adjacent areas, a weighted nonlinear Levenberg-Marquardt fitting algorithm is used to fit the phase modulation parameters of the highlight area column by column based on the initial value of the phase modulation parameter obtained in step S3; S5: Based on the parameters fitted in steps S2 and S4, the stripe information of the highlight area is calculated row by row.
[0005] The generally used fitting algorithm is the nonlinear Levenberg-Marquardt fitting algorithm, while the present invention adopts a weighted nonlinear Levenberg-Marquardt fitting algorithm. The weighted nonlinear Levenberg-Marquardt fitting algorithm can provide better fitting results and stronger robustness when processing data points with different qualities or different importance, especially in data containing highlight information.
[0006] The present invention analyzes the fringe pattern in the non-highlight area in the captured fringe projection image containing highlights, determines the constant and amplitude parameters of the projection fringe through a fitting algorithm, and utilizes the continuity of the mirror area and the fixed relative geometry between the projector and the object for the key initial phase parameters encoding shape information. This basis enables the iterative optimization strategy to gradually estimate the fringe intensity in the mirror area, which allows the restored fringe data to be seamlessly integrated with the original non-highlight information.
[0007] During the highlight area restoration process, the present invention accurately processes the stripe information and effectively connects it with the adjacent non-highlight area data. This processing method ensures the continuity and consistency of the global data, so that the restored point cloud image has no obvious faults in both vision and data structure.
[0008] The present invention compares the point cloud side profiles of the repaired point cloud image with those of the unprocessed point cloud image. The results show that the repaired point cloud image has obvious advantages in side profile analysis, which indicates that this method can effectively repair the morphological distortion and point cloud holes caused by highlights. BRIEF DESCRIPTION OF THE DRAWINGS
[0009] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0010] Figure 1 This is a flow chart of the prior-guided repair method for strong highlights in fringe projection profilometry images described in the present invention.
[0011] Figure 2 This is an expanded view of the roughly extracted highlight information described in the present invention.
[0012] Figure 3This is a calculation diagram of the initial value of the phase modulation parameter according to the smooth transition principle of the present invention.
[0013] Figure 4 The variable region algorithm is used for iterative calculation of the present invention. Direction Information diagram.
[0014] Figure 5 This is the final effect diagram after the processing described in the present invention.
[0015] Figure 6 This is a distribution diagram of pixel coordinate intensity of the highlight area before and after the processing described in the present invention.
[0016] Figure 7 The three-dimensional point cloud obtained from the original image and the three-dimensional point cloud obtained from the highlight repaired image described in the present invention are compared with the side contours of the point clouds. DETAILED DESCRIPTION
[0017] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0018] like Figure 1 As shown, a priori guided repair method for strong highlight areas in fringe projection profilometry images includes the following steps: S1: Identify and locate the highlight area of the fringe projection profilometry image. The specific method is as follows: S1.1 Diffuse reflection component in fringe image Calculating the threshold , whose size is m×n×3, then Flatten into a two-dimensional matrix Its size is Represents pixels in RGB channels, from The dark channel , as shown in formula (1): (1) in Represents the operation of finding the minimum value; S1.2 By calculation The mean and standard deviation of To determine the specular reflection and non-highlight components, as shown in formula (2): (2) S1.3 In the present invention The value is less than Values greater than are considered non-highlight components. The value of is considered as the highlight component, the highlight area is obtained, the pixels that meet this condition are collected, and it is defined as the geometric factor , as shown in formula (3): (3) S1.4 can Reshape The matrix is then normalized and binarized into a logical matrix to estimate the largest specular highlight area, thus identifying and extracting the largest connected highlight area As the dominant area T, in order to obtain its surrounding pixels ,right Perform expansion operation , as shown in formula (4): (4) S1.5 where is the result after the expansion operation, 𝑀𝑎𝑠𝑘 represents the convolution mask or kernel ( ), whose shape can be square, rectangle, straight line or circle. Perform convolution operation and scan For each pixel in Represents an empty set. At each pixel position, an “AND” operation is performed between the elements of its mask and the corresponding binary image pixels. If all “AND” operation results are 0, the target pixel value is set to 0, otherwise it is set to 1. and The surrounding pixels , as shown in formula (5): (5) S1.6 uses the least squares technique to determine the adjustment scale of each pixel, and the pixel information and The values representing the highlighted area and the surrounding area can be found in Using pixel coordinates and As an index, the information of the same pixel and The mirror information represents the highlighted area and its surrounding area, and the pixel coordinates can also be used and get , and then get the detected highlight area , as shown in formula (6): (6) Represents a reshaping operation. If we directly adjust the scale and Calculate and get the detected highlight pixel information The matrix cannot be directly converted into a matrix of the same dimension as the original image. After finding all the highlighted areas of the stripe image After that, they are converted and used as masks to prepare for the subsequent fringe repair in these areas. Considering that the vicinity of the highlight area is easily disturbed by "high-intensity" noise, these noises will interfere with the subsequent phase recognition and unwrapping algorithms, so the repair range is expanded and the extended algorithm is used from get .
[0019] S3: By analyzing the fringe information of the upper and lower adjacent areas of the highlight area, the nonlinear Levenberg-Marquardt fitting algorithm is used to fit the initial values of the phase modulation parameters of the sinusoidal fringes row by row. The specific method is as follows: S3.1 Since the pixel intensity in the fringe image is cosine distributed, the pixel information restoration in the highlight area must follow this distribution and be smoothly integrated with the information in the surrounding non-highlight area. Therefore, the present invention proposes a nonlinear Levenberg-Marquardt fitting algorithm based on the weight matrix to solve the restoration problem, as shown in formula (7): (7) in and represent the background intensity and modulation intensity respectively, T represents the edge period, and yes The coordinates of the pixels in ; The essence of the fitting algorithm proposed in S3.2 is to optimize the parameters of the model function to minimize the sum of squares of the residuals. Since the pixel information of the highlighted area is unreliable, the area Expand to the region As the basis of the fitting area, as shown in formula (8): (8) The expanded area is as follows Figure 2 As shown; S3.3 and is the coordinate of the pixel in the fitting area. Based on the reflection principle, the connected highlight area observed in the image means that the corresponding surface morphology of the object shows a smooth transition with the non-highlight area. Therefore, the present invention utilizes the image from the extended area. The phase information of Get the initial value of phase modulation parameters based on direction In order to describe the present invention in detail, the following direction Several rows of pixels above and below the region are used as references, and cosine function fitting is used to derive the initial values of the phase modulation parameters.
[0020] S2: By analyzing the stripe information of the left and right adjacent areas of the highlight area, the piecewise cubic Hermite interpolation polynomial algorithm is used to determine the background intensity and modulation amplitude of the sinusoidal stripes in the highlight area row by row and S4: Based on the stripe information of the left and right adjacent areas, a weighted nonlinear Levenberg-Marquardt fitting algorithm is used to fit the phase modulation parameters of the highlight area column by column based on the initial value of the phase modulation parameter obtained in step S3. The specific method is as follows: S2.4.1 In non-highlight areas, the changes in A and B are not significant, such as Figure 3 As shown, the present invention directly adopts the average value obtained by fitting from the reference area. It reflects the morphological information and uses interpolation to derive the initial value of the phase modulation parameter according to the principle of smooth transition. Initial value of internal phase modulation parameters Interpolation is performed in the region The fitting results are based on the piecewise cubic Hermite interpolation polynomial (PCHIP), as shown in formula (9): (9) Here ( ) yes The coordinates of . Coefficients in a piecewise cubic polynomial representing an interval , , and When determined, the polynomial satisfies the interpolation and shape-preserving criteria; S2.4.2 Therefore, the system of equations for each interval is as shown in formula (10): (10) and Yes and The key aspects of PCHIP are They are determined by the weighted harmonic mean of the slopes of adjacent intervals. If the data are monotonic, the interpolation factors will also be monotonic, so for each interval Get the difference and slope ; S2.4.3 Once the piecewise cubic polynomial is obtained , you can Incorporate the model to obtain the initial This method ensures that Information along The variability of direction also ensures In order to obtain reliable fitting results using the Levenberg-Marquardt algorithm, the present invention proposes a weight matrix representing the reliability of pixel information. , as shown in formula (11): (11) S2.4.4 From formula (11), the pixel coordinates are in the region When , the information of the pixel is considered unreliable, so its weight in the fitting process is assigned to 0. Then the elements of the Jacobian matrix are the partial derivatives of the residual with respect to the parameters, as shown in formula (12): (12) S2.4.5 The algorithm updates the parameter rules as shown in formula (13): (13) in is the damping factor that controls the step size. As the iteration proceeds, the parameter estimate will be updated, and the Jacobian matrix It will be recalculated at each iteration until the algorithm converges to the best fit parameters, and the process is repeated until convergence is achieved.
[0021] S5: Based on the parameters fitted in steps S2 and S4, the stripe information of the highlight area is calculated row by row. The specific method is as follows: By repeatedly calling the weighted Levenberg-Marquardt algorithm, we get ( ) For area , and then the pixel information corresponding to each row is derived according to the cosine function. However, along Direction Single The information may result in a planar point cloud, in which case the variable-region Levenberg-Marquardt algorithm is used to iteratively compute Direction Information such as Figure 4As shown, the calculated information is then substituted into the cosine function to obtain the corresponding pixel information. During the iteration process, the pattern of regional changes follows specific rules, as shown in formula (14): (14) in Represents the operation of the weighted Levenberg-Marquardt algorithm, which contains two key parameters and Used as the initial value of the phase modulation parameter, Indicates the changing area, that is, the range of the fitting area changes with the calculated The pixel position varies. Indicates the need Areas repaired in the direction The pixel index offset within .
[0022] The present invention selects a picture containing highlights as the experimental object, and uses the priori guidance of the strong highlights in the fringe projection profilometry image to finally obtain a highlight suppressed image. Figure 5 As shown, the final image is then analyzed and compared with the original image, as shown in Figure 6 As shown, its stripe information has been repaired. Figure 7 The above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A priori guided restoration method for strong highlight areas in fringe projection profilometry images, characterized in that: The following steps are involved: S1: Identify and locate the highlight area of the fringe projection profilometry image; S2: By analyzing the stripe information of the left and right adjacent areas of the highlight area, the background intensity and modulation amplitude of the sinusoidal stripes in the highlight area are determined row by row using the piecewise cubic Hermite interpolation polynomial algorithm; S3: By analyzing the fringe information of the upper and lower adjacent areas of the highlight area, the nonlinear Levenberg-Marquardt fitting algorithm is used to fit the phase modulation parameters of the sinusoidal fringes line by line. S4: Based on the fringe information of the left and right adjacent areas, a weighted nonlinear Levenberg-Marquardt fitting algorithm is used to fit the phase modulation parameters of the highlight area column by column based on the initial value of the phase modulation parameter obtained in step S3; S5: Based on the parameters fitted in steps S2 and S4, the stripe information of the highlight area is calculated row by row.
2. The method for a priori guided restoration of strong highlights in fringe projection profilometry images according to claim 1, characterized in that: The method for identifying and locating the highlight area of the fringe projection profilometry image described in step S1 is: (1) Through the diffuse reflection component in the fringe image Calculating the threshold , whose size is m×n×3, then Flatten into a 2D matrix Its size is Represents pixels in RGB channels, from The dark channel , as shown in formula (1): (1) in Represents the operation of finding the minimum value; (2) By calculation The mean and standard deviation of To determine the specular reflection and non-highlight components, as shown in formula (2): (2) Represents an operation to calculate the average value represents the operation of calculating the standard deviation, 𝜂 is the mirror degree of the fringe image It determines the intensity and area of the specular highlight, and 𝜂=0.4 is applicable to the FPP system proposed in the present invention; (3) In the present invention The value is less than Values greater than are considered non-highlight components. The value of is considered as the highlight component, and the pixels that meet this condition are collected and defined as the geometric factor , as shown in formula (3): (3) (4) Among them for The result after expansion operation is obtained by operating difference and The surrounding pixels get ; (5) Use the least squares technique to determine the adjustment scale of each pixel, the pixel information and The values representing the highlighted area and the surrounding area can be found in Using pixel coordinates and As an index, the information of the same pixel and The mirror information represents the highlighted area and its surrounding area, and the pixel coordinates can also be used and get , and then get the detected highlight area , as shown in formula (4): (4) It means reshaping exercise.
3. The method for a priori guided restoration of strong highlights in fringe projection profilometry images according to claim 1, characterized in that: The method of step S3 is: (1) Since the pixel intensity in the fringe image is cosine distributed, the pixel information restoration in the highlight area must follow this distribution and be smoothly integrated with the information in the surrounding non-highlight area. Therefore, the present invention proposes a nonlinear Levenberg-Marquardt fitting algorithm to solve the restoration problem, as shown in formula (5): (5) in and represent the background intensity and modulation intensity respectively, T represents the edge period, and yes The coordinates of the pixels in ; (2) The essence of the proposed fitting algorithm is to optimize the parameters of the model function to minimize the sum of squares of the residuals. Since the pixel information of the highlighted area is unreliable, the area Expand to the region As the basis of the fitting area, as shown in formula (6): (6) (3) and is the coordinate of the pixel in the fitting area. Based on the reflection principle, the connected highlight area observed in the image means that the corresponding surface morphology of the object shows a smooth transition with the non-highlight area. Therefore, the present invention utilizes the image from the extended area. The phase information of Get the initial value of phase modulation parameters based on direction In order to describe the present invention in detail, the following direction Several rows of pixels above and below the region are used as references, and cosine function fitting is used to derive the initial values of the phase modulation parameters.
4. The method for a priori guided restoration of strong highlights in fringe projection profilometry images according to claim 1, characterized in that: The method of step S2 and step S4 is: (1) In the non-highlighted area, the changes in A and B are not significant, so the present invention directly uses the average value obtained by fitting from the reference area. It reflects the morphological information and uses interpolation to derive the initial value of the phase modulation parameter according to the principle of smooth transition. Initial value of internal phase modulation parameters Interpolation is performed in the region The fitting results are based on piecewise cubic Hermite interpolation polynomials (PCHIP); (2) Once the piecewise cubic polynomial is obtained , you can Incorporate the model to obtain the initial This method ensures that Information along The variability of direction also ensures In order to obtain reliable fitting results using the Levenberg-Marquardt algorithm, the present invention proposes a weight matrix representing the reliability of pixel information. , as shown in formula (7): (7) (3) According to formula (11), the pixel coordinates are located in the region When , the information of the pixel is considered unreliable, so its weight in the fitting process is assigned to 0. Then the elements of the Jacobian matrix are the partial derivatives of the residual with respect to the parameters, as shown in formula (8): (8) (4) The algorithm updates the parameter rules as shown in formula (9): (9) in is the damping factor that controls the step size. As the iteration proceeds, the parameter estimate will be updated, and the Jacobian matrix It will be recalculated at each iteration until the algorithm converges to the best fit parameters, and the process is repeated until convergence is achieved.
5. The method for a priori guided restoration of strong highlights in fringe projection profilometry images according to claim 1, characterized in that: The method of step S5 is: By repeatedly calling the weighted Levenberg-Marquardt algorithm, we get ( ) For area , and then the pixel information corresponding to each row is derived according to the cosine function. However, along Direction Single The information may result in a planar point cloud, in which case the variable-region Levenberg-Marquardt algorithm is used to iteratively compute Direction Then, the calculated information is substituted into the cosine function to obtain the corresponding pixel information. During the iteration process, the pattern of regional changes follows specific rules, as shown in formula (10): (10) in Represents the operation of the weighted Levenberg-Marquardt algorithm, which contains two key parameters and Used as the initial value of the phase modulation parameter, Indicates the changing area, that is, the range of the fitting area changes with the calculated The pixel position varies. Indicates the need Areas repaired in the direction Pixel index offset within .
Citation Information
Patent Citations
Area projection optical 3D profile measuring method for object with high dynamic range
CN106091981A
3D microscopic measurement method based on tilt-shift projector and double telecentric camera
CN115824090A
Phase mapping high-precision projector calibration method based on grating fringe projection
CN116912334A
Motion error compensation method for fringe projection phase shift profilometry
CN118172255A
Hybrid light measurement method for measuring three-dimensional profile
US20200166333A1
