A priori-guided inpainting method for strong highlight areas in fringe projection profilometry images

Abstract

The present invention proposes a priori guided repair method for strong highlight areas in fringe projection profilometry images, aiming to remove and repair highlights in fringe projection measurement images of specular reflective objects. The fringe patterns in the non-highlight areas are analyzed and the constant and amplitude parameters of the projected fringes are determined using a fitting algorithm. For the initial values ​​of the key phase modulation parameters that encode shape information, the continuity of the specular reflection area and the fixed relative geometry between the projector and the object are used, followed by iterative optimization, so that the fringe intensity in the highlight area can be gradually calculated. The results show that the repaired fringe information is well connected with the adjacent non-highlight information, ensuring global consistency and continuity. Three-dimensional measurement experiments show that this method can effectively repair morphological distortion and point cloud holes caused by highlights.

Description

Technical Field

[0001] The present invention relates to the field of optical three-dimensional measurement, and in particular to removing highlights from projection images collected by fringe projection profilometry and repairing fringe information. Background Art

[0002] Highlight restoration technology is widely used in visual inspection, accurately capturing minute surface details and shapes, providing crucial geometric features and color information for final detection. When measuring targets using fringe projection profilometry in extreme optical environments, the high reflectivity of the target surface and the extreme external lighting conditions often result in a significant amount of highlight components in the captured fringe image, leading to errors in fringe information and, in turn, compromising target measurement accuracy. Therefore, highlight removal and fringe restoration are crucial in the field of optical 3D measurement. The primary challenge in both highlight removal and fringe restoration lies in utilizing the limited information available in a single image to suppress highlights and restore fringe information. Traditional pure image processing-based methods are typically only suitable for images with weak highlight components, and the accuracy requirements for restored color and texture information are relatively low, often sufficing only to meet basic feature recognition requirements. However, the target measurement results in fringe projection profilometry are deeply dependent on the phase information of the fringe image. Therefore, existing highlight removal methods cannot meet the requirements for accurate fringe image restoration and exhibit significant limitations. Consequently, the development of an accurate highlight restoration method for fringe images is urgently needed. Summary of the Invention

[0003] The present invention proposes a priori guided restoration method for strong highlights in fringe projection profilometry images, processes the restored multi-frame phase-shifted fringe images, and overcomes the problems of point cloud loss and morphological distortion caused by highlights.

[0004] To achieve the above object, the technical solution of the present invention is implemented as follows: a priori guided restoration method for strong highlights in fringe projection profilometry images comprises the following steps:

[0005] S1: Identify and locate the highlight area of ​​the fringe projection profilometry image;

[0006] S2: By analyzing the fringe information of the left and right adjacent areas of the highlight area, the background intensity and modulation amplitude of the sinusoidal fringe in the highlight area are determined row by row using the piecewise cubic Hermite interpolation polynomial algorithm;

[0007] S3: By analyzing the fringe information of the upper and lower adjacent areas of the highlight area, the nonlinear Levenberg-Marquardt fitting algorithm is used to fit the initial values ​​of the phase modulation parameters of the sinusoidal fringes row by row;

[0008] S4: Based on the fringe information of the left and right adjacent areas, a weighted nonlinear Levenberg-Marquardt fitting algorithm is used to fit the phase modulation parameters of the highlight area column by column based on the initial phase modulation parameter values ​​obtained in step S3;

[0009] S5: Based on the parameters fitted in steps S2 and S4, the stripe information of the highlight area is calculated row by row.

[0010] The commonly used fitting algorithm is the nonlinear Levenberg-Marquardt fitting algorithm, while the present invention adopts a weighted nonlinear Levenberg-Marquardt fitting algorithm. The weighted nonlinear Levenberg-Marquardt fitting algorithm can provide better fitting results and stronger robustness when processing data points with different qualities or different importance, especially in data containing highlight information.

[0011] The present invention analyzes the fringe patterns in the non-highlight area of ​​a captured fringe projection image containing highlights, and determines the constants and amplitude parameters of the projected fringe through a fitting algorithm. For the key initial phase parameters encoding shape information, the continuity of the mirror area and the fixed relative geometry between the projector and the object are utilized. This basis enables an iterative optimization strategy to gradually estimate the fringe intensity in the mirror area, which allows the restored fringe data to be seamlessly integrated with the original non-highlight information.

[0012] During the highlight area restoration process, the present invention accurately processes the stripe information and effectively connects it with the adjacent non-highlight area data. This processing method ensures the continuity and consistency of the global data, so that the restored point cloud image has no obvious faults in both visual and data structure.

[0013] The present invention compares the point cloud side profiles of the repaired point cloud image with the unprocessed point cloud image. The results show that the repaired point cloud image has obvious advantages in side profile analysis, which indicates that this method can effectively repair the morphological distortion and point cloud holes caused by highlights. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0015] Figure 1 This is a flow chart of the prior-guided restoration method for strong highlights in fringe projection profilometry images described in the present invention.

[0016] Figure 2 This is an expanded view of the roughly extracted highlight information described in the present invention.

[0017] Figure 3 This is a calculation diagram of the smooth transition principle and the initial value of the phase modulation parameter according to the present invention.

[0018] Figure 4 The variable region algorithm is used for iterative calculation in the present invention. Directional Information diagram.

[0019] Figure 5 This is the final effect diagram after the processing described in the present invention.

[0020] Figure 6 This is a distribution diagram of pixel coordinate intensity in the highlight area before and after the processing described in the present invention.

[0021] Figure 7 The three-dimensional point cloud obtained from the original image and the three-dimensional point cloud obtained from the highlight repaired image described in the present invention are compared, and the side profiles of the point clouds are compared. DETAILED DESCRIPTION

[0022] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without creative work are within the scope of protection of the present invention.

[0023] like Figure 1 As shown, a priori guided restoration method for strong highlight areas in fringe projection profilometry images includes the following steps:

[0024] S1: Identify and locate the highlight area of ​​the fringe projection profilometry image. The specific method is as follows:

[0025] S1.1 Diffuse reflection component in fringe image Calculating the threshold , whose size is m×n×3, then Flattened into a two-dimensional matrix Its size is Represents pixels in RGB channels, from Obtained from the dark channel , as shown in formula (1):

[0026] (1)

[0027] in Represents the operation of finding the minimum value;

[0028] S1.2 By calculation The mean and standard deviation of To determine the specular and non-highlight components, as shown in formula (2):

[0029] (2)

[0030] S1.3 In the present invention The value is less than The value of is considered as non-highlight component, and greater than The value of is considered as the highlight component, the highlight area is obtained, the pixels that meet this condition are collected, and it is defined as the geometric factor , as shown in formula (3):

[0031] (3)

[0032] S1.4 can Reshape into The matrix is ​​then normalized and binarized into a logical matrix to estimate the largest specular highlight area, thus identifying and extracting the largest connected highlight area As the dominant area T, in order to get its surrounding pixels ,right Perform expansion operations , as shown in formula (4):

[0033] (4)

[0034] S1.5 where is the result after the expansion operation, 𝑀𝑎𝑠𝑘 represents the convolution mask or kernel ( ), which can be a square, rectangle, line or circle. Perform convolution operation and scan For each pixel in Represents an empty set, and performs an AND operation between the elements of its mask and the corresponding binary image pixels at each pixel position. If all the AND operations are 0, the target pixel value is set to 0, otherwise it is set to 1. and surrounding pixels , as shown in formula (5):

[0035] (5)

[0036] S1.6 uses the least squares technique to determine the adjustment scale of each pixel, and the pixel information and The values ​​representing the highlighted and surrounding areas can be found in Using pixel coordinates and As an index, the information of the pixel is similar and The mirror information represents the highlighted area and its surrounding area, and the pixel coordinates can also be used and get , and then get the detected highlight area , as shown in formula (6):

[0037] (6)

[0038] Represents a reshape operation, if directly based on the adjustment scale and Calculation is performed, and what is detected is the highlight pixel information The matrix cannot be directly converted into a matrix of the same dimension as the original image. When finding all the highlight areas of the stripe image Then they are converted and used as masks to prepare for the subsequent stripe repair in these areas. Considering that the area near the highlight area is easily disturbed by "high intensity" noise, these noises will interfere with the subsequent phase recognition and unwrapping algorithms, the repair range is expanded and the extended algorithm is used from get .

[0039] S3: By analyzing the fringe information of the upper and lower adjacent areas of the highlight area, the nonlinear Levenberg-Marquardt fitting algorithm is used to fit the initial values ​​of the phase modulation parameters of the sinusoidal fringes row by row. The specific method is as follows:

[0040] S3.1 Since the pixel intensity in the fringe image is cosine distributed, the pixel information restoration in the highlight area must follow this distribution and be smoothly integrated with the information in the surrounding non-highlight area. Therefore, this paper proposes a nonlinear Levenberg-Marquardt fitting algorithm based on the weight matrix to solve the restoration problem, as shown in formula (7):

[0041] (7)

[0042] in and represent the background intensity and modulation intensity respectively, T represents the edge period, and yes The coordinates of the pixel in ;

[0043] The essence of the fitting algorithm proposed in S3.2 is to optimize the parameters of the model function to minimize the sum of squares of the residuals. Since the pixel information of the highlighted area is unreliable, the area Expand to the region As the basis of the fitting area, as shown in formula (8):

[0044] (8)

[0045] The expanded area is as follows Figure 2 As shown;

[0046] S3.3 and is the coordinate of the pixel in the fitting area. Based on the reflection principle, the connected highlight area observed in the image means that the corresponding surface morphology of the object shows a smooth transition with the non-highlight area. Therefore, the present invention utilizes the image from the extended area. The phase information of Get the initial value of phase modulation parameters based on direction For the purpose of describing the present invention in detail, direction Several rows of pixels above and below the region are used as references, and cosine function fitting is used to derive the initial values ​​of the phase modulation parameters.

[0047] S2: By analyzing the fringe information of the left and right adjacent areas of the highlight area, the background intensity and modulation amplitude of the sinusoidal fringe in the highlight area are determined row by row using the piecewise cubic Hermite interpolation polynomial algorithm. S4: Based on the fringe information of the left and right adjacent areas, a weighted nonlinear Levenberg-Marquardt fitting algorithm is used to fit the phase modulation parameters of the highlight area column by column based on the initial phase modulation parameter values ​​obtained in step S3. The specific method is as follows:

[0048] S2.4.1 In non-highlight areas, the changes in A and B are not significant, such as Figure 3 As shown, the present invention directly adopts the average value obtained by fitting from the reference area. Reflecting the morphological information, the initial value of the phase modulation parameter is derived by interpolation following the principle of smooth transition. Initial value of internal phase modulation parameters Interpolation is performed in the region The fitting results are based on the piecewise cubic Hermite interpolation polynomial (PCHIP), as shown in formula (9):

[0049] (9)

[0050] Here ( ) yes The coordinates of . Coefficients of a piecewise cubic polynomial representing an interval , , and When determined, the polynomial satisfies the interpolation and shape-preserving criteria;

[0051] S2.4.2 Therefore, the system of equations for each interval is as shown in formula (10):

[0052] (10)

[0053] and Yes and The key aspects of PCHIP are They are determined by the weighted harmonic mean of the slopes of adjacent intervals. If the data are monotonic, the interpolation factors will also be monotonic, so for each interval Get the difference and slope ;

[0054] S2.4.3 Once the piecewise cubic polynomial is obtained , you can Incorporate the model to obtain the initial This method ensures that Information along The variability of direction also ensures Information continuity, in order to obtain reliable fitting results using the Levenberg-Marquardt algorithm, the present invention proposes a weight matrix representing the reliability of pixel information , as shown in formula (11):

[0055] (11)

[0056] S2.4.4 According to formula (11), the pixel coordinates are located in the region When , the information of the pixel is considered unreliable, so its weight in the fitting process is assigned to 0, then the elements of the Jacobian matrix are the partial derivatives of the residual with respect to the parameters, as shown in formula (12): (12)

[0057] S2.4.5 The algorithm updates the parameter rules as shown in formula (13):

[0058] (13)

[0059] in Is the damping factor that controls the step size. As the iteration proceeds, the parameter estimate will be updated, and the Jacobian matrix It will be recalculated in each iteration until the algorithm converges to the best-fit parameters, and the process is repeated until convergence is achieved.

[0060] S5: Based on the parameters fitted in steps S2 and S4, the stripe information of the highlight area is calculated row by row. The specific method is as follows:

[0061] By repeatedly calling the weighted Levenberg-Marquardt algorithm, we get ( ) For the area , then the pixel information corresponding to each row is derived according to the cosine function, but along Single direction The information may result in a planar point cloud. In this case, the variable region Levenberg-Marquardt algorithm is used to iteratively calculate Directional Information such as Figure 4 As shown, the calculated information is then substituted into the cosine function to obtain the corresponding pixel information. During the iteration process, the pattern of regional changes follows specific rules, as shown in formula (14):

[0062] (14)

[0063] in Represents the operation of the weighted Levenberg-Marquardt algorithm, which contains two key parameters and Used as the initial value of the phase modulation parameter, Indicates the changing area, that is, the range of the fitting area changes with the calculated The pixel position changes, Indicates the need Area repaired in the direction Pixel index offset within .

[0064] The present invention selects a picture containing highlights as the experimental object, and uses the priori guidance of the strong highlights in the fringe projection profilometry image to finally obtain a highlight suppressed image. Figure 5 As shown, the final image is then analyzed and compared with the original image, as shown in Figure 6 As shown, its stripe information has been repaired. Figure 7The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included in the scope of protection of the present invention.

Claims

1. A priori guided restoration method for strong highlight areas in fringe projection profilometry images, characterized in that: The following steps are involved: S1: Identify and locate the highlight area of ​​the fringe projection profilometry image; S2: By analyzing the fringe information of the left and right adjacent areas of the highlight area, the background intensity and modulation amplitude of the sinusoidal fringe in the highlight area are determined row by row using the piecewise cubic Hermite interpolation polynomial algorithm; S3: By analyzing the fringe information of the upper and lower adjacent areas of the highlight area, the nonlinear Levenberg-Marquardt fitting algorithm is used to fit the phase modulation parameters of the sinusoidal fringes line by line. The initial value of the phase modulation parameters; S4: Based on the fringe information of the left and right adjacent areas, a weighted nonlinear Levenberg-Marquardt fitting algorithm is used to fit the phase modulation parameters of the highlight area column by column based on the initial phase modulation parameter values ​​obtained in step S3; S5: Based on the parameters fitted in steps S2 and S4, the method for calculating the stripe information of the highlight area row by row is as follows: By repeatedly calling the weighted Levenberg-Marquardt algorithm, we get For region R E，i，j , and then the pixel information corresponding to each row is derived according to the cosine function. However, a single Obj information along the u direction may result in a planar point cloud. In this case, the variable area Levenberg-Marquardt algorithm is used to iteratively calculate the u direction. Then, the calculated information is substituted into the cosine function to obtain the corresponding pixel information. During the iteration process, the pattern of regional changes follows specific rules, as shown in formula (10): Where LM(·) represents the operation of the weighted Levenberg-Marquardt algorithm, which contains two key parameters and Used as the initial value of the phase modulation parameter, Indicates the changing area, that is, the range of the fitting area changes with the calculated R E,i，j (u(θ e ), v) varies with the pixel position, θ e Indicates the area R that needs to be repaired in the u direction E,i，j Pixel index offset within .

2. The prior-guided restoration method for strong highlight areas in fringe projection profilometry images according to claim 1, characterized in that: The method of step S1 is: (1) Through the diffuse reflection component I in the fringe image F，i Calculate the threshold T M , whose size is m×n×3, then I F，i Flattened into a two-dimensional matrix M F，i Its size is (mn) × 3, representing the pixels in the RGB channels, starting from M F，i M is obtained in the dark channel F，i,min , as shown in formula (1): M F，i,min (k)=Min <M F，i (l,k)>,a∈[1,mn],b∈[1,3] (1) Where Min<·> represents the operation of finding the minimum value; (2) By calculating M F,i,min The mean and standard deviation of the intensity threshold T M，i To determine the specular reflection and non-highlight components, as shown in formula (2): T M，i =Mean(M F，i,min )+η·Std(M F,i,min ) (2) Mean(·) represents the operation of calculating the mean value, Std(·) represents the operation of calculating the standard deviation, and η is the mirror degree of the fringe image. F，i It determines the intensity and area of ​​the specular highlight, and η = 0.4 is suitable for the proposed FPP system; (3)M F，i,min The value is less than T M，i The value of is considered as non-highlight component, and is greater than T M，i The value of is considered as the highlight component, and the pixels that meet this condition are collected and defined as the geometric factor I β，i , as shown in formula (3): I β，i =(M F，i,min -T M，i )·(M F，i,min >T M，i ) (3) (4) R Dil，i R Dom，i The result after expansion operation is obtained by operating difference R Dil，i and R Dom，i The surrounding pixels get R sur，i ; (5) Use the least squares technique to determine the adjustment scale of each pixel, the pixel information I Sur，i and I Dom，i The values ​​representing the protruding area and the surrounding area can be found in M F，i Using the pixel coordinate R sur，i and R Dom，i As an index, similarly, the pixel information β Sur，i and β Dom，i The mirror information represents the highlighted area and its surrounding area, and the pixel coordinates R Sur，i and R Dom，i Get I β，i , and then get the detected highlight area R H,i,j , as shown in formula (4): Resh<·> means reshaping exercise.

3. The prior-guided restoration method for strong highlight areas in fringe projection profilometry images according to claim 1, characterized in that: The method of step S3 is: (1) Since the pixel intensity in the fringe image is cosine distributed, the pixel information restoration in the highlight area must follow this distribution and be smoothly integrated with the information in the surrounding non-highlight area. Therefore, a nonlinear Levenberg-Marquardt fitting algorithm is proposed to solve the restoration problem, as shown in formula (5): Where A and B represent the background intensity and modulation intensity respectively, T represents the edge period, u and v are R E，i，j The coordinates of the pixel in ; (2) The essence of the proposed fitting algorithm is to optimize the parameters of the model function to minimize the sum of squares of the residuals. Since the pixel information of the highlighted area is unreliable, the area R E，i，j Expand to the region As the basis of the fitting area, as shown in formula (6): (3) u and v are the coordinates of the pixels in the fitting area. Based on the reflection principle, the connected highlight area observed in the image means that the corresponding surface morphology of the object shows a smooth transition with the non-highlight area. Therefore, the image from the extended area is used. The phase information of the phase modulation parameter is obtained based on the v direction. For detailed description, select the phase modulation parameter along the v direction R E，i，j Several rows of pixels above and below the region are used as references, and cosine function fitting is used to derive the initial values ​​of the phase modulation parameters.

4. The method for repairing strong highlight areas in fringe projection profilometry images according to claim 1, characterized in that: The method of step S2 and step S4 is: (1) In the non-highlight area, the changes of A and B are not significant, so the average value obtained from the reference area is directly used. In contrast, Obj reflects the morphological information. Following the principle of smooth transition, interpolation is used to derive the initial value of its phase modulation parameter. E，i，j Initial value of internal phase modulation parameters Interpolation is performed in the region The fitting results are based on piecewise cubic Hermite interpolation polynomials; (2) Once the piecewise cubic polynomial Γ is obtained α , we can incorporate α into the model to get the initial value for each row of pixels This method ensures both the variability of Obj information along the v direction and the continuity of Obj information. In order to obtain reliable fitting results using the Levenberg-Marquardt algorithm, a weight matrix ω representing the reliability of pixel information is proposed. i，j , as shown in formula (7): (3) According to formula (11), the pixel coordinates are located in the region R E，i，j When , the information of the pixel is considered unreliable, so its weight in the fitting process is assigned to 0, then the elements of the Jacobian matrix are the partial derivatives of the residual with respect to the parameters, as shown in formula (8): (4) The algorithm updates the parameter rules as shown in formula (9): Where λ is the damping factor that controls the step size. As the iteration proceeds, the parameter estimate will be updated, and the Jacobian matrix It will be recalculated in each iteration until the algorithm converges to the best-fit parameters, and the process is repeated until convergence is achieved.

Images