Metal surface measurement method and system based on photometric stereo and binocular structured light

Abstract

The application belongs to the technical field of three-dimensional measurement, and particularly discloses a metal surface measurement method and system based on photometric stereo and binocular structured light. The method comprises the following steps: acquiring original depth point cloud of a mirror surface region of a surface to be measured by using a binocular camera; acquiring complete normal vector data of the surface to be measured by using a photometric stereo method; and fusing the original depth point cloud and the normal vector data to calculate surface point cloud of the surface to be measured. The measurement data of the binocular structured light and the near-field photometric stereo are combined, the missing measurement data of the metal surface caused by the mirror reflection is complemented, the mutual fusion of the height data and the normal vector data is utilized, and the accuracy of surface reconstruction is improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of three-dimensional measurement, and more particularly relates to a metal surface measurement method and system based on photometric stereo and binocular structured light. BACKGROUND

[0002] At present, machine vision is widely used in the industrial field. Among them, the structured light three-dimensional measurement technology can restore the surface height by projecting the stripe pattern on the target surface. This method has high measurement accuracy and fast speed. However, this method has the phenomenon of excessive smoothing in some details. The photometric stereo technology in machine vision uses light sources of different directions to irradiate the surface of the object to obtain high-precision surface normal vectors, and then uses the normal vector integration to obtain the surface height. Although photometric stereo can obtain rich surface details, there is a cumulative error in the integration to solve the height, which makes the recovered surface three-dimensional information less accurate.

[0003] In order to solve the above problems, the two methods can be fused to further improve the measurement accuracy. Patent document CN1130487214A discloses a product qualification rate detection system and method based on the combination of photometric stereo technology and structured light technology. This method uses surface structured light to measure the surface three-dimensional information, obtains the normal vector of each point, combines the normal vector data measured by photometric stereo, obtains the optimized normal vector, and then performs three-dimensional reconstruction. Thus, three-dimensional data with higher accuracy and clearer details are obtained. However, this method needs to ensure that both methods obtain complete surface point cloud when fusing. When measuring the metal surface with mirror high light, the data measured by structured light will have missing holes, thereby affecting the accuracy.

[0004] Based on the above defects and deficiencies, there is an urgent need in the art to propose a metal surface fusion measurement method based on photometric stereo and binocular structured light to solve the problem of missing measurement data caused by mirror reflection of the metal surface in the prior art. SUMMARY

[0005] In view of the above defects or improvement needs of the prior art, the present application provides a metal surface measurement method and system based on photometric stereo and binocular structured light, wherein the characteristics of the binocular structured light with higher measurement accuracy in the structured light method and the optimized near-field point light source photometric stereo technology under perspective projection are combined, a metal surface fusion measurement method based on photometric stereo and binocular structured light is correspondingly designed, the original depth point cloud with missing mirror surface area data is obtained by using binocular structured light, and the complete high-precision normal vector data of the measured surface is obtained by using photometric stereo, then the three-dimensional point cloud and the normal vector data are fused, so that the integrity and accuracy of the image are improved. The specific fusion method is to estimate the position error and the normal error by using the three-dimensional point cloud and the normal data respectively, and the sum of the position error and the normal error is minimized by iteration through the optimization method. In order to distinguish the data of the highlight area and other areas, different weight factor values are designed. In addition, the same camera system is used in the two methods of the method, which effectively avoids the additional device error, calibration error and registration error, and the device parameters need not be repeatedly adjusted, only one measurement can estimate the complete point cloud, and the surface reconstruction efficiency is improved.

[0006] To achieve the above object, according to one aspect of the present application, a metal surface measurement method based on photometric stereo and binocular structured light is provided, comprising the following steps:

[0007] S1 uses a binocular camera to obtain an original depth point cloud of a measured surface with missing mirror surface area data. In this step, the binocular structured light technology used is to project a sinusoidal intensity distribution stripe pattern on the target surface, that is, the stripe projection profile method, and through the combination with stereo vision, two cameras are used to shoot the target surface, and the three-dimensional information of the surface is obtained by using phase information for stereo matching.

[0008] S2 uses a photometric stereo method to obtain complete normal vector data of the measured surface. In this step, based on the Lambertian reflection model, the surface irradiance equation is used to describe the brightness of the object surface under light source irradiation, at the same time, in order to solve the problem of mirror reflection of the metal surface, the mirror reflection component in the image needs to be separated, and the shadow also affects the recovery accuracy, so it is also removed. In addition, in this step, a more realistic perspective projection camera model and a near-field point light source model are used to construct the mapping relationship of the coordinate points (x, y) on the image plane to the target surface points (X, Y, Z), and a new surface irradiance equation is derived accordingly:

[0009]

[0010] Where N (x,y)= (n1, n2, n3), k = 1, 2,..., m. Equation (8) is the reconstruction model proposed by the present application. Due to the introduction of the perspective projection and the near-field point light source model, the model defined by the equation is nonlinear. The model has a total of five unknowns (Z, p, n1, n2, n3), wherein the normal vector (n1, n2, n3) is a unit vector, so only two components need to be solved. Because the normal vector is coupled with the surface height, a reference plane Z0 is proposed to replace the height of the real target surface. The direction between the coordinates of each point on the reference plane and the position of each light source is approximated as the light source direction of each point on the target surface, as shown in Figure 7 Then, the surface reflectivity p and the normal vector (n1, n2, n3) are solved by using the least squares method.

[0011] S3 fuses the original depth point cloud and the normal vector data to calculate the surface point cloud of the measured surface.

[0012] Further, before S1, the present application measurement system also needs to be calibrated to obtain the internal and external parameters of the camera. The calibration of the camera in the present application is a conventional technical means. The binocular camera calibration methods in the prior art are all applicable to the present application, and thus, the text will not be described in detail.

[0013] As a further preferred, step S1 comprises the following steps:

[0014] S11 obtains the light intensity of a set of phase-shifted sinusoidal waves at an arbitrary spatial point;

[0015] S12 projects the above phase-shifted sinusoidal waves to the measured surface in sequence to obtain a distorted fringe distribution;

[0016] S13 unwraps the distorted fringe distribution to obtain a continuous phase map, and calculates the three-dimensional coordinates of the points on the measured surface according to the continuous phase map, thereby obtaining the original depth point cloud of the measured surface mirror area data missing.

[0017] As a further preferred, in step S13, the solving equation of the wrapped phase is as follows:

[0018]

[0019] In the formula, I n (x, y) is the distorted fringe distribution, and n represents the phase shift index n = 0, 1, 2,..., N-1.

[0020] As a further preferred, step S2 comprises the following steps:

[0021] S21 uses a surface irradiation equation to describe the brightness of the object surface when illuminated by a light source;

[0022] S22 adopts a pixel intensity separation method to remove highlight and shadow regions, and extracts a Lambert region to calculate a normal vector;

[0023] S23 constructs a mapping relationship from a coordinate point on an image plane to a target surface point according to a perspective projection camera model and a near-field point light source model, obtains a light source direction at each point on the surface according to the mapping relationship and a perspective projection relationship, and solves surface reflectivity and a normal vector according to the light source direction.

[0024] As a further preferred, in step S21, the calculation model of the brightness includes:

[0025]

[0026] In the formula, is the surface point brightness under the kth light source, L k is a light source direction vector, N (x,y) is a surface normal vector, and p is a surface reflectivity coefficient, and the vectors in the equation are unit vectors;

[0027] In step S22, the calculation model of the Lambert region for calculating the normal vector includes:

[0028] O k d = {i k (u,v) ∈ O | αi k min < i k (u,v) < βi k max}

[0029] In the formula, k is a light source label, O k d is a set of extracted Lambert components in the image, i k (u,v) is a pixel intensity at a (u,v) position in the image, and O is a set of all pixel intensities in the image, i k min and i k max are minimum and maximum pixel values in the image target region, respectively, and a and β are separation coefficients;

[0030] In step S22, the normal vector at each pixel point is solved by at least three Lambert points.

[0031] As a further preferred, step S23 includes:

[0032] S231 constructs a perspective projection camera model and a near-field point light source model, and a mapping relationship from a coordinate point (x,y) on an image plane to a target surface point (X,Y,Z) under the model is:

[0033]

[0034] where f is the focal length of the camera;

[0035] S232 the light source direction at each point P on the target surface k (x,y) = (X,Y,Z) is calculated according to the perspective projection relationship; and the line connecting the LED point light source position P k = (X k ,Y k ,Z k ) and the point, and the light source direction at each point on the target surface is obtained by substituting the perspective projection relationship;

[0036] S233 a new surface irradiance equation is constructed according to the calculation model of the brightness and the light source direction at each point on the target surface.

[0037] As a further preferred, step S3 comprises the following steps:

[0038] S31 the path integral method is used to reconstruct the surface to obtain height data according to the complete normal vector data of the surface to be measured;

[0039] S32 it is assumed that the curved surface generated by the image normal is connected by a plurality of small triangular planes, and the vertices of each triangular plane correspond to the center point of a single pixel point in the imaging plane, so that the normal vector data and the three-dimensional coordinates of the original depth point cloud fall on the center point;

[0040] S33 the three-dimensional coordinates of each point on the surface are adjusted using the normal vector at the vertex of the triangular plane, the position error and the normal error expression are derived using the three-dimensional coordinates as the position constraint, the optimized position of each point on the target surface is obtained, and thus the surface point cloud is obtained.

[0041] As a further preferred, step S33 specifically comprises:

[0042] S331 for a given vertex s, the position error E P is defined as the sum of the squares of the distance between the optimized vertex position P s = (X,Y,Z) and the original vertex position P s 0 = (X0,Y0,Z0), and the normal vector direction is used to constrain the adjustment direction of the vertex in the optimization process, and if Δs is the optimized adjustment value, the adjustment vector is ΔsN s ;

[0043] S332 the error of the angle θ between the tangent and the normal vector on the curved surface is optimized, and the normal error E Nthe square sum of the cosine values of the included angle;

[0044] S333 Assuming that r, t are two adjacent vertices of the triangular mesh with s as the vertex, the tangent of the vertex s on the mesh can be represented by the vector , and the three-dimensional coordinates of r and t here are to use the optimized values, that is,

[0045] S334 Considering that the two errors have different influences under different conditions, a weighting factor λ is introduced to construct the position error and normal error expression, and the expression is taken as the objective function;

[0046] S335 The weighted least squares method is used to optimize the objective function, and the optimized position P of each point of the target surface is finally obtained, so as to obtain the surface point cloud.

[0047] More specifically, in the above step S33, in order to improve the accuracy of reconstruction, the normal vector at the vertex is used to adjust the three-dimensional coordinates of each point on the surface, and the three-dimensional coordinates are used as position constraints. The position error and normal error expression is derived to obtain the optimized value of the three-dimensional coordinates.

[0048] For a given vertex s, the position error E P is defined as the square sum of the distance between the optimized vertex position P s =(X, Y, Z) and the original vertex position P s 0 =(X0, Y0, Z0). The normal vector is used to constrain the adjustment direction of the vertex in the optimization process. If Δs is the optimized adjustment value, since the normal vector is a unit vector, the adjustment vector is ΔsN s , and the position error is defined as

[0049]

[0050] On the surface, the normal vector at a point should be perpendicular to the tangent of the surface at the point. The error of the included angle θ between the tangent and the normal vector can be optimized. In order to facilitate calculation, the normal error E N is defined as the square sum of the cosine values of the included angle:

[0051]

[0052] Where T s is the tangent of the triangular plane. Assuming that r, t are two adjacent vertices of the triangular mesh with s as the vertex, the tangent of the vertex s on the mesh can be represented by the vector , and the three-dimensional coordinates of r and t here are to use the optimized values, that is, Substituting equation (14) into equation (14) gives:

[0053]

[0054] Considering that the two errors have different influences under different conditions, the target function is defined as:

[0055]

[0056] The weighted factor λ∈[0, 1] is used to control the role of the normal vector in the optimization process, and the larger λ is, the greater the adjustment role of the normal vector in the optimization process is. Since the recovery effect of binocular structured light measurement is good in high light areas, the value of λ is set to be smaller in these areas. In the area of missing holes, the height value calculated by the integral of the normal vector needs to be adjusted greatly, so the value of λ is set to be larger in the high light area. The specific value needs to be selected according to experience. The optimization process adopts the weighted least square method, and finally the optimized position P of each point on the target surface is obtained, so as to obtain the surface point cloud.

[0057] According to another aspect of the present application, a metal surface measurement system based on photometric stereo and binocular structured light is also provided for implementing the above method.

[0058] Overall, compared with the prior art, the above technical solutions conceived by the present application mainly have the following technical advantages:

[0059] 1. The present application firstly uses binocular structured light to obtain the original depth point cloud of the missing data in the mirror area, and uses photometric stereo to obtain the complete high-precision normal vector data of the measured surface, and then fuses the three-dimensional point cloud and the normal vector data, thereby improving the integrity and accuracy of the image. The specific fusion method is to estimate the position error and the normal error by using the three-dimensional point cloud and the normal data respectively, and to make the sum of the position error and the normal error minimum by iteration through the optimization method. In order to distinguish the data of the high light area from the data of other areas, different weight factor values are designed. In addition, the two methods of the present application use the same camera system, which effectively avoids additional device errors, calibration errors and registration errors, and does not need to repeatedly adjust the device parameters. Only one measurement is needed to estimate the complete point cloud, thereby improving the surface reconstruction efficiency.

[0060] 2. The present application improves the applicability and accuracy of the photometric stereo method by deriving the model of the near-field point light source photometric stereo under perspective projection.

[0061] 3. The present application removes the influence of high light and shadow by pixel intensity, and completes the complete normal vector recovery of the metal surface.

[0062] 4. The present application combines the measurement data of binocular structured light and near-field photometric stereo, not only complements the missing measurement data of the metal surface caused by mirror reflection, but also improves the accuracy of surface reconstruction by mutual fusion and optimization of height data and normal vector data. Attached Figure Description

[0063] Figure 1 This is a flowchart of a preferred embodiment of the present invention relating to a method for measuring metal surfaces based on photometric stereo and binocular structured light;

[0064] Figure 2 This is a flowchart of the binocular structured light algorithm involved in the method of this invention;

[0065] Figure 3 This is a schematic diagram of the binocular structured light measurement device involved in the method of the present invention;

[0066] Figure 4 (a) in the image is the original image captured by the camera when projecting stripes. Figure 4 (b) in the image shows the missing point cloud data in the magnified area.

[0067] Figure 5 This is a flowchart of the near-field photometric stereo algorithm involved in the method of this invention;

[0068] Figure 6 This is a schematic diagram of the camera perspective projection model involved in the method of the present invention;

[0069] Figure 7 This is a schematic diagram of the reference plane involved in the method of the present invention;

[0070] Figure 8 This is a schematic diagram of the triangular mesh division of the imaging plane involved in the method of the present invention.

[0071] In all the accompanying drawings, the same reference numerals denote the same technical features, specifically: 1-camera, 2-target surface, 3-projector. Detailed Implementation

[0072] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0073] like Figure 1 As shown in the figure, the present invention provides a metal surface measurement method based on photometric stereo and binocular structured light. It adopts binocular structured light, which has higher measurement accuracy among structured light methods, and near-field point light source photometric stereo technology under optimized perspective projection, and proposes a fusion measurement method to solve the problem of specular reflection on metal surfaces.

[0074] Firstly, the original depth point cloud with missing mirror region data is obtained by using binocular structured light, and the high-precision normal vector data of the surface to be measured is obtained by using photometric stereo, and then the three-dimensional point cloud and the normal vector data are fused, so that the integrity and accuracy of the image are improved. The specific fusion method is to estimate the position error and the normal error by using the three-dimensional point cloud and the normal data respectively, and the sum of the position error and the normal error is minimized by iteration by using the optimization method. In order to distinguish the data of the highlight area and other areas, different weight factor values are designed. In addition, the two methods of the method use the same camera system, which effectively avoids the additional device error, calibration error and registration error, and does not need to repeatedly adjust the device parameters, and only needs to be measured once to estimate the complete point cloud, thereby improving the surface reconstruction efficiency.

[0075] Step 1: system calibration

[0076] The two methods used in the fusion method (binocular structured light measurement and near-field photometric stereo measurement) need to be calibrated. The binocular structured light needs to be calibrated for the left and right cameras, and the present application uses Zhang Zhengyou calibration method to obtain the internal parameters and relative position of the two cameras. The near-field photometric stereo needs to calibrate the position of the light source, and the present application uses a mirror ceramic ball for calibration.

[0077] Step 2: binocular structured light measurement

[0078] The present application uses binocular structured light measurement technology, and the specific flow chart is as shown in Figure 2 .

[0079] The binocular structured light technology used in the present application is to project a sinusoidal intensity distribution of a stripe pattern to the target surface, that is, the stripe projection profile method, and through the combination with stereo vision, two cameras are used to shoot the target surface, and the three-dimensional information of the surface is obtained by using phase information for stereo matching, and the device schematic diagram is as shown in Figure 3 .

[0080] When a set of phase-shifted sinusoidal waveforms is used, at the point (x p ,y p ) in the projector space, the intensity value is represented as:

[0081]

[0082] Where n represents the phase shift index n=0,1,2,...,N-1. The mean a p and the amplitude b p are usually 0.5 to cover the entire dynamic range of the projector, is the frequency (period / pixel) of the sinusoidal stripe. The present application uses a four-step phase shift method, that is, N=4, ( / 1000 pixels).

[0083] After the pattern is sequentially projected onto the object surface, the distorted fringe distribution captured by the camera (denoted as I n (x,y)) is:

[0084]

[0085] is the corresponding wrapped phase, which can be obtained by the following equation:

[0086]

[0087] It can be seen that the phase obtained by using the arctangent formula is limited in the range of [-π, π], which is called wrapped phase, and needs to be unwrapped to obtain the continuous phase map. The present application uses a three-frequency phase unwrapping method to respectively solve the absolute phase under the left and right cameras, and then calculates the three-dimensional coordinates of each point on the target surface according to the relative position of the cameras obtained by the previous calibration, thereby obtaining the point cloud data.

[0088] The 8-bit gray scale camera used in the present application has an intensity range of 0-255 after imaging. When the pixel intensity captured by the camera exceeds 255 due to the mirror reflection of the object surface, saturation will occur. The metal surface produced by machining belongs to a non-Lambertian surface. For the highlight area, the image intensity I has been saturated, and the fringe information of the point cannot be accurately obtained, so the phase information of the highlight area cannot be obtained. Therefore, for binocular structured light, point cloud missing holes in the highlight area cannot be avoided. Figure 4 The point cloud missing condition when measuring a metal blade is shown.

[0089] Step 3: Near-field photometric stereo measurement

[0090] Photometric stereo technology is a technology for estimating surface normal vectors using images under different direction light sources. The specific flowchart is as shown in Figure 5 .

[0091] The classical photometric stereo is based on the Lambertian reflection model, and the surface irradiation equation is used to describe the brightness of the object surface under the light source:

[0092]

[0093] where I is the brightness of the surface point under the kth light source, L k is the light source direction vector, N (x,y) is the surface normal vector, and ρ is the surface reflectance coefficient. The vectors in the equation are all unit vectors.

[0094] The present application mainly solves the problem of mirror reflection of metal surface, so it is necessary to separate the mirror reflection component in the image, and the shadow also affects the recovery accuracy, so it is removed together. The present application uses pixel intensity separation method to remove highlight and shadow area, extracts Lambert area to calculate normal vector:

[0095] O k d ={i k (u,v)∈O|αi k min <i k (u,v)<βi k max} (5)

[0096] Where k is the light source label, O k d is the set of extracted Lambert components in the image, i k (u,v) is the pixel intensity at position (u,v) in the image, O is the set of all pixel intensities in the image, i k min and i k max are the minimum and maximum pixel values in the image target area respectively, and alpha and beta are separation coefficients. In the case where the light source direction is known, the normal vector at each pixel point can be calculated by three Lambert points, and 14 light sources are set, which theoretically has enough intensity information to recover the surface normal vector.

[0097] Meanwhile, the present application uses a more realistic perspective projection camera model and a near-field point light source model, as Figure 6 shown.

[0098] The mapping relationship of the coordinate point (x,y) on the image plane to the target surface point (X,Y,Z) under this model is:

[0099]

[0100] Where f is the focal length of the camera. Under this model, the light source direction at each point P k (x,y)=(X,Y,Z) on the target surface is parallel to the line connecting the LED point light source position P k =(X k ,Y k ,Z k ) and the point, and substituting the perspective projection relationship, the light source direction at each point on the surface is obtained:

[0101]

[0102] It can be seen that the light source direction of each point on the target surface is directly related to the Z coordinate of the point, which also makes the solution more complicated. Through the above derivation, a new surface irradiance equation is obtained:

[0103]

[0104] where N (x,y) =(n1, n2, n3), k = 1, 2,..., m. Equation (8) is the reconstruction model proposed by the present application. Due to the introduction of perspective projection and near-field point light source model, the model defined by the equation is nonlinear. The model has a total of five unknowns (Z, p, n1, n2, n3), among which the normal vector (n1, n2, n3) is a unit vector, so only two components need to be solved. Because the normal vector is coupled with the surface height, a reference plane Z0 is proposed to replace the height of the real target surface. The direction between the coordinates of each point on the reference plane and each light source position is approximated as the light source direction of each point on the target surface, as shown in Figure 7 Then the surface reflectivity p and the normal vector (n1, n2, n3) are solved by using the least squares method.

[0105] Step 4: Fusion of three-dimensional data and normal vector data to calculate surface point cloud

[0106] After obtaining the complete normal vector data of the target surface, the height data needs to be reconstructed by the normal vector. The method used by the present application is the path integral method. Any surface can be represented by the following formula:

[0107] Z = f (X, Y) (9)

[0108] From equation (2.10), the unit normal vector of any point P (X, Y, Z) on the surface of the object can be expressed as:

[0109]

[0110] Assuming that the measured value of the unit normal vector of a point on the target surface is (n1, n2, n3), then:

[0111]

[0112] The partial derivative along the path W is integrated to obtain the surface, that is:

[0113]

[0114] where W is an arbitrary curve from a fixed point to point (x, y); c is the integral constant, representing the surface height of the starting point.

[0115] In the measurement system of this invention, only the height value at the missing data hole needs to be calculated. Therefore, the height value measured by binocular structured light can be used as the boundary condition in the integration process to calculate the height value for subsequent fusion optimization.

[0116] Any three non-collinear adjacent points on the target surface can define a plane. The surface generated using the image method can be viewed as being composed of many tiny triangular planes connected together, with the vertex of each triangular plane corresponding to the center point of a single pixel in the imaging plane. Furthermore, binocular structured light measurement and near-field photometric stereo use the same set of cameras, naturally aligning the normal vector map and depth map. Therefore, the normal vectors and 3D coordinates of the surface points obtained in the previous steps all fall on this center point. A schematic diagram of the triangular mesh division of the imaging plane is shown below. Figure 8 As shown.

[0117] To improve the accuracy of the reconstruction, the 3D coordinates of each point on the surface are adjusted using the normal vector at the vertex, and these 3D coordinates are used as position constraints. Expressions for position error and normal error are derived to obtain optimized values ​​for the 3D coordinates.

[0118] For a given vertex s, the position error E P Defined as the optimized vertex position P s = (X,Y,Z) and the original vertex position P s 0 =The sum of squares of the distances between (X0, Y0, Z0). The direction of the normal vector is used to constrain the adjustment of vertices during the optimization process. If Δs is the optimized adjustment value, and since the normal vector is a unit vector, the adjustment vector is ΔsN. s The position error is defined as

[0119]

[0120] On a curved surface, the normal vector at a point should be perpendicular to the surface tangent at that point. The error of the angle θ between the tangent and the normal vector can be optimized. For ease of calculation, we define the normal error E. N The sum of the squares of the cosines of the included angle:

[0121]

[0122] Where T s Let r and t be the tangent to the triangular plane. Suppose r and t are two adjacent vertices of a triangular mesh with vertex s, then the tangent to vertex s on this mesh can be represented by the vector t. This indicates that the optimized values ​​should be used for the three-dimensional coordinates of r and t, i.e. Substituting into equation (14), we get:

[0123]

[0124] Considering that the two errors have different influences under different conditions, the target function is defined as:

[0125]

[0126] The weighting factor λ∈[0, 1] is used to control the role of the normal vector in the optimization process. The larger λ is, the greater the adjustment role of the normal vector in the optimization process is. Since the recovery effect of the structured light measurement of binocular structure is good in the high light area, the value of λ in these areas is small. In the area of missing holes, the height value calculated by the integral of the normal vector needs to be adjusted greatly, so the value of λ in the high light area is large. The specific value needs to be selected according to experience. The optimization process adopts the weighted least squares method, and finally the optimized position P of each point of the target surface is obtained, so as to obtain the surface point cloud.

[0127] Those skilled in the art can easily understand that the above description is only a preferred embodiment of the present application, and is not intended to limit the present application. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A photometric stereo and binocular structured light based method for measuring a metal surface, characterized in that, The method comprises the following steps: S1: acquiring original depth point cloud of the measured surface with missing data of the mirror surface region by using a binocular camera; S2: acquiring complete normal vector data of the measured surface by using a photometric stereo method; S3: fusing the original depth point cloud and the normal vector data to calculate the surface point cloud of the measured surface; Step S3 comprises the following steps: S31: reconstructing the surface to obtain height data by using a path integral method according to the complete normal vector data of the measured surface; S32: assuming that the curved surface generated by the image law is connected by a plurality of tiny triangular planes, and the vertex of each triangular plane corresponds to the center point of a single pixel point in the imaging plane, therefore, the three-dimensional coordinates of the normal vector data and the original depth point cloud all fall on the center point; S33: adjusting the three-dimensional coordinates of each point on the surface by using the normal vector at the vertex of the triangular plane, deriving the position error and the normal error expression by using the three-dimensional coordinates as the position constraint, and obtaining the optimized position of each point on the target surface to obtain the surface point cloud.

2. The method of claim 1, wherein, Step S1 comprises the following steps: S11: acquiring the light intensity of a group of sinusoidal waves with phase shift at an arbitrary spatial point; S12: projecting the sinusoidal waves with phase shift to the measured surface in sequence to acquire the distorted fringe distribution; S13: performing unwrapping on the distorted fringe distribution to acquire a continuous phase map, and calculating the three-dimensional coordinates of the points on the measured surface according to the continuous phase map to acquire the original depth point cloud of the measured surface with missing data of the mirror surface region.

3. The method of claim 2, wherein, In step S13, the solving equation of the wrapped phase is as follows: where I n (x,y) is the distorted fringe pattern distribution, n represents the phase shift index n = 0, 1, 2,..., N - 1.

4. The method of claim 1, wherein, Step S2 comprises the following steps: S21: using a surface irradiance equation to describe the brightness of the object surface when the object surface is irradiated by a light source; S22: using a pixel intensity separation method to remove the highlight and shadow area, and extracting a Lambert area to calculate the normal vector; S23: constructing the mapping relationship from the coordinate point on the image plane to the target surface point according to the perspective projection camera model and the near-field point light source model, obtaining the light source direction at each point on the surface according to the mapping relationship and the perspective projection relationship, and solving the surface reflectivity and the normal vector according to the light source direction.

5. The method of claim 4, wherein, In step S21, the calculation model of the brightness comprises: wherein Lkis the surface luminance under the kth light source, L k is the light source direction vector, N (x,y) is the surface normal vector, p is the surface reflection coefficient, and the vectors in the equation are unit vectors; In step S22, the calculation model of the normal vector of the Lambert area comprises: O k d = {i k (u,v) e O | αi k min < i k (u,v) < βi k max} where k is the light source index, O k d is the set of extracted Lambertian components in the image, i k (u,v) is the pixel intensity at position (u,v) in the image, O is the set of all pixel intensities in the image, i k min and i k max are the minimum and maximum pixel values in the image target region, respectively, and a, b are the separation coefficients. In step S22, in the case where the light source direction is known, the normal vector at each pixel point is solved by at least three Lambert points.

6. The method of claim 5, wherein, Step S23 comprises: S231: constructing the perspective projection camera model and the near-field point light source model, and the mapping relationship from the coordinate point (x, y) on the image plane to the target surface point (X, Y, Z) under the model is as follows: In the formula, f is the focal length of the camera; S232 the light source direction at each point P on the target surface under this model k (x,y) = (X,Y,Z) at the light source direction with the LED point light source position P k = (X k ,Y k ,Z k ) and the line connecting this point, and substituting the perspective projection relationship, the light source direction at each point on the target surface is obtained; S233: constructing a new surface irradiance equation according to the calculation model of the brightness and the light source direction at each point on the target surface.

7. The method according to any one of claims 1 to 6, characterized in that, Step S33 specifically comprises: S331 For a given vertex s, the position error E P is defined as the sum of the squared distances between the optimized vertex position P s = (X, Y, Z) and the original vertex position P s 0 = (X0, Y0, Z0), the adjustment direction of the vertex during the optimization process is constrained by the normal vector direction. If Δs is the optimized adjustment value, the adjustment vector is ΔsN s ; S332 On the curved surface, the error of the included angle θ between the tangent and the normal vector is optimized, and the normal error E is defined N is the square sum of the cosine values of the included angles S333 Suppose r, t are two adjacent vertices of the triangular mesh with s as the apex, then the tangent line at vertex s on the mesh can be represented by the vector , and here the three-dimensional coordinates of r and t are to use the optimized values, that is P t = P t 0 + ΔtN t ; S334: considering that the two errors have different influences under different conditions, introducing a weighting factor λ to construct the position error and the normal error expression, and taking the expression as the objective function; S335: optimizing the objective function by using a weighted least square method to finally obtain the optimized position P of each point on the target surface, thereby obtaining the surface point cloud.

8. A photometric stereo and binocular structured light based metal surface measurement system for implementing the method of any one of claims 1-7.

Images