Three-dimensional measurement method based on variable sight line imaging system

Abstract

The invention provides a three-dimensional measurement method based on a variable sight line imaging system, which comprises the following steps of: dynamically scanning and imaging through the variable sight line system, shooting images of a target to be measured under different sight lines, and calculating to obtain three-dimensional coordinate information by utilizing the provided method according to aberration of the images shot by different sight lines and imaging parameters of corresponding sight lines. Meanwhile, the method realizes visual three-dimensional measurement only through a single compact variable-sight-line imaging system, and also solves the problem that visual measurement cannot give consideration to both large scenes and high precision.

Description

Technical Field

[0001] The present invention belongs to the technical field of three-dimensional vision measurement, and in particular relates to a three-dimensional measurement method of a variable line-of-sight imaging system based on a scanning galvanometer. Background Art

[0002] 3D measurement technology involves acquiring three-dimensional data of an object or scene to accurately reconstruct its shape, structure, and position. Traditional 3D measurement methods primarily rely on methods such as laser scanning, structured light, stereo vision, and radar. However, with the development of computer vision technology, visual 3D measurement has become a research area that has attracted considerable attention in recent years. Visual 3D measurement uses a camera to capture scene images and uses image processing algorithms to recover the three-dimensional spatial information. It can be used in a variety of applications, including industrial inspection, autonomous driving, virtual reality, cultural heritage preservation, and architectural surveying. Visual 3D measurement does not require contact with the object, making it suitable for fragile or hard-to-reach objects. Compared with traditional methods such as laser scanning, visual 3D measurement systems typically use ordinary cameras and computers, resulting in lower equipment costs. Software algorithms can flexibly adjust parameters such as viewing angle and focal length to adapt to different scenarios, offering high flexibility. However, as high-end manufacturing industries, such as rail transit, aerospace, shipbuilding, and wind turbines, increasingly utilize larger and more integrated equipment, industrial 3D measurement also demands large-scale measurement capabilities.

[0003] In large-scale, industrial inspection applications, visual inspection systems must be able to cover a wide field of view while also having high resolution for the local details of the target object. Traditional monitoring systems face an irreconcilable conflict between detection range and high local resolution, requiring both to be met simultaneously. To address this issue, a large-scale 3D measurement system with variable line-of-sight imaging based on a scanning galvanometer is proposed. By leveraging the scanning galvanometer's ability to deflect light, the field of view is dynamically altered, expanding the detection range without sacrificing local details. Furthermore, during the variable line-of-sight imaging process, target feature points can be reconstructed for 3D measurement applications.

[0004] In the prior art, for example, CN113175899A discloses a three-dimensional imaging model of a variable-line-of-sight imaging system combining a camera and a galvanometer, and a calibration method thereof; CN116823964A discloses a physical parameter model of a camera-galvanometer variable-line-of-sight system, and a calibration method thereof. Although modeling and calibration methods for variable-line-of-sight imaging systems are proposed, the above-mentioned technical contents both model and calibrate the variable-line-of-sight imaging systems using two different methods, respectively, and establish a correspondence between imaging pixels and spatial information of different variable-line-of-sight imaging systems. These patents cannot realize the three-dimensional measurement function of a single variable-line-of-sight imaging system. Summary of the Invention

[0005] Technical solution: In order to solve the above technical problems, the present invention proposes a method for reconstructing target feature points of a variable line of sight imaging system under the three-dimensional measurement function of only a single variable line of sight imaging system. The specific technical content is as follows:

[0006] A three-dimensional measurement method based on a variable line of sight imaging system, wherein the method is based on measurement performed by a variable line of sight imaging device, and the specific steps are as follows:

[0007] Step 1: calibrate the variable-view imaging device to obtain spatial information corresponding to pixel coordinates under different deflections;

[0008] Step 2: while controlling the galvanometer deflection, scan and shoot the target to be measured to obtain images of the target to be measured under different lines of sight;

[0009] Step 3: Select the scanned images at different sight lines B n The target image I is captured at the same time n , through I n Perform image feature extraction and matching to obtain the spatial point P m In different images i n Pixel coordinates in

[0010] Step 4: Calculate the image i by using the calibration results of the variable line of sight imaging device n Pixel coordinates in The corresponding spatial information is calculated to obtain the spatial point P m The three-dimensional coordinates of .

[0011] As an improvement, the calibration method in step 1 selects a data-driven calibration method, specifically, first calibrating the spatial imaging line parameters (n x ,n y ,n z ,c x ,c y ,c z ), where (n x ,n y ,n z ) is the direction vector of the straight line, (c x ,c y ,c z ) is the coordinate of the line control point; then establish the four-element imaging factor Q(a, b, u, v) to the six-element line parameter L(n x ,n y ,n z ,c x ,c y ,c z ) mapping M:Q→L.

[0012] As an improvement, after calibration, the method for calculating the spatial feature point P in step 4 is:

[0013] (1) Calculate the initial value of the three-dimensional coordinates according to the collinearity constraint of the spatial feature points P;

[0014] (2) Establish a nonlinear optimization model based on minimizing the sum of the distances from the spatial feature point P to the straight line, bring the calculated initial values of the three-dimensional coordinates into the optimization model, and solve to obtain the optimal solution of the three-dimensional point coordinates.

[0015] As an improvement, the specific method for calculating the initial value of the three-dimensional point is:

[0016] (1) Image I n Corresponding galvanometer deflection control value (a, b) n and pixel p in the image n The imaging factor Q formed by the pixel coordinates (u, v) n (a, b, u, v) is brought into the mapping M to obtain the spatial line L corresponding to the feature point on the image n (n x ,n y ,n z ,c x ,c y ,c z ), whose equation is expressed in dot form as:

[0017] [x,y,z] ％ =ξ(n x ,n y ,n z ] ％ +(c x ,c y ,c z ] ％ (Formula 1)

[0018] Where ξ is the coefficient of the straight line;

[0019] (2) The spatial point P is on the line L n (n x ,n y ,n z ,c x ,c y ,c z ), then bring the P coordinate (X, Y, Z) into the straight line L n public

[0020] In formula 1, we have:

[0021]

[0022] After linear elimination of ξ, we can get:

[0023]

[0024] There are N images I n (n=1,2,…,N;N≥2) When the spatial point P is observed at the same time, N linear equations of formula 3 are obtained. The equations are combined and solved by the least squares method to obtain the solution of the equation group, which is the initial value of the spatial point to be determined.

[0025] As an improvement, the specific steps to obtain the optimal solution after calculating the initial value of the three-dimensional coordinates of the spatial point in step 4 are as follows:

[0026] (1) Establish an optimization model to minimize the sum of the imaging straight-line distances from a spatial point to the corresponding pixel points on all lines of sight:

[0027]

[0028] Among them, dist((a,b) n ,p n ,P) represents the spatial point P and the galvanometer deflection control amount (a,b) n Lower pixel coordinate p n The distance function of the corresponding spatial straight line calculated by the data-driven model, is the three-dimensional coordinate of the optimal spatial point;

[0029] (2) The least squares solution of the linear equations is substituted into the optimization model as the initial value, and the optimal solution of the model is calculated by the Levenberg-Marquardt algorithm, which is the three-dimensional coordinate of the spatial point to be measured.

[0030] As an improvement, the calibration method in step 1 selects the calibration method of physical parameters. Calibrate first. After the calibration is completed, the camera imaging parameter matrix K, distortion parameter D, and the control amount of different galvanometer deflection (a, b) of the variable line of sight imaging system are obtained. n The gaze pose matrix [R|t] under n .

[0031] As an improvement, after calibration, the method for calculating the spatial feature point P in step 4 is:

[0032] (1) Calculate the initial three-dimensional coordinates based on the pose constraints of the spatial feature point P under multiple views;

[0033] (2) Establish a reprojection error and minimum nonlinear optimization model based on the spatial feature point P to the imaging line of sight, bring the calculated initial value of the three-dimensional coordinate into the optimization model, and solve to obtain the optimal solution of the three-dimensional point coordinates.

[0034] The specific calculation steps for calculating the initial value of the three-dimensional coordinates of the spatial feature point P are as follows:

[0035] (1) According to image I n Corresponding galvanometer deflection control value (a, b) n Bring in the physical parameter model and get the current sight pose matrix [R|t] n ;

[0036] (2) According to image I n Middle pixel p n Pixel coordinates (u,v) n , and the parameters of the variable line of sight imaging system, let:

[0037]

[0038] Substituting formula 5 into the camera pinhole imaging model yields:

[0039]

[0040] Among them, s is the imaging scale factor; matrix K is the perspective projection transformation matrix, and then multiply K by Formula 6 on the left 91 , then:

[0041]

[0042] Let [x,y,1] be T =K 91 [u,v,1] ％ , then:

[0043]

[0044] Eliminate the coefficient s and sort out the linear equations for the coordinates (X, Y, Z) of the point to be solved:

[0045]

[0046] There are N images I taken under different sight lines n (n=1,2,…,N;N≥2)When the spatial point is observed at the same time, N groups of linear equations shown in Formula 9 are established, and the overdetermined equations about the coordinates (X,Y,Z) of the spatial point are obtained by combining them. The solution of the equations obtained by the least squares method is the initial value of the spatial point to be determined.

[0047] The specific steps to obtain the optimal solution after calculating the initial value of the three-dimensional point are:

[0048] (1) Establish an optimization model that minimizes the sum of the distances between the spatial point and the reprojected pixel points on all sight lines and the actual captured pixel points:

[0049]

[0050] Among them, reproj((a,b) n ,p n ,P) represents the deflection control signal (a,b) at the spatial point P n The reprojected pixel coordinates calculated based on the physical parameter model of the variable line of sight imaging system and the pixel coordinates p in the image are n The distance error function between Indicates the optimal solution of the three-dimensional coordinates of the spatial point under this model;

[0051] (2) The least squares solution of the linear equations is substituted as the initial value, and the optimal solution of the model is calculated by the Levenberg-Marquardt algorithm, which is the three-dimensional coordinate of the point to be measured.

[0052] Beneficial effect: The three-dimensional measurement method of the variable line of sight imaging system based on the combination of a two-dimensional scanning galvanometer and a visual camera proposed in the present invention is based on the parallax generated when the variable line of sight imaging system observes the same target point from different lines of sight during the scanning imaging process, and realizes three-dimensional reconstruction of the target point according to the proposed calculation method.

[0053] This method innovatively uses only a compact image acquisition device to quickly reconstruct the target in three dimensions without any external interference. It solves the problem that traditional visual three-dimensional reconstruction requires multiple cameras or a single camera needs to be moved multiple times. It has high application value in industrial manufacturing inspection, autonomous driving visual sensing, and drone navigation. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 This is a flow chart of the three-dimensional measurement method of the variable line of sight imaging system of the present invention. DETAILED DESCRIPTION

[0055] The technical solutions in the embodiments of the present invention will be described clearly and completely below so that those skilled in the art can better understand the advantages and features of the present invention and thus more clearly define the scope of protection of the present invention. The embodiments described in the present invention are only some of the embodiments of the present invention, not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without making any creative work shall fall within the scope of protection of the present invention.

[0056] This invention provides a three-dimensional measurement method based on a variable-line-of-sight imaging system. This method uses the system to dynamically scan and image the target under test under different lines of sight. Based on the aberrations of the images captured at different lines of sight and the imaging parameters of the corresponding lines of sight, the provided method is used to calculate the three-dimensional coordinate information. This method achieves visual three-dimensional measurement using only a single, compact variable-line-of-sight imaging system, further resolving the issue of visual measurement's inability to balance large scenes with high precision.

[0057] Example 1

[0058] Based on the data-driven model of the variable line-of-sight imaging system, the three-dimensional measurement method provided by this patent is used to perform three-dimensional reconstruction of spatial points.

[0059] Step 1: Using a movable glass plate covered with coding dots as a calibration plate, the variable line-of-sight system is controlled to scan and capture the plate. The plate is moved multiple times to generate spatial points corresponding to pixels at different distances. The corresponding spatial points are then fitted to obtain imaging light lines, resulting in a total of 137,423 sets of sample data for the mapping M:Q→L. This mapping is fitted using a single hidden layer neural network (SLFN), and the neural network parameters are rapidly solved using an extreme learning machine (ELM), resulting in the final calibration results for the data-driven model.

[0060] Step 2: Perform three-dimensional measurement on a standard-sized ruler placed in space. There are three coding blocks on the ruler, and each coding block has 8 marking points. m,k ,m=1,2,3;j=1,2,…,8 to represent the marking points on each coding block. In the experiment, the ruler is placed about 3m away from the variable line of sight imaging system, and the variable line of sight imaging system is controlled to scan and shoot, and 27 galvanometer deflection control signals (a, b) are obtained. n ,n=1,2,…,27 The calibration rod image I captured by the sight under control n .

[0061] Step 3: Extract all captured scale images I n The pixel coordinates of the marker points in the coding block in the , and the marker points on all the extracted coding blocks are matched with the same name points, and the pixel coordinates are recorded

[0062] Step 4: Calculate the image I by using the calibration results of the variable line of sight imaging device n Pixel coordinates in The corresponding spatial information is calculated to obtain the feature point The corresponding space point P m,k The three-dimensional coordinates of .

[0063] To evaluate the accuracy of 3D point reconstruction, the distances between the eight markers on encoder block 2 and the 16 markers on encoder blocks 1 and 3, respectively, were calculated for both the actual and reconstructed ruler points. The distance errors between the two were calculated, yielding 128 error results with a mean of 1.007 mm and a standard deviation of 0.835 mm. The data-driven model-based 3D reconstruction method achieved a reconstruction accuracy of approximately 1 mm for a ruler approximately 1 meter long at a distance of 3 meters, validating the effectiveness of this method.

[0064] Example 2

[0065] Based on the physical model of the variable line of sight imaging system, the three-dimensional measurement method provided by this patent is used to perform three-dimensional reconstruction of spatial points.

[0066] Step 1: Control the variable line of sight imaging system to shoot a calibration plate with 99 evenly distributed marking points, and move the position of the calibration plate multiple times to acquire a total of 16 images. Calibrate the internal parameters of the camera of the variable line of sight imaging system to obtain the imaging matrix K and distortion parameter D. Then, calibrate the galvanometer deflection parameters of the variable line of sight imaging system through a glass curtain wall with a coding block to obtain the initial deflection angle α. q , β q And the first-order coefficient k of the deflection angle with respect to the deflection control quantity tS 、k uS Finally, the variable sightline imaging system is calibrated by scanning and photographing the entire glass curtain wall. The specific steps are as follows:

[0067] 1. Pre-plan the values of the deflection control signal quantity. a and b are respectively a=5n-50, b=5n-50, n=1,2,…,19, 19 values each, and a total of 19×19=361 sets of control signal quantities (a, b) are obtained. n ;

[0068] 2. Place the variable sight imaging system in a suitable position pos l At (a,b) n , control the sight deflection shooting, extract the center of the mark point on the code point block in the 361 captured images, and get the l Different sight lines B n The pixel coordinates of the marker points captured Through the code point block number id and point P on the calibration plate id Match and get matching point pairs

[0069] 3. According to the matching point Calculate each line of sight B through the PnP algorithm n The corresponding virtual camera pose matrix The calculated And the corresponding control signal (a, b) n Substitute the formula AX=ZB to construct an overdetermined set of equations and solve it to obtain the relative pose matrix between the camera and the galvanometer and at position pos l The galvanometer pose matrix under

[0070] 4. Change the position of the variable sight imaging system and the calibration wall, repeat steps 2 and 3, and get a total of 3 positions pos l , l=1,2,3 The coordinates of the marker points collected And get the solution results respectively

[0071] 5. The relative pose matrix between the camera and the galvanometer is obtained In pos l , the galvanometer pose matrix at positions l = 1, 2, 3 The previously calibrated galvanometer parameter vector g, camera imaging matrix K, and camera distortion parameter d are used as initial values and substituted into Equation (11) for nonlinear optimization to obtain the optimal parameters, thus completing the calibration of the physical parameter model.

[0072]

[0073] in, is a spatial point P id The variable line of sight imaging system is located at pos l At, in sight B n The reprojected pixel coordinates under .

[0074] Step 2: Reconstruct the marker points in the coded point block pasted on a target object with shape features, control the variable line of sight imaging system to take multiple line of sight images of the target object placed in the field of view, and take an image every 2 units of signal quantity, and take a total of 81 target images I n ,n=1,2,…,81.

[0075] Step 3: Extract all captured scale images I n The pixel coordinates of the marker points in the coded blocks in the 18 extracted coded blocks are matched with the same-name points, and the pixel coordinates are recorded.

[0076] Step 4: Calculate the image I by using the calibration results of the variable line of sight imaging device n Pixel coordinates in The corresponding spatial information is calculated to obtain the feature point The corresponding space point Pz{ The three-dimensional coordinates of .

[0077] In order to evaluate the accuracy of 3D point reconstruction, a commercial measurement device TRITOP was used to reconstruct the marker points on the target to obtain a total of 144 spatial point coordinates on 18 coding blocks. We calculated the distances between each pair of the 144 markers measured by TRITOP, yielding a total of 10,296 distance values. We also calculated the distances between the 144 markers reconstructed using the physical parameter imaging model, comparing the error between the two sets of distance values. The root mean square error between the point-to-point distances in the two sets of points was calculated to be 0.398 mm, confirming the effectiveness of this method.

[0078] The above-described embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the patent for this invention shall be determined by the appended claims.

Claims

1. A three-dimensional measurement method based on a variable line-of-sight imaging system, characterized by: The method is based on measurement performed by a variable line of sight imaging device, and the specific steps are: Step 1: calibrate the variable-view imaging device to obtain spatial information corresponding to pixel coordinates under different deflections; Step 2: while controlling the galvanometer deflection, scan and shoot the target to be measured to obtain images of the target to be measured under different lines of sight; Step 3: Select N, N≥2 galvanometer deflection control signals (a, b) from the scanned image. n (n=2,3,…,N) simultaneously captures the target image I n , by image I n Perform image feature extraction and matching to obtain spatial feature points P in different images I n The pixel coordinate p in n ; Step 4: Calculate the image I by using the calibration results of the variable line of sight imaging device n The pixel coordinate p in n The corresponding spatial information is further calculated to obtain the three-dimensional coordinates of the spatial feature point P.

2. The three-dimensional measurement method based on the variable line of sight imaging system according to claim 1, characterized in that: In step 1, the calibration method is a data-driven calibration method. The calibration method is: first calibrate the spatial imaging line parameters (n) corresponding to the pixel point (u, v) under different galvanometer deflection control values (a, b). x ,n y ,n z ,c x ,c y ,c z ), where (n x ,n y ,n z ) is the direction vector of the straight line, (c x ,c y ,c z ) is the coordinate of the line control point; then establish the four-element imaging factor Q(a, b, u, v) to the six-element line parameter L(n x ,n y ,n z ,c x ,c y ,c z ) mapping M:Q→L.

3. The three-dimensional measurement method based on the variable line of sight imaging system according to claim 2, characterized in that: After calibration, the method for calculating the spatial feature points P in step 4 is: (1) Calculate the initial value of the three-dimensional coordinates according to the collinearity constraint of the spatial feature points P; (2) Establish a nonlinear optimization model based on minimizing the sum of the distances from the spatial feature point P to the straight line, bring the calculated initial values of the three-dimensional coordinates into the optimization model, and solve to obtain the optimal solution of the three-dimensional point coordinates.

4. The three-dimensional measurement method based on the variable line of sight imaging system according to claim 3, characterized in that: The specific method for calculating the initial value of a three-dimensional point is: (1) Image I n Corresponding galvanometer deflection control value (a, b) n and pixel p in the image n The imaging factor Q formed by the pixel coordinates (u, v) n (a, b, u, v) is brought into the mapping M to obtain the spatial straight line corresponding to the feature point on the image L n (n x ,n y ,n z ,c x ,c y ,c z ), whose equation is expressed in dot form as: [x,y,z] $ =ξ[n x ,n y ,n z ] $ +[c x ,c y ,c z ] $ (Formula 1) Where ξ is the coefficient of the straight line; (2) The spatial point P is on the line L n (n x ,n y ,n z ,c x ,c y ,c z ), then bring the P coordinate (X, Y, Z) into the straight line L n In formula 1, we have: After linear elimination of ξ, we can get: There are N images I n (n=1,2,…,N;N≥2) When the spatial point P is observed at the same time, N linear equations of formula 3 are obtained. The equations are combined and solved by the least squares method to obtain the solution of the equation group, which is the initial value of the spatial point to be determined.

5. The three-dimensional measurement method based on the variable line of sight imaging system according to claim 3 or 4, characterized in that: The specific steps to obtain the optimal solution after calculating the initial value of the three-dimensional point are: (1) Establish an optimization model to minimize the sum of the imaging straight-line distances from a spatial point to the corresponding pixel points on all lines of sight: Among them, dist((a,b) n ,p n ,P) represents the spatial point P and the galvanometer deflection control amount (a,b) n Lower pixel coordinate p n The distance function of the corresponding spatial straight line calculated by the data-driven model, is the three-dimensional coordinate of the optimal spatial point; (2) The least squares solution of the linear equations is substituted as the initial value, and the optimal solution of the model is calculated by the Levenberg-Marquardt algorithm, which is the three-dimensional coordinate of the point to be measured.

6. The three-dimensional measurement method based on the variable line of sight imaging system according to claim 1, characterized in that: In step 1, the calibration method selects the physical parameter calibration method, and calibrates first. After the calibration is completed, the camera imaging parameter matrix K, distortion parameter D, and the galvanometer deflection control amount (a, b) of the variable line of sight imaging system are obtained. n The gaze pose matrix [R|t] under n .

7. The three-dimensional measurement method based on the variable line of sight imaging system according to claim 6, characterized in that: After calibration, the method for calculating the spatial feature points P in step 4 is: (1) Calculate the initial three-dimensional coordinates based on the pose constraints of the spatial feature point P under multiple views; (2) Establish a reprojection error and minimum nonlinear optimization model based on the spatial feature point P to the imaging line of sight, bring the calculated initial value of the three-dimensional coordinate into the optimization model, and solve to obtain the optimal solution of the three-dimensional point coordinates.

8. The three-dimensional measurement method based on the variable line of sight imaging system according to claim 7, characterized in that: The specific calculation steps for calculating the initial value of the three-dimensional coordinates of the spatial feature point P are as follows: (1) According to image I n Corresponding galvanometer deflection control value (a, b) n Bring in the physical parameter model and get the current sight pose matrix [R|t] n ; (2) According to image I n Middle pixel p n Pixel coordinates (u,v) n , and the parameters of the variable line of sight imaging system, let: Substituting formula 5 into the camera pinhole imaging model yields: Among them, s is the imaging scale factor; matrix K is the perspective projection transformation matrix, and then multiply K by Formula 6 on the left 61 , then: Let [x,y,1] be $ =K 61 [u,v,1] $ , then: Eliminate the coefficient s and sort out the linear equations for the coordinates (X, Y, Z) of the point to be solved: There are N images I taken under different sight lines n (n=1,2,…,N;N≥2)When the spatial point is observed at the same time, N groups of linear equations shown in Formula 9 are established, and the overdetermined equations about the coordinates (X,Y,Z) of the spatial point are obtained by combining them. The solution of the equations obtained by the least squares method is the initial value of the spatial point to be determined.

9. The three-dimensional measurement method based on the variable line of sight imaging system according to claim 7 or 8, characterized in that: The specific steps to obtain the optimal solution after calculating the initial value of the three-dimensional point are: (1) Establish an optimization model that minimizes the sum of the distances between the spatial point and the reprojected pixel points on all sight lines and the actual captured pixel points: Among them, reproj((a,b) n ,p n ,P) represents the deflection control signal (a,b) at the spatial point P n The reprojected pixel coordinates calculated based on the physical parameter model of the variable line of sight imaging system and the pixel coordinates p in the image are n The distance error function between Indicates the optimal solution of the three-dimensional coordinates of the spatial point under this model; (2) The least squares solution of the linear equations is substituted as the initial value, and the optimal solution of the model is calculated by the Levenberg-Marquardt algorithm, which is the three-dimensional coordinate of the point to be measured.

Images