Three-dimensional shape measurement method for imaging across special-shaped transparent mechanism

Abstract

A three-dimensional shape measurement method for imaging across a special-shaped transparent mechanism comprises the steps that in the calibration stage, a binocular camera and a calibration plate are arranged on the two sides of the special-shaped transparent mechanism respectively, an image of the calibration plate is collected through the special-shaped transparent mechanism, and therefore a binocular refraction model containing the characteristics of the special-shaped transparent mechanism is established, and calibration is achieved through a single image calibration method; in the online stage, through the calibrated binocular refraction model, a to-be-measured object image collected in real time is processed to obtain a three-dimensional shape. Aiming at the defects that the calibration process is not suitable for on-machine measurement and the influence of porthole refraction is not considered, so that the measurement precision is low, the influence of the refraction effect on the object three-dimensional shape measurement method can be corrected, the whole-field shape measurement precision is 0.4%, and the actual engineering measurement requirement is met.

Description

Technical Field

[0001] This invention relates to a technology in the field of binocular vision, specifically a three-dimensional topography measurement method for imaging across irregularly shaped transparent structures. Background Technology

[0002] Existing optical measurement systems installed within the cabin fail to account for image distortion caused by the refraction effect of multi-layered glass structures in portholes when measuring the morphology of objects outside the cabin, thus affecting the accuracy of the measurement results. Furthermore, existing methods for correcting the effects of cross-medium imaging often only describe the refracted light path through parallel transparent mechanisms, which is not applicable to multi-layered structures of varying shapes. Additionally, current technologies require prior information on the calibration plate's position, meaning that the transparent structure must be disassembled for pre-calibration to determine the plate's position before reassembling the transparent structure for overall model calibration. This disassembly and assembly process is unsuitable for onboard online measurements and cannot describe the impact of irregularly shaped transparent mechanisms on the optical path. Summary of the Invention

[0003] To address the aforementioned shortcomings of existing technologies, this invention proposes a three-dimensional topography measurement method for imaging across irregularly shaped transparent mechanisms. It addresses the deficiencies of low measurement accuracy caused by the inapplicability of on-machine measurement during calibration and the failure to consider the effects of porthole refraction. This method can correct the influence of porthole refraction on the three-dimensional topography measurement method, achieving a full-field topography measurement accuracy of 0.4%, which meets the requirements of practical engineering measurements.

[0004] This invention is achieved through the following technical solution:

[0005] This invention relates to a three-dimensional topography measurement method for imaging across an irregularly shaped transparent mechanism. In the calibration stage, a binocular camera and a calibration plate are respectively set on both sides of the irregularly shaped transparent mechanism, and images of the calibration plate are acquired through the irregularly shaped transparent mechanism to establish a binocular refraction model containing the characteristics of the irregularly shaped transparent mechanism. The calibration is then achieved through a single-image calibration method. In the online stage, the real-time acquired images of the object under test are processed using the calibrated binocular refraction model to obtain the three-dimensional topography.

[0006] The aforementioned binocular refraction model refers to: assuming the camera acts as a reverse light source, and establishing a binocular refraction model using the law of refraction. ,in: Let be the refraction matrix, k be the depth coefficient, [X,Y,Z]' be the physical coordinates of any point in space, [u,v]' be the pixel coordinates of that point mapped to the imaging plane, A be the camera intrinsic parameter matrix, I be the identity matrix, [R,T] be the rotation and translation relationship between the two cameras, and subscripts 1 and 2 represent the two cameras respectively.

[0007] The aforementioned irregularly shaped transparent mechanism includes a planar transparent mechanism and a cylindrical transparent mechanism.

[0008] The aforementioned refraction matrix , where: matrix , rodrigues(m,n) is the Rodrigues rotation formula, which represents a rotation about n as the axis of rotation and m as the angle. , N1 is the surface normal vector of the flat transparent mechanism, and N2 and N3 are the normal vectors corresponding to the intersection points of the imaging light rays when they cross the front and rear surfaces of the cylindrical transparent mechanism, respectively; d1 is the distance from the camera to the flat transparent mechanism, d2 is the thickness of the transparent mechanism, d3 is the distance from the flat transparent mechanism to the cylindrical transparent mechanism, and d4 is the thickness of the cylindrical transparent mechanism; v0 is the unit direction vector of the outgoing light obtained according to the camera intrinsic parameters A, pixel coordinates [u, v], and camera focal length f, r = ‖[u, v, f]‖; air refractive index n1, and porthole refractive index n2.

[0009] The calibration process involves using the Harris corner detection algorithm to extract image coordinates of feature points in images captured from the calibration plate positioned outside the porthole, based on the pose relationship between the camera and the calibration plate, as well as the camera's internal parameters. This information is then substituted into the binocular refraction model. Specifically, this includes:

[0010] Step 1) Based on the pose relationship between the porthole and the single calibration plate relative to the two cameras, obtain the objective function. , where: first error , , These represent the spatial coordinates of each feature point on the calibration plate relative to the two cameras, calculated using a binocular refraction model; the second error... , The result was obtained using a binocular refraction model. The spatial coordinates of adjacent feature points, where const is the actual distance between adjacent feature points on the calibration plate; and n is the number of feature points.

[0011] Step 2) Set the initial parameters and their range for the binocular refraction model;

[0012] The initial parameters include: parameters to be optimized and fixed parameters, wherein: the parameters to be optimized include the spatial positioning parameters of the flat transparent mechanism, the spatial positioning parameters of the cylindrical transparent mechanism, and the spatial positioning parameters of the calibration plate; the fixed parameters include the refractive index of the porthole, the thickness of the flat transparent mechanism, the inner and outer diameters of the cylindrical transparent mechanism, and the actual distance between adjacent feature points of the calibration plate.

[0013] All the spatial positioning parameters to be optimized can be roughly measured using simple measuring tools such as calipers, and the optimization range is set to ±10%.

[0014] Step 3) Substitute the initial parameters into the binocular refraction model and calculate the objective function. If the objective function is less than the first threshold, proceed to step 6; otherwise, continue to step 4.

[0015] Step 4) Change the values ​​of the parameters to be optimized based on the Sequential Quadratic Programming (SQP) algorithm, substitute them into the binocular refraction model, and calculate the updated objective function;

[0016] Step 5) If the updated objective function is less than the first threshold or the difference between the objective function before and after the update is greater than the second threshold, proceed to step 6; otherwise, return to step 4.

[0017] Step 6) Substitute the obtained parameters into the calibration binocular refraction model. ,in: , This is the calibrated homography matrix for each camera.

[0018] The aforementioned three-dimensional topography is obtained as follows: based on a binocular refraction model, the pixel coordinates of any feature point on the surface of the object under test in both cameras are determined using a feature point matching algorithm. )and( By traversing all feature points using the least squares method, the spatial coordinates of the entire field shape of the object under test can be obtained. , , ).

[0019] The field of view of the binocular camera covers the entire calibration board to ensure that all feature points on the same calibration board can be imaged in the binocular camera.

[0020] Technical effect

[0021] This invention improves the accuracy of object shape measurement by establishing a binocular camera-based refractive imaging optical path model across an irregularly shaped transparent structure and correcting the influence of optical refraction on image point distortion caused by the irregularly shaped transparent structure. Furthermore, the method calibration improves the accuracy of model parameters, thereby enhancing the accuracy of object shape measurement. Simultaneously, it avoids the cumbersome steps of disassembling and reinstalling the transparent structure, thus enabling in-machine measurement and real-time correction. Attached Figure Description

[0022] Figure 1 This is a flowchart of the present invention;

[0023] Figure 2 This is a schematic diagram of the layout for an example embodiment;

[0024] Figure 3 This is a schematic diagram of an actual experiment for an example. Detailed Implementation

[0025] like Figure 2 As shown, this embodiment relates to a three-dimensional topography measurement device 1 for imaging across a porthole, including: a first camera 101 and a second camera 102 constituting the measurement unit 1, wherein: the first camera 101 and the second camera 102 are simultaneously installed in the cabin 3, and acquire images of the calibration plate 6 or the object to be measured 7 mounted on the wing 5 through the porthole 4, and the measurement unit 1 is connected to the calculation unit 2.

[0026] The computing unit 2 includes a single image calibration module and a three-dimensional shape measurement module. The single image calibration module extracts corner points and substitutes them into the binocular refraction model and optimizes the objective function based on the image information of the calibration plate 6 set outside the cabin acquired by the binocular camera to obtain the optimal solution result of the parameters to be optimized. The three-dimensional shape measurement module extracts feature points and substitutes them into the fully calibrated binocular refraction model based on the image information of the object to be measured 7 to obtain the full-field shape of the object to be measured.

[0027] like Figure 1 As shown, this embodiment illustrates a three-dimensional topography measurement method for imaging across irregularly shaped transparent mechanisms based on the aforementioned device, specifically including:

[0028] S1. Set up measuring device 1.

[0029] S2. Obtain the internal and external parameters of the first camera 101 and the second camera 102 through camera calibration methods.

[0030] The camera calibration method used in this embodiment is Zhang's calibration method.

[0031] S3. Images of calibration board 6 are acquired using the first camera 101 and the second camera 102.

[0032] S4. The constructed binocular refraction model is calibrated using a single-image calibration method, specifically including:

[0033] 4.1) Substitute the pixel coordinates of the feature points in calibration board 6 and the actual spacing between each feature point into the binocular refraction model to obtain the calculated spatial coordinates of the feature points. Specifically, assuming the camera is a reverse light source, a binocular refraction model is established using the law of refraction. ,in: , where: matrix , rodrigues(m,n) is the Rodrigues rotation formula, which represents a rotation about n as the axis of rotation and m as the angle. , N1 is the surface normal vector of the flat transparent mechanism, N2 and N3 are the normal vectors at the intersection points of the imaging light rays when they cross the front and rear surfaces of the cylindrical transparent mechanism, respectively; d1 is the distance from the camera to the flat transparent mechanism, d2 is the thickness of the transparent mechanism, d3 is the distance from the flat transparent mechanism to the cylindrical transparent mechanism, and d4 is the thickness of the cylindrical transparent mechanism; v0 is the unit direction vector of the outgoing light obtained from the camera intrinsic parameters A, pixel coordinates [u, v], and camera focal length f, r = ‖[u, v, f]‖; air refractive index n1, porthole refractive index n2; k is the depth coefficient, [X, Y, Z]' is the physical coordinates of any point in space, [u, v]' is the pixel coordinates of that point mapped to the imaging plane, A is the camera intrinsic parameter matrix, I is the identity matrix, [R, T] is the rotation and translation relationship between the two cameras, and the subscripts 1 and 2 represent the two cameras respectively.

[0034] 4.2) By comparing the spatial coordinates of the same feature point corresponding to the two cameras calculated in step 4.1), the objective function is established: By optimizing the parameters of the refraction model to minimize the objective function, more accurate refraction model parameters can be obtained, achieving complete model calibration. This objective function is actually established by simultaneously utilizing the overlap accuracy of the same feature point calculated by the binocular camera and the deviation between the distance and actual values ​​of adjacent feature points. Optimization can be achieved using a single image, avoiding the limitations of online measurements on aircraft, such as the inability to adjust the calibration plate's attitude and the inability to install a large number of matching targets. Specifically: , , These are the spatial coordinates of each feature point on the calibration plate relative to the two cameras, calculated using the refraction model. , The result was obtained using the refraction model. The spatial coordinates of adjacent feature points, where const is the actual distance between adjacent feature points on the calibration plate; and n is the number of feature points.

[0035] 4.3) Input the initial parameters and optimization range of the model, including the parameters to be optimized and the fixed parameters. The parameters to be optimized include the spatial positioning parameters of the flat transparent mechanism, the cylindrical transparent mechanism, and the calibration plate. The fixed parameters include the refractive index of the porthole, the thickness of the flat transparent mechanism, the inner and outer diameters of the cylindrical transparent mechanism, and the actual distance between adjacent feature points of the calibration plate. The initial values ​​of all spatial positioning parameters to be optimized can be roughly measured using simple measuring tools such as calipers, and the optimization range is set to ±10%.

[0036] 4.4) Substitute the initial parameters into the refraction model and calculate the objective function, i.e., the total error value err. When the total error value err is less than the first threshold (set to 0.1 mm in this embodiment), it indicates that the initial parameters are already optimal. Then, execute step 4.7 and output the parameters; otherwise, continue to step 4.5.

[0037] 4.5) Based on the Sequential Quadratic Programming (SQP) algorithm, after changing the value of the parameter to be optimized, substitute it into the binocular refraction model and calculate the updated total error value. The specific SQP operation is as follows: call the fmincon function in Matlab and set the algorithm parameters to the SQP algorithm. This algorithm has a fast convergence speed and high accuracy.

[0038] 4.6) If the total error value is less than 0.1 mm or the difference Δerr between the total error values ​​before and after the update is less than the first threshold (set to 0.000001 mm in this embodiment), continue to step 4.7); otherwise, return to step 4.5.

[0039] 4.7) Output parameters. At this point, the parameters to be optimized are the optimal solution. The binocular refraction model with the optimal parameters to be optimized is called a fully calibrated binocular refraction model.

[0040] S5. The binocular camera acquires images of the object under test 7, and these images are then substituted into the fully calibrated binocular refraction model: , , Given the calibrated homography matrices of each camera, the pixel coordinates of any feature point on the surface of the object under test in both cameras can be determined using a feature point matching algorithm. )and( Finally, by traversing all i feature points using the least squares method, the spatial coordinates of the overall shape of the object under test can be obtained. , , ).

[0041] Through specific practical experiments, such as Figure 3 As shown, using a 9mm thick flat transparent mechanism with a refractive index of 1.60, and a 9mm thick cylindrical transparent mechanism with an inner diameter of 4680mm and a refractive index of 1.59, with a camera resolution of 3648×5472 pixels, a measurement field of view of approximately 300mm×200mm, and a calibration plate with a calibration area of ​​0.12m×0.09m and a checkerboard pattern with equal spacing of 10mm, the three-dimensional morphology of a calibration plate with a flatness of 0.02mm was measured. Running the above dual-image calibration method with the initial parameters shown in Table 1 yields the optimal values ​​for the parameters to be optimized.

[0042] Table 1

[0043] Substituting the optimal value into the refraction model yields a fully calibrated refraction model. The matching information of all feature points on the surface of the calibration plate is then substituted into this fully calibrated model, ultimately obtaining the point cloud information of the calibration plate's surface morphology. As shown in Table 2, plane fitting was performed using the point cloud obtained by this invention. The flatness of the fitted plane was 0.04 mm. Compared to the standard value of 10 mm, the measured average corner interval had a relative error of 0.6% and an absolute error of 0.06 mm. The measured plane depth was 514.11 mm, compared to the reference depth of 515.11 mm, with a relative error of 0.19% and an absolute error of 1 mm. Compared to measuring the three-dimensional morphology of the calibration plate across the porthole without considering the porthole refraction effect, the depth measurement accuracy is improved by approximately 6 times, demonstrating the effectiveness of this invention.

[0044] Table 2

[0045] Compared with existing technologies, this invention calibrates a binocular refractive model using a single-image calibration method, and then uses the fully calibrated refractive model to accurately measure the three-dimensional shape of objects outside the cabin, thus correcting image point distortion caused by refraction effects during imaging across the porthole. The refractive model is established based on physical laws and can objectively and accurately describe the actual behavior of the refracted light path. Therefore, the accuracy of existing technologies that often use approximations to treat refraction effects as lens radial distortion is far inferior to this method. This invention completes calibration by simultaneously acquiring the same frame of calibration image across the porthole using a binocular camera, requiring only one step. This avoids the impracticality of existing technologies that require disassembling the porthole for measurement and enables real-time calibration in dynamic environments.

[0046] The above-described specific implementations can be partially adjusted by those skilled in the art in different ways without departing from the principles and purpose of the present invention. The scope of protection of the present invention is defined by the claims and is not limited to the above-described specific implementations. All implementation schemes within the scope of the claims are bound by the present invention.

Claims

1. A three-dimensional topography measurement method for imaging across irregularly shaped transparent mechanisms, characterized in that, In the calibration phase, a binocular camera and a calibration plate are respectively set on both sides of the irregular transparent mechanism, and images of the calibration plate are acquired through the irregular transparent mechanism to establish a binocular refraction model containing the characteristics of the irregular transparent mechanism. The calibration is then achieved through a single-image calibration method. In the online phase, the real-time acquired images of the object under test are processed using the calibrated binocular refraction model to obtain the three-dimensional morphology. The aforementioned irregularly shaped transparent mechanism includes a planar transparent mechanism and a cylindrical transparent mechanism.

2. The three-dimensional topography measurement method for imaging across irregularly shaped transparent mechanisms according to claim 1, characterized in that, The aforementioned binocular refraction model refers to: assuming the camera acts as a reverse light source, and establishing a binocular refraction model using the law of refraction. ,in: Let A be the refraction matrix, k be the depth coefficient, [X,Y,Z]' be the physical coordinates of any point in space, [u,v]' be the pixel coordinates of that point mapped to the imaging plane, A be the camera intrinsic parameter matrix, I be the identity matrix, [R,T] be the rotation and translation relationship between the two cameras, and subscripts 1 and 2 represent the two cameras respectively. The aforementioned refraction matrix , where: matrix , rodrigues(m,n) is the Rodrigues rotation formula, which represents a rotation about n as the axis of rotation and m as the angle. , N1 is the surface normal vector of the flat transparent mechanism, and N2 and N3 are the normal vectors corresponding to the intersection points of the imaging light rays when they cross the front and rear surfaces of the cylindrical transparent mechanism, respectively; d1 is the distance from the camera to the flat transparent mechanism, d2 is the thickness of the transparent mechanism, d3 is the distance from the flat transparent mechanism to the cylindrical transparent mechanism, and d4 is the thickness of the cylindrical transparent mechanism; v0 is the unit direction vector of the outgoing light obtained according to the camera intrinsic parameters A, pixel coordinates [u, v], and camera focal length f, r = ‖[u, v, f]‖; air refractive index n1, and porthole refractive index n2.

3. The three-dimensional topography measurement method for imaging across irregularly shaped transparent mechanisms according to claim 1 or 2, characterized in that, The calibration process involves using the Harris corner extraction algorithm to extract the image coordinates of feature points in the image of the calibration plate set outside the porthole, based on the pose relationship between the camera and the calibration plate and the camera's internal parameters, and then substituting this information into the binocular refraction model.

4. The three-dimensional topography measurement method for imaging across irregularly shaped transparent mechanisms according to claim 1 or 2, characterized in that, The calibration specifically includes: Step 1) Based on the pose relationship between the porthole and the single calibration plate relative to the two cameras, obtain the objective function. , where: first error , , These represent the spatial coordinates of each feature point on the calibration plate relative to the two cameras, calculated using a binocular refraction model; the second error... , The result was obtained using a binocular refraction model. The spatial coordinates of adjacent feature points, where const is the actual distance between adjacent feature points on the calibration board; n is the number of feature points; Step 2) Set the initial parameters and their range for the binocular refraction model; Step 3) Substitute the initial parameters into the binocular refraction model and calculate the objective function. If the objective function is less than the first threshold, proceed to step 6; otherwise, continue to step 4. Step 4) Change the values ​​of the parameters to be optimized based on the Sequential Quadratic Programming (SQP) algorithm, substitute them into the binocular refraction model, and calculate the updated objective function; Step 5) If the updated objective function is less than the first threshold or the difference between the objective function before and after the update is greater than the second threshold, proceed to step 6; otherwise, return to step 4. Step 6) Substitute the obtained parameters into the calibration binocular refraction model. ,in: , This is the calibrated homography matrix for each camera.

5. The three-dimensional topography measurement method for imaging across irregularly shaped transparent mechanisms according to claim 4, characterized in that, The initial parameters include: parameters to be optimized and fixed parameters, wherein: the parameters to be optimized include the spatial positioning parameters of the flat transparent mechanism, the spatial positioning parameters of the cylindrical transparent mechanism, and the spatial positioning parameters of the calibration plate; the fixed parameters include the refractive index of the porthole, the thickness of the flat transparent mechanism, the inner and outer diameters of the cylindrical transparent mechanism, and the actual distance between adjacent feature points of the calibration plate. All the spatial positioning parameters to be optimized can be roughly measured using simple measuring tools such as calipers, and the optimization range is set to ±10%.

6. The three-dimensional topography measurement method for imaging across irregularly shaped transparent mechanisms according to claim 1, characterized in that, The aforementioned three-dimensional topography is obtained as follows: based on a binocular refraction model, the pixel coordinates of any feature point on the surface of the object under test in both cameras are determined using a feature point matching algorithm. )and( By traversing all feature points using the least squares method, the spatial coordinates of the entire field shape of the object under test can be obtained. , , ).

Images