Lens distortion correction method and system based on least square method and neural network

Abstract

The invention discloses a lens distortion correction method and system based on a least square method and a neural network, and relates to the technical field of three-dimensional scanning, and the method comprises the steps: calibrating a system, obtaining an internal parameter matrix and an external parameter matrix of a camera and a projector, and determining a corresponding relation between a world three-dimensional coordinate and a camera pixel coordinate; constructing a neural network; reconstructing the surface of a flat plate sufficient to cover the view field of the camera and the projector, and after expansion and fitting, training a neural network by using the point clouds P'of different heights after linear fitting and the point clouds Q 'after plane fitting based on the corresponding relation to obtain a trained neural network; and taking actual three-dimensional world coordinates as input, and predicting ideal three-dimensional world coordinates by using the trained neural network. According to the invention, based on the least square method and the neural network, the mapping relation between the actual planes with different heights and the fitting plane is obtained, and high-precision measurement and correction of lens distortion are realized.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional scanning technology, and in particular to a lens distortion correction method and system based on least square method and neural network. Background Art

[0002] At present, with the continuous advancement of computer vision and sensor technology and the research of many scholars, the accuracy of structured light has been improved: the structured light system combined with industrial lenses and image processing technology has become more powerful and flexible, so it has become one of the hot fields in 3D reconstruction. Among them, fringe projection profilometry based on phase measurement profilometry has been widely used due to its high accuracy. Fringe projection profilometry usually consists of an industrial camera and a digital projector. The projector generates a fringe pattern and illuminates the surface of the object. The computer obtains the pattern captured by the camera on the surface of the object and decodes it to calculate the depth information.

[0003] The accuracy of fringe projection profilometry is affected by factors such as intensity noise, nonlinear error, and lens distortion. Among them, factors such as intensity noise and nonlinear error can be reduced by using the large-step phase shift method. Lens distortion is difficult to eliminate due to its essential characteristics, but there are still many methods proposed to address this problem, and predistortion is one of them. For example, Chen WX et al. mentioned in Research on projector distortion correction of PMP system using predistortion fringe [J]. Acta Photonica Sinica, 2009, 38 (10): 2698-2701 that this method applies reverse distortion to the ideal phase according to the calibrated projector lens distortion parameters, so that it becomes an ideal grating again after lens distortion during the projection process. Based on this method, a variety of methods are derived, such as adding epipolar constraints, compensating for distortion errors, etc. to improve compensation accuracy, but this method is not suitable for binary fringe patterns, which will cause such fringe patterns to produce sawtooth phenomena at the binary boundary, affecting the reconstruction accuracy.

[0004] Based on the fact that rational functions can describe the carrier phase distribution of any measurement system, many methods have been proposed. For example, Peng et al. proposed an adaptive fringe projection technology in Suppression of projector distortion in phase-measuring profilometry by projecting adaptive fringe patterns [J]. Optics Express, 2016, 24(19): 21846-21860. This technology uses Zernike polynomial fitting to obtain the curved carrier phase, and uses this to correct the pixel points obtained on the projection plane. Finally, an adaptive fringe pattern is constructed based on the corrected pixel points to eliminate the carrier phase bending caused by the projector distortion. Xing et al. noticed in Iterative calibration method for measurement system having lens distortions in fringe projection profilometry [J]. Optics Express, 2020, 28(2): 1177-1196 that the distortion of the projection lens may deform the fitted phase map, thereby causing errors in the projection parameter estimation. Therefore, an iterative strategy is proposed to overcome this problem: by alternating phase fitting and parameter estimation, the internal and external parameters of the projector and the lens distortion coefficient can be accurately determined.

[0005] Of course, mathematical methods are not limited to phase fitting. For example, the least squares fitting can be used to accurately solve the distortion parameters of the lens, but this method does not take into account the lens distortion of the projector. Considering this problem, Zhang et al. established a scale offset model to characterize the distortion of the projector in Correcting projector lens distortion in real time with a scale-offset model for structured light illumination [J]. Optics Express, 2022, 30 (14): 24507-24522, thereby speeding up the distortion correction through a lookup table. Other studies have combined camera lens distortion and projector lens distortion for correction, integrating rational functions into the mathematical expression of the height from the surface to reduce the impact of lens distortion.

[0006] As we all know, a trained BP neural network can approximate any function, so neural networks have a lot of potential in the field of optical imaging. Many scholars use various networks to improve the accuracy of structured light 3D measurement. Lv et al. proposed a DNN that can find the mapping relationship between the 3D coordinates of an object and its corresponding distortion error in Projector distortion correction in 3D shape measurement using a structured-light system by deep neural networks[J].Optics Letters, 2020, 45(1): 204-207, which can compensate for the measurement error caused by projector distortion. Son et al. proposed a fully automatic calibration method for multi-projector-camera system in Multiple projector camera calibration by fiducial marker detection[J].IEEE Access, 2023. This method estimates the correspondence between the projector and the camera based on AprilTag tags and YOLOv8 with deformable convolution, and can automatically complete the calibration and distortion calibration process.

[0007] Therefore, there is an urgent need for a solution for lens distortion correction based on the corresponding relationship between the projector and the camera. Summary of the invention

[0008] The purpose of the present invention is to provide a lens distortion correction method and system based on the least squares method and neural network, obtain the mapping relationship between planes at different heights and the fitting plane based on the least squares method and the neural network, correct the finally generated point cloud, and realize high-precision measurement and correction of lens distortion.

[0009] To achieve the above object, the present invention provides the following solutions:

[0010] A lens distortion correction method based on least squares method and neural network includes the following steps:

[0011] S1, calibration system: the measurement system where the camera and projector are located, obtains the internal parameter matrix and external parameter matrix of the camera and projector, and determines the correspondence between the world 3D coordinates and the camera pixel coordinates;

[0012] S2, build a neural network;

[0013] S3, using the large-step phase shift method to reconstruct a flat surface that is sufficient to cover the field of view of the camera and the projector, obtain the actual plane point cloud and expand it;

[0014] S4. Obtain a plurality of undistorted point clouds P at different heights and a point cloud Q dedistorted by OpenCV according to the expanded actual plane point cloud. Based on the corresponding relationship determined in S1, linear fitting is performed on P and Q respectively using the least squares method to obtain point clouds P' and Q' at different heights after linear fitting. Then, plane fitting is performed on Q' to obtain Q".

[0015] S5, using the point cloud P' as an input data set of the neural network and Q" as a label data set of the neural network to train the neural network to obtain a trained neural network;

[0016] S6. Taking the actual three-dimensional world coordinates as input, using the trained neural network to predict the ideal three-dimensional world coordinates.

[0017] Furthermore, in S3, the method for expanding the actual plane point cloud is: expanding a sufficient length on the Z axis with a fixed step length, and the fixed step length is determined by averaging the distance between adjacent points.

[0018] Further, the S4, according to the expanded actual plane point cloud, obtains a plurality of undistorted point clouds P at different heights and a point cloud Q dedistorted by OpenCV, and based on the corresponding relationship determined in S1, linearly fits P and Q respectively by using the least squares method to obtain point clouds P' and Q' at different heights after linear fitting, and then plane fits Q' to obtain Q", specifically including:

[0019] S401, take the expanded actual plane point cloud P at multiple heights, perform least squares linear fitting on the three-dimensional coordinates corresponding to the same pixel point at each different height, and obtain the point cloud P' of each height plane. The coordinates of P' are As input dataset;

[0020] S402. A large number of expanded actual plane point clouds of different heights are obtained and dedistorted by using OpenCV to obtain point cloud Q. The coordinates of point cloud Q' are obtained by using the least squares linear fitting method described in S401. Then, plane fitting is performed on each plane point cloud Q' to obtain the target point cloud coordinates Q' as a label data set.

[0021] Further, the S401, taking the expanded actual plane point cloud P at multiple heights, respectively performs least squares linear fitting on the three-dimensional coordinates corresponding to the same pixel point at each different height, to obtain the point cloud P' of each height plane, the coordinates of P' are As input data sets, specifically including:

[0022] For the value on the X-axis: get the x mapped to the same pixel at different heights according to the pixel coordinate index, and then use the least squares linear fit to get the new value:

[0023]

[0024] Finally, get the x value of the same pixel at different heights.

[0025]

[0026] In the formula, h n is the distance between n different planes and the camera, is the x mapped by the same pixel coordinate at different distances, and i=1,2,..,u,j=1,2,...,v, and is the linear fitting relationship coefficient of the corresponding pixel point on the x-axis;

[0027] Then use the same method to get the value on the Y axis as the value on the X axis and the value on the Z axis Finally, the point cloud coordinates of each height plane are obtained as input to the neural network.

[0028] Furthermore, in S2, the neural network includes an input layer, a hidden layer, and an output layer, wherein the sizes of the input layer and the output layer are both 3, the size of the hidden layer is 15, and the activation function adopts Sigmoid.

[0029] Furthermore, the training of the neural network specifically includes: using MSE as the loss function of the neural network, and the algorithm adopted for training the neural network is a Bayesian regularization algorithm.

[0030] The present invention also provides a lens distortion correction system based on the least squares method and a neural network, which is used to execute the lens distortion correction method based on the least squares method and a neural network, comprising:

[0031] The projection matrix acquisition module is used to calibrate the system: the measurement system where the camera and projector are located, obtain the internal parameter matrix and external parameter matrix of the camera and projector, and determine the correspondence between the world 3D coordinates and the camera pixel coordinates;

[0032] Neural network building blocks, used to build neural networks;

[0033] An expansion module is used to reconstruct a flat surface sufficient to cover the field of view of the camera and the projector using a large-step phase shift method to obtain an actual plane point cloud and expand it;

[0034] The fitting module is used to obtain the undistorted point clouds P and the point clouds Q dedistorted by OpenCV at multiple different heights according to the expanded actual plane point cloud. Based on the corresponding relationship determined by S1, the least squares method is used to perform linear fitting on P and Q respectively to obtain the point clouds P' and Q' at different heights after linear fitting, and then plane fitting is performed on Q' to obtain Q".

[0035] A training module is used to train the neural network using the point cloud P' as an input data set of the neural network and Q" as a label data set of the neural network to obtain a trained neural network;

[0036] The prediction module is used to use the actual three-dimensional world coordinates as input and use the trained neural network to predict the ideal three-dimensional world coordinates.

[0037] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects: the lens distortion correction method and system based on the least squares method and neural network provided by the present invention analyze the influence of lens distortion on the final imaging, and propose a method based on linear fitting and neural network. In this method, a flat surface is first reconstructed under the full field of view of the camera, so that each camera pixel can find the corresponding three-dimensional point, and the undistorted point cloud and the point cloud dedistorted by OpenCV at multiple heights are obtained. The point clouds at each height are linearly fitted to obtain point clouds $P'$ and $Q'$ at different heights, and then $Q'$ is plane fitted to obtain $Q"$. Finally, a neural network is used to map $P'$ to $Q"$ to achieve the purpose of correcting the distortion error.

[0038] This method obtains plane data with height differences through least squares linear fitting, and uses least squares plane fitting to obtain planes of corresponding heights. Thus, a mapping relationship between planes of different heights and the fitting planes under the neural network model is established, and the point cloud generated by the camera and projector can be directly corrected. After experimental verification, the accuracy of the peak-to-valley value of the average point spacing is improved by 97.24%, and the accuracy of the root mean square value of the average point spacing is improved by 96.66%. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0040] Figure 1 This is a flow chart of a lens distortion correction method based on least squares method and neural network according to an embodiment of the present invention;

[0041] Figure 2 This is the fitting process of the data of each height plane point cloud on the X-axis in the embodiment of the present invention;

[0042] Figure 3 is a schematic diagram of a plane point cloud affected by an embodiment of the present invention;

[0043] Figure 4 The least squares linear fit of a plate point cloud at a certain height to the value on the X-axis in an embodiment of the present invention;

[0044] Figure 5 Schematic diagram of the least squares plane fitting process of a certain height plane point cloud in an embodiment of the present invention;

[0045] Figure 6 Schematic diagram of the fitting effect in the embodiment of the present invention, wherein (a) represents the schematic diagram before plane fitting, and (b) represents the schematic diagram after plane fitting;

[0046] Figure 7 A schematic diagram of a neural network in an embodiment of the present invention;

[0047] Figure 8 Schematic diagram of average loss change in an embodiment of the present invention, where A represents training loss and B represents test loss;

[0048] Fig. 9 : The plane reconstruction effect diagram in the embodiment of the present invention, wherein (a), (c), and (e) are respectively the point cloud diagram without distortion correction, the point cloud diagram after distortion correction by OpenCV, and the dedistorted point cloud diagram predicted by the neural network, and (b), (d), and (f) are the point spacing distribution diagrams of the above three point clouds and the ideal point cloud;

[0049] Fig.10 Graphs showing the test results of a stacked flat plate experiment in an embodiment of the present invention, wherein, in Figure (a), the left figure is a schematic diagram of the original point cloud, and the right figure is an enlarged view; in Figure (b), the left figure is a schematic diagram of an OpenCV dedistorted point cloud, and the right figure is an enlarged view; in Figure (c), the left figure is a schematic diagram of a neural network dedistorted point cloud, and the right figure is an enlarged view. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] The purpose of the present invention is to provide a lens distortion correction method and system based on the least squares method and neural network, obtain the mapping relationship between planes at different heights and the fitting plane based on the least squares method and the neural network, correct the finally generated point cloud, and realize high-precision measurement and correction of lens distortion.

[0052] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0053] like Figure 1 As shown, a lens distortion correction method based on least squares method and neural network provided by an embodiment of the present invention includes the following steps:

[0054] S1. Calibration system: calibrate the measurement system where the camera and projector are located, obtain the internal parameter matrix and external parameter matrix of the camera and projector, and determine the correspondence between the world 3D coordinates and the camera pixel coordinates;

[0055] S2, build a neural network;

[0056] S3, using the large-step phase shift method to reconstruct a flat surface that is sufficient to cover the field of view of the camera and the projector, obtain the actual plane point cloud and expand it;

[0057] S4. Obtain a plurality of undistorted point clouds P at different heights and a point cloud Q dedistorted by OpenCV according to the expanded actual plane point cloud. Based on the corresponding relationship determined in S1, linear fitting is performed on P and Q respectively using the least squares method to obtain point clouds P' and Q' at different heights after linear fitting. Then, plane fitting is performed on Q' to obtain Q".

[0058] S5, using the point cloud P' as an input data set of the neural network and Q" as a label data set of the neural network to train the neural network to obtain a trained neural network;

[0059] S6. Taking the actual three-dimensional world coordinates as input, using the trained neural network to predict the ideal three-dimensional world coordinates.

[0060] In this embodiment, in S3, the method for expanding the actual plane point cloud is: expanding a sufficient length on the Z axis with a fixed step length, and the fixed step length is determined by averaging the distance between adjacent points.

[0061] In this embodiment, the step S4, obtaining a plurality of undistorted point clouds P and a point cloud Q dedistorted by OpenCV at different heights according to the expanded actual plane point cloud, and performing linear fitting on P and Q respectively by using the least squares method based on the corresponding relationship determined by S1, to obtain point clouds P' and Q' at different heights after linear fitting, and then performing plane fitting on Q' to obtain Q", specifically includes:

[0062] S401, take the expanded actual plane point cloud P at multiple heights, perform least squares linear fitting on the three-dimensional coordinates corresponding to the same pixel point at each different height, and obtain the point cloud P' of each height plane. The coordinates of P' are As input dataset;

[0063] S402. A large number of expanded actual plane point clouds of different heights are obtained and dedistorted by using OpenCV to obtain point cloud Q. The coordinates of point cloud Q' are obtained by using the least squares linear fitting method described in S401. Then, plane fitting is performed on each plane point cloud Q' to obtain the target point cloud coordinates Q' as a label data set.

[0064] In this embodiment, the S401, taking the expanded actual plane point cloud at multiple heights, respectively performs least squares linear fitting on the three-dimensional coordinates corresponding to the same pixel point at each different height, to obtain the point cloud coordinates of each height plane As input data sets, specifically including:

[0065] For the value on the X-axis: get the x mapped to the same pixel at different heights according to the pixel coordinate index, and then use the least squares linear fit to get the new value:

[0066]

[0067] Finally, get the x value of the same pixel at different heights.

[0068]

[0069] In the formula, h n is the distance between n different planes and the camera, is the x mapped by the same pixel coordinate at different distances, and i=1,2,..,u,j=1,2,...,v, and is the linear fitting coefficient of the corresponding pixel point on x,

[0070] Then use the same method to get the value on the Y axis as the value on the X axis and the value on the Z axis Finally, the point cloud coordinates of each height plane are obtained as input to the neural network.

[0071] In this embodiment, the neural network includes an input layer, a hidden layer, and an output layer, wherein the size of the input layer and the output layer are both 3, the size of the hidden layer is 15, and the activation function adopts Sigmoid.

[0072] In this embodiment, the training of the neural network specifically includes: using MSE as the loss function of the neural network, and the algorithm used for training the neural network is a Bayesian regularization algorithm.

[0073] Example 2

[0074] Based on the method described in Example 1, this example provides a further implementation method.

[0075] 2.1. Effect of lens distortion

[0076] Before starting the projection reconstruction, the camera and projector must be calibrated. The purpose is to determine the intrinsic and extrinsic parameters and establish the relationship between the absolute phase and the 3D point cloud. Here we briefly review the calibration method of Cui

[16] : the camera is used as a pinhole model, and the world coordinates (X w ,Y w ,Z w ) and the camera pixel coordinates (u c ,v c ) have a corresponding relationship:

[0077]

[0078] Among them, s c is the proportional coefficient, the internal parameter matrix and f is the focal length of the camera, dx and dy are the physical sizes of the pixel in the horizontal and vertical directions of the image, (u0, v0) are the coordinates of the origin in the camera coordinate system, and the external parameter matrix is Where R c is a 3×3 rotation matrix, T c is a 3×1 translation matrix.

[0079] Since the optical path of the projector is opposite to that of the camera, the projector can be regarded as an inverse camera, and its model is similar to the camera model (this principle comes from the study of Calibration of fringe projection profilometry: A comparative review [J]. Optics and lasers in engineering, 2021, 143: 106622 proposed by Feng S et al.):

[0080]

[0081] According to equations (1) and (2), the projection matrix A of the camera and projector can be obtained: c and A p :

[0082]

[0083] Combining equations (1), (2), (3) and (4), we can obtain the conversion formula

[18] :

[0084]

[0085] It should be noted that equations (1) and (2) are obtained under the condition that the model is completely ideal. In practice, due to reasons such as lens design and manufacturing, the shape and position of objects in the image will be distorted, that is, the shape and size of objects in the image do not match the actual scene, which affects the processing and analysis of the image. Therefore, distortion correction is often required. Generally speaking, image distortion is often divided into radial distortion and tangential distortion. The distortion model can be expressed by the following equation:

[0086]

[0087] Where k1, k2 and k3 are radial distortion coefficients, p1 and p2 are tangential distortion coefficients. Currently, these distortion coefficients can be obtained through OpenCV, MATLAB and other software. It should be noted that the more distortion coefficients involved in modeling, the better. Too many coefficients may cause the distortion coefficients obtained through the program to become unstable. is the distortion coordinate, (x c ,y c ) are ideal coordinates, and:

[0088]

[0089] The distorted pixel coordinates can be obtained by the following formula:

[0090]

[0091] Unlike cameras that can use formula (6) to directly calibrate distorted images, projectors that cannot capture images cannot know the corresponding relationship between their pixel points and object points. Most of the current solutions are to pre-distort the stripe pattern similar to formula (6) in advance, and then project it out to achieve the purpose of distortion correction.

[0092] According to formula (5), both the camera pixel coordinates and the projector pixel coordinates are involved in the calculation of the three-dimensional coordinates. When the image is distorted, according to formula (8), the pixel points are affected by the distortion, resulting in the distortion of the three-dimensional points.

[0093] 2.2 Compensation method

[0094] OpenCV provides a de-distortion technology, so it can be used to correct lens distortion. However, experiments show that it cannot completely correct the distortion, and the correction performance still needs to be further improved. The specific analysis is later.

[0095] It can be known that in structured light 3D reconstruction, without considering lens distortion, when a plane is reconstructed, a point cloud with sufficiently low flatness should be obtained. Based on this, in this embodiment, the least squares method and neural network are used to learn the error between the actual plane and the ideal plane, so as to correct the overall error pixel by pixel. The specific steps are as follows.

[0096] Step 1: Calibrate the system and obtain the intrinsic parameter matrix and extrinsic parameter matrix of the camera and projector.

[0097] Step 2: Construct a neural network and design it into one input layer, one hidden layer, and one output layer. The size of the input layer and the output layer is 3, the size of the hidden layer is 15, and the activation function is Sigmoid.

[0098] Step 3: Create a data set. As described in 2.1, each camera pixel has a different degree of distortion. Therefore, in order to ensure that the distortion of the entire field of view can be corrected and to avoid nonlinear errors, a large-step phase shift method is used to reconstruct a flat surface that is sufficient to cover the field of view of the camera and projector to obtain the actual point cloud. However, due to insufficient plane height information, only using the plane point cloud data obtained at a fixed distance to train the network will result in the network being unable to learn information at different heights. Therefore, the obtained data set needs to be expanded: expand the Z axis with a fixed step length to a sufficient length, and the fixed step length is determined by the average distance between adjacent points. In practice, it is difficult to accurately obtain plane data at each different height. Considering that the optical path in the camera field of view is a straight line, it can be seen that at different heights, the three-dimensional coordinates corresponding to the same camera pixel have a linear relationship.

[0099] According to the above, take the plane point cloud data reconstructed at n different heights, and perform least squares linear fitting on the three-dimensional coordinates corresponding to the same pixel point at each different height. Taking the value on the X-axis as an example, the specific fitting process is as follows: Figure 2 shown.

[0100] like Figure 2As shown, first obtain the x mapped to the same pixel at different heights according to the pixel coordinate index, and finally use the least squares linear fitting to obtain the new value:

[0101]

[0102] In formula (9), h n is the distance between n different planes and the camera, is the x mapped by the same pixel coordinate at different distances, and i=1,2,..,u,j=1,2,...,v, and is the linear fitting coefficient of the corresponding pixel point on x, and finally the x of the same pixel point at different heights is obtained through formula (10): as well as The point cloud coordinates of each height plane are obtained by a similar method. as input to the neural network.

[0103] The input data set of the neural network is completed as mentioned above. Next, the label data set of the neural network is produced. Because a large number of planar point clouds of different heights are required, its preprocessing is consistent with the input data set. In order to ensure the quality of the data, the point cloud obtained by OpenCV dedistortion technology is used as the source point cloud of the label.

[0104] After obtaining a plane point cloud set with sufficient height, plane fitting is performed on each plane point cloud. Since point cloud data is easily affected by measurement system errors, noise or sampling methods, the arrangement of the obtained point cloud data in space becomes disordered, such as Figure 3 ,Due to the influence of errors, the edge point cloud appears wavy, which will affect the point cloud data obtained by fitting ,the actual planes at different heights.

[0105] Therefore, before regressing the plane point cloud to the same plane, it is necessary to first regress these wave point clouds. Taking the value on the X-axis as an example, the specific fitting process is as follows: Figure 3 shown.

[0106] according to Figure 4 , we still use the linear fitting method similar to formula (9) and formula (10) to process the point cloud arrangement in space. The value on the Y axis is the same. Finally, according to Fitting with the least squares plane to obtain The specific fitting process is as follows Figure 5 As shown, the final As labels for neural networks.

[0107] The final fitting effect is as follows Figure 6 As shown, Figure 6(a) is a schematic diagram of a plane point cloud fitted at a certain height. Figure 6 (b) is a schematic diagram of the plane point cloud after plane fitting. It can be seen that after linear fitting, the point cloud is no longer arranged in a wavy shape.

[0108] Step 4: Train the network. Randomly extract enough data from each plane with different heights to supplement the data set. Since the height difference is small, if a data set containing all heights is used for training, it is easy to cause insufficient network accuracy. In order to ensure accuracy and reduce the model size, planes with large height differences are spliced ​​into one data set for training. When training the network, randomly extract 70% of the data set as the training set and 30% as the test set. The final model architecture is as follows: Figure 7 As shown, it should be noted that the input point cloud is fitted from the actual point cloud obtained without Opencv dedistortion technology, and the target point cloud is fitted from the actual point cloud obtained by Opencv dedistortion technology, because this will make it closer to the real ideal point cloud.

[0109] When training the model, MSE is used as the loss function of the network, the round number is 1000, and the Bayesian regularization algorithm is selected as the training algorithm. This algorithm is different from the traditional point estimation algorithm. Because the posterior distribution of weights and biases in the neural network is calculated, it can provide information about the uncertainty of the model parameters. In areas with more data, the training data can narrow the posterior distribution of the parameters, while in areas with less data, the posterior distribution of the parameters will be wider. Therefore, the algorithm can accurately fit in areas with more data, and retain greater uncertainty in areas with less data (in point cloud datasets, areas with less data are usually data noise), thereby improving the robustness and generalization ability of the network

[20] . This is applicable to overly large point cloud datasets.

[0110] Step 5: Predict data and enter the actual 3D world coordinates (X w ,Y w ,Z w ), predicting the ideal 3D world coordinates through the trained network

[0111] In a further embodiment, a three-dimensional structured light measurement system is used, which includes an industrial camera (Hikvision MV-CS050-10GM-PRO) with a resolution of 2448×1848 and two digital projectors (DLP4500) with a resolution of 912×1140. It should be noted that in this experiment, the resolution of the camera was adjusted to 2000×1700 and only one side of the projector was used to verify the method. The specific process is as follows:

[0112] Calibrate the system according to step 1 and get the projection matrix of the camera and projector:

[0113]

[0114] After building the neural network according to step 2 and preparing the data set according to step 3, the neural network was trained. In this experiment, five planes of different heights were used to fit the point clouds of different heights. The average loss changes of multiple networks trained are shown in the figure below: Figure 8 As shown, the optimal average training loss is 6.85307×10 -5 , the optimal average test loss is 6.85041×10 -5 ,It is not difficult to see that the two curves show a parallel downward trend, indicating that the network training is good.

[0115] According to step (5), a plane is taken for verification and the point spacing of the point cloud is measured to measure the actual performance of the network. The final result is as follows: Fig. 9 As shown in the figure, (a), (c), and (e) are the point cloud images without distortion correction, the point cloud images after OpenCV distortion correction, and the dedistorted point cloud images predicted by the neural network, respectively. Fig. 9 (b), (d), and (f) are the point spacing distribution diagrams of the above three point clouds and the ideal point cloud, respectively.

[0116] according to Fig. 9 Compared with Table 1, the original point cloud has a large degree of distortion, so the point spacing error with the ideal point cloud is large, and its peak-to-valley value (PV) is 13.9119 mm, and the root mean square error (RMSE) is 4.0606 mm. After using the de-distortion technology of OpenCV, although the PV and RMSE are reduced by 4.9445 mm and 0.4170 mm respectively, they are not improved much. After de-distortion, the PV and RMSE are reduced by 13.4740 mm and 3.9707 mm respectively, which reflects the superiority of the present invention.

[0117] Table 1 Planar reconstruction experimental results (unit: mm)

[0118]

[0119] It is not difficult to see from the experimental results that the network model accurately predicts the three-dimensional points of the ideal plane. In order to further verify the performance of the network, the flat plate was measured and predicted at different heights, and the results are shown in Table 2. It can be seen that with the increase of height, the degree of distortion of the actual point cloud is gradually reduced. Before dedistortion, the average point spacing PV and RMSE of the actual point cloud and the ideal point cloud were 13.4275mm and 3.9209mm respectively. After using the neural network for dedistortion, the average point spacing PV and RMSE dropped to 0.3714mm and 0.1304mm respectively, and the performance was improved by 97.24% and 96.66% respectively.

[0120] Table 2 Experimental results of plane reconstruction at different heights (unit: mm)

[0121]

[0122]

[0123] According to the statistics in Table 2, the network has satisfactory performance. In order to verify the correction effect of the plane at different heights, three standard flat plates are overlapped and placed in a step-like shape in the scanner for 3D reconstruction, as shown in Fig.10 As shown, Fig.10 (a) is the original point cloud and its enlarged image. Fig.10 (b) is the point cloud dedistorted using OpenCV and its method diagram. Fig.10 (c) is the point cloud dedistorted by the neural network and its enlarged image. It is not difficult to see that the point cloud after OpenCV dedistortion is smoother than the original point cloud, but it is obvious from its enlarged image that there is still bending on the edge of the step. After dedistortion by our method, we can get a point cloud that is closer to the ideal one.

[0124] It should be noted that although the designed network effectively alleviates the impact of lens distortion, there are still problems that need to be solved or improved in the proposed method. For example, since the ideal plane and plane data at different heights are all fitted, they themselves carry certain residuals. Therefore, if you want to improve the accuracy of the model, finding more accurate ideal planes and planes at different heights is a problem that needs to be faced.

[0125] Example 3

[0126] This embodiment provides a lens distortion correction system based on the least squares method and a neural network, which is used to execute the lens distortion correction method based on the least squares method and a neural network described in Embodiment 1 or 2, including:

[0127] The projection matrix acquisition module is used to calibrate the measurement system where the camera and projector are located, obtain the internal parameter matrix and external parameter matrix of the camera and projector, and obtain the projection matrix corresponding to the camera and projector according to the internal parameter matrix and the external parameter matrix;

[0128] Neural network building blocks, used to build neural networks;

[0129] Expansion module: used to reconstruct a flat surface that is large enough to cover the field of view of the camera and projector using the large-step phase shift method, obtain the actual plane point cloud and expand it;

[0130] The fitting module is used to obtain the undistorted point clouds P and the point clouds Q dedistorted by OpenCV at multiple different heights according to the expanded actual plane point cloud, and use the least squares method to perform linear fitting on P and Q respectively to obtain the point clouds $P'$ and $Q'$ at different heights after linear fitting, and then perform plane fitting on $Q'$ to obtain $Q"$;

[0131] A training module is used to train the neural network using the point cloud $P'$ as an input data set of the neural network and $Q'$ as a label data set of the neural network to obtain a trained neural network;

[0132] The prediction module is used to use the actual three-dimensional world coordinates as input and use the trained neural network to predict the ideal three-dimensional world coordinates.

[0133] In summary, the present invention provides a distortion correction method based on linear fitting and neural network. The network is used to establish a mapping relationship between actual three-dimensional world coordinates and ideal three-dimensional points. The actual plane data is collected by the measurement system, and the plane data at different heights are approximated by linear fitting of multiple actual plane data at different heights. The ideal plane data is obtained by analyzing and fitting the actual plane data after the OpenCV dedistortion technology. Since the plane data is obtained in the full field of view of the camera, the distortion correction described in this method is actually performed pixel by pixel. Experimental results show that after applying the network for distortion correction, the measurement accuracy reaches 0.1304mm in the root mean square value of the average point spacing, which is an increase of 96.66%.

[0134] The remaining technical features in this embodiment can be flexibly selected by those skilled in the art according to actual conditions to meet different specific practical needs. However, it is obvious to those skilled in the art that it is not necessary to adopt these specific details to implement the present invention. In other examples, in order to avoid confusing the present invention, the well-known components, structures or parts are not specifically described, which are all within the technical protection scope defined by the technical solution claimed for protection in the claims of the present invention.

[0135] Modifications and changes made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the scope of protection of the claims attached to the present invention. In the above description, a large number of specific details are set forth in order to provide a thorough understanding of the present invention. However, it is obvious to those of ordinary skill in the art that these specific details are not necessary to practice the present invention. In other examples, in order to avoid confusing the present invention, well-known technologies, such as specific construction details, operating conditions and other technical conditions, are not specifically described.

[0136] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A lens distortion correction method based on least squares method and neural network, characterized in that: The following steps are involved: S1, calibration system: the measurement system where the camera and projector are located, obtains the internal parameter matrix and external parameter matrix of the camera and projector, and determines the correspondence between the world 3D coordinates and the camera pixel coordinates; S2, build a neural network; S3, using the large-step phase shift method to reconstruct a flat surface that is sufficient to cover the field of view of the camera and the projector, obtain the actual plane point cloud and expand it; S4. Obtain a plurality of undistorted point clouds P at different heights and a point cloud Q dedistorted by OpenCV according to the expanded actual plane point cloud. Based on the corresponding relationship determined in S1, linear fitting is performed on P and Q respectively using the least squares method to obtain point clouds P' and Q' at different heights after linear fitting. Then, plane fitting is performed on Q' to obtain Q". S5, using the point cloud P' as an input data set of the neural network and Q" as a label data set of the neural network to train the neural network to obtain a trained neural network; S6. Taking the actual three-dimensional world coordinates as input, using the trained neural network to predict the ideal three-dimensional world coordinates.

2. The lens distortion correction method based on least squares method and neural network according to claim 1, characterized in that: In S3, the method for expanding the actual plane point cloud is: expanding a sufficient length on the Z axis with a fixed step length, and the fixed step length is determined by averaging the distance between adjacent points.

3. The lens distortion correction method based on least squares method and neural network according to claim 1, characterized in that: S4, according to the expanded actual plane point cloud, obtains a plurality of undistorted point clouds P at different heights and a point cloud Q dedistorted by OpenCV, and based on the corresponding relationship determined in S1, linearly fits P and Q respectively by using the least squares method to obtain point clouds P' and Q' at different heights after linear fitting, and then plane fits Q' to obtain Q", specifically including: S401, take the expanded actual plane point cloud P at multiple heights, perform least squares linear fitting on the three-dimensional coordinates corresponding to the same pixel point at each different height, and obtain the point cloud P' of each height plane. The coordinates of P' are As input dataset; S402. A large number of expanded actual plane point clouds of different heights are obtained and dedistorted by using OpenCV to obtain point cloud Q. The coordinates of point cloud Q' are obtained by using the least squares linear fitting method described in S401. Then, plane fitting is performed on each plane point cloud Q' to obtain the target point cloud coordinates Q' as a label data set.

4. The lens distortion correction method based on least squares method and neural network according to claim 3, characterized in that: S401, taking the expanded actual plane point cloud P at multiple heights, respectively performing least square linear fitting on the three-dimensional coordinates corresponding to the same pixel point at each different height, to obtain the point cloud P' of each height plane, the coordinates of P' are As input data sets, specifically including: For the value on the X-axis: get the x mapped to the same pixel at different heights according to the pixel coordinate index, and then use the least squares linear fit to get the new value: Finally, get the x value of the same pixel at different heights. In the formula, h n is the distance between n different planes and the camera, is the x mapped by the same pixel coordinate at different distances, and i=1,2,..,u,j=1,2,...,v, and is the linear fitting relationship coefficient of the corresponding pixel point on the x-axis; Then use the same method to get the value on the Y axis as the value on the X axis and the value on the Z axis Finally, the point cloud coordinates of each height plane are obtained as input to the neural network.

5. The lens distortion correction method based on least squares method and neural network according to claim 1, characterized in that: In S2, the neural network includes an input layer, a hidden layer, and an output layer, wherein the sizes of the input layer and the output layer are both 3, the size of the hidden layer is 15, and the activation function uses Sigmoid.

6. The lens distortion correction method based on least squares method and neural network according to claim 1, characterized in that: The training of the neural network specifically includes: using MSE as the loss function of the neural network, and the algorithm adopted for training the neural network is a Bayesian regularization algorithm.

7. A lens distortion correction system based on least squares method and neural network, used to execute the lens distortion correction method based on least squares method and neural network according to any one of claims 1 to 6, characterized in that: include: The projection matrix acquisition module is used to calibrate the system: the measurement system where the camera and projector are located, obtain the internal parameter matrix and external parameter matrix of the camera and projector, and determine the correspondence between the world 3D coordinates and the camera pixel coordinates; Neural network building blocks, used to build neural networks; An expansion module is used to reconstruct a flat surface sufficient to cover the field of view of the camera and the projector using a large-step phase shift method to obtain an actual plane point cloud and expand it; The fitting module is used to obtain the undistorted point clouds P and the point clouds Q dedistorted by OpenCV at multiple different heights according to the expanded actual plane point cloud. Based on the corresponding relationship determined by S1, the least squares method is used to perform linear fitting on P and Q respectively to obtain the point clouds P' and Q' at different heights after linear fitting, and then plane fitting is performed on Q' to obtain Q". A training module is used to train the neural network using the point cloud P' as an input data set of the neural network and Q" as a label data set of the neural network to obtain a trained neural network; The prediction module is used to use the actual three-dimensional world coordinates as input and use the trained neural network to predict the ideal three-dimensional world coordinates.

Images