Lens Distortion Correction Method and System Based on Least Squares Method and Neural Network

Abstract

The present invention discloses a lens distortion correction method and system based on the least squares method and neural network, which relates to the field of three-dimensional scanning technology. The method includes: calibrating the system to obtain the internal parameter matrix and external parameter matrix of the camera and the projector, and determining the correspondence between the three-dimensional world coordinates and the camera pixel coordinates; constructing a neural network; reconstructing the surface of a flat plate sufficient to cover the fields of view of the camera and the projector, and after expansion and fitting, based on the correspondence, training the neural network using the point cloud P' at different heights after linear fitting and the point cloud Q'' after plane fitting to obtain a trained neural network; using the actual three-dimensional world coordinates as input and predicting the ideal three-dimensional world coordinates using the trained neural network. The present invention, based on the least squares method and neural network, obtains the mapping relationship between the actual plane at different heights and the fitted plane, and realizes high-precision measurement and correction of lens distortion.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional scanning, and particularly to a lens distortion correction method and system based on the least squares method and neural network. Background Art

[0002] Currently, with the continuous progress of computer vision and sensor technology and the research efforts of many scholars, the accuracy of structured light has been improved: the structured light system combined with industrial lenses and image processing technology has become more powerful and flexible, so it has become one of the popular fields in three-dimensional reconstruction. Among them, the fringe projection profilometry based on phase measurement profilometry has been widely used due to its high accuracy. The fringe projection profilometry usually consists of an industrial camera and a digital projector. The projector generates a fringe pattern and irradiates it on the object surface. After the computer obtains the pattern captured by the camera on the object surface, it decodes the pattern to calculate the depth information.

[0003] The accuracy of fringe projection profilometry is affected by reasons such as intensity noise, non-linear error, and lens distortion. Among them, factors such as intensity noise and non-linear error can reduce the error by using the large-step phase shift method. Lens distortion is difficult to eliminate due to its inherent characteristics, but there are still many methods proposed for this problem. Predistortion is one of them. For example, Chen WX et al. mentioned in "Research on projector distortion correction of PMP system using predistortion fringe[J].Acta Photonica Sinica, 2009, 38(10): 2698 - 2701" that this method applies reverse distortion to the ideal phase according to the calibrated projector lens distortion parameters, so that it becomes an ideal grating again after passing through the lens distortion during the projection process. Based on this method, various methods have been derived. For example, methods such as adding epipolar constraints and compensating for distortion errors are used to improve the compensation accuracy. However, this method is not applicable to binary fringe patterns, which will cause sawtooth phenomena at the binary boundary, affecting the reconstruction accuracy.

[0004] Based on the fact that the carrier phase distribution of any measurement system can be described by rational functions, various methods have also been proposed. For example, Peng et al. proposed an adaptive fringe projection technique in Suppression of projector distortion in phase-measuring profilometry by projecting adaptive fringe patterns [J]. Optics express, 2016, 24(19): 21846-21860. This technique uses Zernike polynomial fitting to obtain the bent carrier phase, and corrects the pixel points obtained on the projection plane with this, and finally constructs an adaptive fringe pattern according to the corrected pixel points to eliminate the carrier phase bending caused by projector distortion. Xing et al. in Iterative calibration method for measurement system having lens distortions in fringe projection profilometry [J]. Optics Express, 2020, 28(2): 1177-1196 noticed that the distortion of the projection lens may deform the fitted phase map, thus causing errors in the estimation of projection parameters. Therefore, an iterative strategy was proposed to overcome this problem: by alternately performing phase fitting and parameter estimation to accurately determine the internal and external parameters of the projector and the lens distortion coefficient.

[0005] Of course, mathematical methods are not limited to phase fitting. For example, the least squares fitting can be used to accurately solve the distortion parameters of the lens, but this method does not take into account the lens distortion of the projector. Considering this problem, Zhang et al. established a scale-offset model to characterize the distortion of the projector in Correcting projector lens distortion in real time with a scale-offset model for structured light illumination [J]. Optics Express, 2022, 30(14): 24507-24522, so as to speed up the distortion correction through a look-up table. Another study combined the camera lens distortion and the projector lens distortion for correction together, and integrated the rational function into the mathematical expression of the out-of-plane height to reduce the influence of lens distortion.

[0006] As is well known, a trained BP neural network can approximate any function. Therefore, neural networks have great potential in the field of optical imaging. Many scholars have used various networks to improve the accuracy of structured light three-dimensional measurement. Lv et al. proposed a DNN in "Projector distortion correction in 3D shape measurement using a structured-light system by deep neural networks[J].Optics Letters, 2020, 45(1): 204-207" that can find the mapping relationship between the three-dimensional coordinates of an object and its corresponding distortion error to compensate for the measurement error caused by projector distortion. Son et al. proposed a fully automatic calibration method for a multi-projector-camera system in "Multiple projector camera calibration by fiducial marker detection[J].IEEE Access, 2023". This method estimates the correspondence between the projector and the camera based on AprilTag markers and YOLOv8 with deformable convolutions, and can automatically complete the calibration and distortion calibration processes.

[0007] Therefore, there is an urgent need for a scheme for lens distortion correction based on the correspondence between the projector and the camera. Summary of the Invention

[0008] The purpose of the present invention is to provide a lens distortion correction method and system based on the least squares method and neural networks. Based on the least squares method and neural networks, the mapping relationship between different height planes and the fitting plane is obtained, and the finally generated point cloud is corrected to achieve high-precision measurement and correction of lens distortion.

[0009] To achieve the above object, the present invention provides the following solution:

[0010] A lens distortion correction method based on the least squares method and neural networks, comprising the following steps:

[0011] S1. Calibrate the system: the measurement system where the camera and the projector are located, obtain the internal parameter matrix and external parameter matrix of the camera and the projector, and determine the correspondence between the three-dimensional world coordinates and the camera pixel coordinates;

[0012] S2. Construct a neural network;

[0013] S3. Use the large-step phase-shifting method to reconstruct the surface of a flat plate large enough to cover the fields of view of the camera and the projector, obtain the actual plane point cloud and expand it;

[0014] S4. Obtain the undistorted point cloud P and the point cloud Q undistorted by OpenCV at multiple different heights based on the expanded actual planar point cloud. Based on the corresponding relationship determined in S1, use the least squares method to perform linear fitting on P and Q respectively to obtain the point clouds P' and Q' at different heights after linear fitting. Then perform planar fitting on Q' to obtain Q'';

[0015] S5. Use the point cloud P' as the input data set of the neural network, and use Q'' as the label data set of the neural network to train the neural network to obtain a trained neural network;

[0016] S6. Use the actual three-dimensional world coordinates as the input, and use the trained neural network to predict the ideal three-dimensional world coordinates.

[0017] Further, in S3, the method for expanding the actual planar point cloud is: expand a sufficient length on the Z-axis at a fixed step size, and the fixed step size is determined by the average adjacent point spacing.

[0018] Further, in S4. Obtain the undistorted point cloud P and the point cloud Q undistorted by OpenCV at multiple different heights based on the expanded actual planar point cloud. Based on the corresponding relationship determined in S1, use the least squares method to perform linear fitting on P and Q respectively to obtain the point clouds P' and Q' at different heights after linear fitting. Then perform planar fitting on Q' to obtain Q'', specifically including:

[0019] S401. Take the expanded actual planar point cloud P at multiple heights, and perform least squares linear fitting on the three-dimensional coordinates corresponding to the same pixel point at each different height respectively to obtain the point cloud P' of each height plane. The coordinates of P' are as the input data set;

[0020] S402. Take a large number of expanded actual planar point clouds at different heights and obtain the point cloud Q after undistorting using OpenCV. Use the least squares linear fitting method described in S401 to obtain the coordinates of the point cloud Q'. Then perform planar fitting on each planar point cloud Q' to obtain the target point cloud coordinates Q'' as the label data set.

[0021] Further, in S401. Take the expanded actual planar point cloud P at multiple heights, and perform least squares linear fitting on the three-dimensional coordinates corresponding to the same pixel point at each different height respectively to obtain the point cloud P' of each height plane. The coordinates of P' are as the input data set, specifically including:

[0022] For the values on the X-axis: Obtain the x mapped by the same pixel point at different heights according to the pixel coordinate index, and then use least squares linear fitting to obtain the new value:

[0023]

[0024] Finally, obtain the x value of the same pixel at different heights, that is

[0025]

[0026] In the formula, h n is the distance between n different planes and the camera, is the x mapped by the same pixel coordinate at different distances, and i = 1, 2,.., u, j = 1, 2,..., v, and are the coefficients of the linear fitting relationship formula of the corresponding pixel point on the x-axis;

[0027] Then, use the same method as processing the values on the X-axis to obtain the values on the Y-axis and the values on the Z-axis Finally, obtain the point cloud coordinates of each height plane as the input of the neural network.

[0028] Furthermore, in S2, the neural network includes an input layer, a hidden layer, and an output layer. Among them, the sizes of the input layer and the output layer are both 3, the size of the hidden layer is 15, and the activation function adopts Sigmoid.

[0029] Furthermore, training the neural network specifically includes: using MSE as the loss function of the neural network, and the algorithm used to train the neural network is the Bayesian regularization algorithm.

[0030] The present invention also provides a lens distortion correction system based on the least square method and the neural network, which is used to execute the lens distortion correction method based on the least square method and the neural network, including:

[0031] A projection matrix acquisition module, which is used to calibrate the system: the measurement system where the camera and the projector are located, obtain the internal parameter matrix and the external parameter matrix of the camera and the projector, and determine the correspondence between the world three-dimensional coordinates and the camera pixel coordinates;

[0032] A neural network construction module, which is used to construct a neural network;

[0033] An expansion module, which is used to reconstruct the surface of a flat plate large enough to cover the fields of view of the camera and the projector by using the large-step phase shift method, obtain the actual plane point cloud and expand it;

[0034] A fitting module, which is used to obtain undistorted point clouds P and point clouds Q that have been undistorted by OpenCV at multiple different heights based on the augmented actual planar point cloud. Based on the corresponding relationship determined by S1, the least squares method is used to perform linear fitting on P and Q respectively, obtaining linearly fitted point clouds P' and Q' at different heights, and then performing planar fitting on Q' to obtain Q'';

[0035] A training module, which is used to use the point cloud P' as the input data set of the neural network and Q'' as the label data set of the neural network to train the neural network, obtaining a trained neural network;

[0036] A prediction module, which is used to take the actual three-dimensional world coordinates as the input and use the trained neural network to predict the ideal three-dimensional world coordinates.

[0037] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects: The lens distortion correction method and system based on the least squares method and neural network provided by the present invention analyze the influence of lens distortion on the final imaging, and propose a method based on linear fitting and neural network. In this method, first, a flat plate surface is reconstructed under the full field of view of the camera, so that each camera pixel point can find the corresponding three-dimensional point, obtaining undistorted point clouds and point clouds that have been undistorted by OpenCV at multiple heights, performing linear fitting on the point clouds at each height to obtain point clouds P' and Q' at different heights, then performing planar fitting on Q' to obtain Q'', and finally using the neural network to map P' to Q'' to achieve the purpose of correcting distortion errors.

[0038] This method obtains planar data with height differences through linear fitting by the least squares method, and uses planar fitting by the least squares method to obtain the corresponding plane at the corresponding height, thereby establishing the mapping relationship between different height planes and the fitted plane under the neural network model, realizing the direct correction of the point clouds generated by the camera and the projector. Through experimental verification, the peak-to-valley value accuracy of the average point spacing is improved by 97.24%, and the root mean square value accuracy of the average point spacing is improved by 96.66%. Description of the Drawings

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required to be used in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0040] Figure 1 It is a flowchart of the lens distortion correction method based on the least squares method and neural network according to the embodiments of the present invention;

[0041] Figure 2The fitting process of the point cloud data on the X-axis at each height plane in the embodiments of the present invention;

[0042] Figure 3 The schematic diagram of the plane point cloud affected in the embodiments of the present invention;

[0043] Figure 4 The least squares linear fitting of the point cloud of a certain height flat plate to the value on the X-axis in the embodiments of the present invention;

[0044] Figure 5 The schematic diagram of the least squares plane fitting process of the point cloud of a certain height plane in the embodiments of the present invention;

[0045] Figure 6 The schematic diagram of the fitting effect in the embodiments of the present invention, where (a) represents the schematic diagram before plane fitting, and (b) represents the schematic diagram after plane fitting;

[0046] Figure 7 The schematic diagram of the neural network in the embodiments of the present invention;

[0047] Figure 8 The schematic diagram of the change of the average loss in the embodiments of the present invention;

[0048] Figure 9 The effect diagram of plane reconstruction in the embodiments of the present invention, where (a), (c), and (e) are the point cloud diagrams without distortion correction, the point cloud diagrams after distortion correction by OpenCV, and the de-distorted point cloud diagrams predicted by the neural network respectively, and (b), (d), and (f) are the point distance distribution diagrams of the above three point clouds and the ideal point cloud respectively;

[0049] Figure 10 The experimental test result diagram of the stacked flat plates in the embodiments of the present invention. In Figure (a), the left figure is the schematic diagram of the original point cloud, and the right figure is the enlarged view. In Figure (b), the left figure is the schematic diagram of the point cloud de-distorted by OpenCV, and the right figure is the enlarged view. In Figure (c), the left figure is the schematic diagram of the point cloud de-distorted by the neural network, and the right figure is the enlarged view. Detailed implementation manners

[0050] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0051] The objective of the present invention is to provide a lens distortion correction method and system based on the least squares method and neural network. By obtaining the mapping relationship between different height planes and the fitting plane based on the least squares method and neural network, the finally generated point cloud is corrected to achieve high-precision measurement and correction of lens distortion.

[0052] To make the above objectives, features, and advantages of the present invention more obvious and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0053] As Figure 1 shown, a lens distortion correction method based on the least squares method and neural network provided by an embodiment of the present invention includes the following steps:

[0054] S1. Calibrate the system: Calibrate the measurement system where the camera and the projector are located, obtain the internal parameter matrix and external parameter matrix of the camera and the projector, and determine the correspondence between the world three-dimensional coordinates and the camera pixel coordinates;

[0055] S2. Construct a neural network;

[0056] S3. Use the large-step phase-shifting method to reconstruct the surface of a flat plate sufficient to cover the fields of view of the camera and the projector, obtain the actual plane point cloud and expand it;

[0057] S4. Based on the expanded actual plane point cloud, obtain the undistorted point cloud P and the point cloud Q that has been undistorted by OpenCV at multiple different heights. Based on the correspondence determined in S1, use the least squares method to perform linear fitting on P and Q respectively to obtain the linearly fitted point clouds P' and Q' at different heights, and then perform plane fitting on Q' to obtain Q";

[0058] S5. Use the point cloud P' as the input data set of the neural network, and use Q" as the label data set of the neural network to train the neural network to obtain a trained neural network;

[0059] S6. Use the actual three-dimensional world coordinates as the input, and use the trained neural network to predict the ideal three-dimensional world coordinates.

[0060] In this embodiment, in S3, the method for expanding the actual plane point cloud is: expand a sufficient length on the Z axis with a fixed step size, and the fixed step size is determined by the average adjacent point spacing.

[0061] In this embodiment, S4: Obtain the undistorted point cloud P and the point cloud Q that has been undistorted by OpenCV at multiple different heights based on the expanded actual planar point cloud. Based on the corresponding relationship determined in S1, use the least squares method to perform linear fitting on P and Q respectively to obtain the point clouds P' and Q' at different heights after linear fitting, and then perform planar fitting on Q' to obtain Q". Specifically, it includes:

[0062] S401: Take the expanded actual planar point cloud P at multiple heights, and perform least squares linear fitting on the three-dimensional coordinates corresponding to the same pixel point at each different height to obtain the point cloud P' of each height plane. The coordinates of P' are used as the input data set;

[0063] S402: Take a large number of expanded actual planar point clouds at different heights and obtain the point cloud Q after undistorting using OpenCV. Use the least squares linear fitting method described in S401 to obtain the coordinates of the point cloud Q', and then perform planar fitting on each planar point cloud Q' to obtain the target point cloud coordinates Q", which are used as the label data set.

[0064] In this embodiment, in S401: Take the expanded actual planar point cloud at multiple heights, and perform least squares linear fitting on the three-dimensional coordinates corresponding to the same pixel point at each different height to obtain the point cloud coordinates of each height plane used as the input data set. Specifically, it includes:

[0065] For the values on the X-axis: Obtain the x mapped by the same pixel point at different heights according to the pixel coordinate index, and then use least squares linear fitting to obtain the new value:

[0066]

[0067] Finally, obtain the x value of the same pixel point at different heights, that is

[0068]

[0069] where h n is the distance between n different planes and the camera, is the x mapped by the same pixel coordinate at different distances, and i = 1, 2,.., u, j = 1, 2,..., v, and are the coefficients of the linear fitting relationship of the corresponding pixel point on x,

[0070] Then use the same method as for processing the values on the X-axis to obtain the values on the Y-axis and the values on the Z-axis Finally, obtain the point cloud coordinates of each height plane As the input of the neural network.

[0071] In this embodiment, the neural network includes an input layer, a hidden layer, and an output layer. Among them, the sizes of the input layer and the output layer are both 3, the size of the hidden layer is 15, and the sigmoid function is used as the activation function.

[0072] In this embodiment, training the neural network specifically includes: using MSE as the loss function of the neural network, and the algorithm used to train the neural network is the Bayesian regularization algorithm.

[0073] Embodiment 2

[0074] Based on the method described in Embodiment 1, this embodiment provides a further implementation method.

[0075] 2.1. Influence of lens distortion

[0076] Before starting projection reconstruction, the camera and the projector need to be calibrated first. The purpose is to determine the internal and external parameters and establish the relationship between the absolute phase and the three-dimensional point cloud at the same time. Here, the Cui calibration method is briefly reviewed: The camera is regarded as a pinhole model, and the world coordinates (X w , Y w , Z w ) and the camera pixel coordinates (u c , v c ) have a corresponding relationship:

[0077]

[0078] Among them, s c is the scale factor, the internal parameter matrix and f is the focal length of the camera, dx and dy are the physical sizes of the pixels in the horizontal and vertical directions of the image respectively, (u0, v0) is the coordinate of the origin in the camera coordinate system, and the external parameter matrix is where R c is a 3×3 rotation matrix, and T c is a 3×1 translation matrix.

[0079] Since the optical path of the projector is opposite to that of the camera, the projector can be regarded as an inverse camera, and its model is similar to the camera model (this principle comes from the research of Feng S et al., Calibration of fringe projection profilometry: A comparative review [J]. Optics and lasers in engineering, 2021, 143: 106622):

[0080]

[0081] According to Equation (1) and Equation (2), the projection matrices A c and A p :

[0082]

[0083] By combining Equation (1), Equation (2), Equation (3) and Equation (4), the conversion formula (5) can be obtained:

[0084]

[0085] It should be noted that Equation (1) and Equation (2) are obtained under the condition that the model is completely ideal. In practice, due to reasons such as the design and manufacture of the lens, the shape and position of the object in the image will be distorted, that is, the shape and size of the object in the image do not match the actual scene, affecting the processing and analysis of the image. Therefore, distortion correction is often required. Generally speaking, image distortion is often divided into radial distortion and tangential distortion, and the distortion model can be expressed by the following formula:

[0086]

[0087] In the formula, k1, k2 and k3 are radial distortion coefficients, and p1 and p2 are tangential distortion coefficients. At present, these distortion coefficients can be obtained through software such as OpenCV and MATLAB. It should be noted that the more distortion coefficients participating in the modeling, the better. Too many coefficients may cause the distortion coefficients obtained through the program to become unstable. The radius of curvature is the distorted coordinate, (x c , y c ) is the ideal coordinate, and:

[0088]

[0089] Then the distorted pixel coordinates can be obtained by the following formula:

[0090]

[0091] Different from the camera that can directly calibrate the distorted image using Equation (6), the projector that cannot capture the image does not know the corresponding relationship between its pixel points and object points. At present, most solutions are to pre-distort and calibrate the fringe pattern in a similar way to Equation (6) in advance, and then project it to achieve the purpose of distortion correction.

[0092] According to Equation (5), whether it is the camera pixel coordinates or the projector pixel coordinates, they are all involved in the calculation of the three-dimensional coordinates. When the image is distorted, according to Equation (8), the pixel points are affected by the distortion, resulting in the distortion of the three-dimensional points.

[0093] 2.2 Compensation Method

[0094] OpenCV provides de-distortion technology, so this technology can be used to correct lens distortion. However, experiments show that it cannot completely correct the distortion, and the correction performance still needs to be further improved. The specific analysis will be presented later.

[0095] It can be known that in structured light three-dimensional reconstruction, when reconstructing a plane without considering lens distortion, the resulting point cloud should have a sufficiently low flatness. Based on this, in this embodiment, the least squares method and neural network are used to learn the error between the actual plane and the ideal plane, so as to correct the overall error pixel by pixel. The specific steps are as follows.

[0096] Step 1: Calibrate the system to obtain the internal parameter matrix and external parameter matrix of the camera and projector.

[0097] Step 2: Construct a neural network. Design the network to have one input layer, one hidden layer, and one output layer. The size of the input layer and output layer is 3, and the size of the hidden layer is 15. The activation function uses Sigmoid.

[0098] Step 3: Make a dataset. As described in 2.1, each camera pixel point has a different degree of distortion. Therefore, to ensure that the distortion of the entire field of view can be corrected and to avoid non-linear errors, the large-step phase-shift method is used to reconstruct the surface of a flat plate large enough to cover the fields of view of the camera and projector to obtain the actual point cloud. However, due to insufficient plane height information, only using the plane point cloud data obtained at a fixed distance to train the network will cause the network to be unable to learn the information at different heights. Therefore, it is necessary to expand the obtained dataset: expand a sufficient length on the Z-axis at a fixed step size, and this fixed step size is determined by the average adjacent point spacing. In practice, it is difficult to accurately obtain the plane data at each different height. Considering that in the camera's field of view, the optical path is a straight line, it can be known that at different heights, the three-dimensional coordinates corresponding to the same camera pixel point have a linear relationship.

[0099] According to the above, take the plane point cloud data reconstructed at n different heights, and perform least squares linear fitting on the three-dimensional coordinates corresponding to the same pixel point at each different height. Taking the values on the X-axis as an example, the specific fitting process is as Figure 2 shown.

[0100] As Figure 2As shown, first, obtain the x values mapped by the same pixel point at different heights according to the pixel coordinate index, and finally use the least squares linear fitting to obtain the new values:

[0101]

[0102] In Equation (9), h n is the distance between n different planes and the camera, is the x value mapped by the same pixel coordinate at different distances, and i = 1, 2,.., u, j = 1, 2,..., v. and are the coefficients of the linear fitting relationship of the corresponding pixel point on x. Finally, obtain the x value of the same pixel point at different heights through Equation (10): and are obtained by a similar method to obtain the point cloud coordinates of each height plane as the input of the neural network.

[0103] The input data set of the neural network is completed as above. Next, make the label data set of the neural network. Since a large number of plane point clouds at different heights are required, its preprocessing is the same as that of the input data set. Moreover, in order to ensure the quality of the data, the point cloud obtained by using the OpenCV undistortion technology is used as the source point cloud of the label.

[0104] When the plane point cloud set at sufficient heights is obtained, plane fitting is performed on each plane point cloud. Since the point cloud data is easily affected by measurement system errors, noise, or sampling methods, the arrangement of the obtained point cloud data in space is disordered, such as Figure 3 , due to the influence of errors, the edge point cloud shows a wavy shape, which will affect the point cloud data obtained by fitting the actual planes at different heights.

[0105] Therefore, before returning the plane point cloud to the same plane, it is first necessary to perform regression and sorting on these wavy point clouds. Still taking the values on the X axis as an example, the specific fitting process is as Figure 3 shown. <l

[0106] According to Figure 4 , a linear fitting method similar to Equation (9) and Equation (10) is still used to process the arrangement of the point cloud in space. The same is true for the values on the Y axis. Finally, according to ]>and the least squares plane fitting to obtain The specific fitting process is as Figure 5 shown, and the finally obtained is used as the label of the neural network.

[0107] The final fitting effect is as Figure 6 shown. Figure 6(a) Schematic diagram of planar point cloud fitted for a certain height. Figure 6 (b) Schematic diagram of planar point cloud after planar fitting. It can be seen that after linear fitting, the point cloud no longer arranges in a wavy pattern.

[0108] Step 4: Train the network. Randomly extract sufficient data from planes with different heights for each to supplement the dataset. Since the height difference is small, if the dataset containing all heights is used for training, it is likely to lead to insufficient network accuracy. To ensure accuracy while reducing the model size, planes with large height differences are spliced into a dataset for training. When training the network, randomly extract 70% of the dataset as the training set and 30% as the test set. The final model architecture is as Figure 7 shown. It should be noted that the input point cloud is fitted from the actual point cloud without using the Opencv undistortion technology, while the target point cloud is fitted from the actual point cloud obtained by using the Opencv undistortion technology, because this will make it closer to the real ideal point cloud.

[0109] When training this model, use MSE as the loss function of the network, the number of epochs is 1000, and the training algorithm selects the Bayesian regularization algorithm. This algorithm is different from the traditional algorithms of point estimation. Since it calculates the posterior distribution of the weights and biases in the neural network, it can provide information about the uncertainty of the model parameters. In areas with more data, the training data can narrow the posterior distribution of the parameters, while in areas with less data, the posterior distribution of the parameters will be wider. Therefore, this algorithm can achieve accurate fitting in areas with more data and retain greater uncertainty in areas with less data (in the point cloud dataset, areas with less data are usually data noise), thereby improving the robustness and generalization ability of the network, which is applicable to large point cloud datasets.

[0110] Step 5: Predict data. Input the actual three-dimensional world coordinates (X w , Y w , Z w ), and predict the ideal three-dimensional world coordinates through the trained network

[0111] In a further embodiment, a three-dimensional structured light measurement system is adopted, which includes an industrial camera (Hikvision MV-CS050-10GM-PRO) with a resolution of 2448×1848 and two digital projectors (DLP4500) with a resolution of 912×1140. It should be noted that in this experiment, the resolution of the camera was adjusted to 2000×1700 and only one side of the projector was used to verify the method. The specific process is as follows:

[0112] Calibrate the system according to Step 1 to obtain the projection matrices of the camera and the projector:

[0113]

[0114] After constructing the neural network according to Step 2 and making the dataset according to Step 3, train the neural network. In this experiment, five planes with different heights were used to fit and obtain the point clouds at each height. The graph of the average loss change of multiple trained networks is as Figure 8 shown. The optimal average training loss is 6.85307×10 -5 , and the optimal average test loss is 6.85041×10 -5 . It is not difficult to see that the two curves show a parallel downward trend, indicating that the network is well trained.

[0115] According to Step (5), select the plane for verification, measure the point spacing of the point cloud to measure the actual performance of the network, and the final result is as Figure 9 shown. Among them, (a), (c), and (e) are the point cloud diagrams without distortion correction, the point cloud diagrams after distortion correction using OpenCV, and the de-distorted point cloud diagrams predicted by the neural network, respectively. Figure 9 (b), (d), and (f) are the point spacing distribution diagrams of the above three point clouds and the ideal point cloud, respectively.

[0116] According to Figure 9 and Table 1, the distortion degree of the original point cloud is large, so the point spacing error from the ideal point cloud is large. Its peak-to-valley value (PV) is 13.9119 mm, and the root mean square error (RMSE) is 4.0606 mm. After using the de-distortion technology of OpenCV, although the PV and RMSE decreased by 4.9445 mm and 0.4170 mm respectively, not much improvement was achieved. After de-distortion, the PV and RMSE decreased by 13.4740 mm and 3.9707 mm respectively, reflecting the superiority of the present invention.

[0117] Table 1 Experimental results of plane reconstruction (unit: mm)

[0118]

[0119] It is not difficult to see from the experimental results that the network model accurately predicts the three-dimensional points of the ideal plane. To further verify the performance of the network, the flat plate was measured and predicted at different heights, and the results are shown in Table 2. It can be seen that as the height increases, the distortion degree of the actual point cloud gradually decreases. Before undistortion, the average point spacing PV and RMSE between the actual point cloud and the ideal point cloud are 13.4275 mm and 3.9209 mm respectively. After using the neural network for undistortion, the average point spacing PV and RMSE decrease to 0.3714 mm and 0.1304 mm respectively, and the performance is improved by 97.24% and 96.66% respectively.

[0120] Table 2 Experimental results of plane reconstruction at different heights (unit: mm)

[0121]

[0122]

[0123] According to the statistics in Table 2, the network has satisfactory performance. To verify the correction effect of the plane at different heights, three standard flat plates were overlapped and placed in a stepped shape for three-dimensional reconstruction in the scanner, as Figure 10 shown, Figure 10 (a) is the original point cloud and its enlarged view, Figure 10 (b) is the point cloud undistorted by OpenCV and its method diagram, Figure 10 (c) is the point cloud undistorted by the neural network and its enlarged view. It is not difficult to see that the point cloud undistorted by OpenCV is smoother than the original point cloud, but it can be clearly seen from its enlarged view that there is still a bending phenomenon on the plane at the stepped edge. After undistortion by our method, a point cloud closer to the ideal can be obtained better.

[0124] It should be noted that although the designed network effectively alleviates the influence brought by lens distortion, there are still problems to be solved or the method needs to be improved in the proposed method. For example, since the ideal plane and the plane data at different heights are both obtained by fitting, they inherently carry certain residuals. Therefore, if we want to improve the accuracy of this model, finding more accurate ideal planes and planes at different heights is a problem that needs to be faced.

[0125] Example 3

[0126] This embodiment provides a lens distortion correction system based on the least squares method and neural network, which is used to execute the lens distortion correction method based on the least squares method and neural network described in Embodiment 1 or 2, including:

[0127] A projection matrix acquisition module, which is used to calibrate the measurement system where the camera and the projector are located, obtain the internal parameter matrix and the external parameter matrix of the camera and the projector, and obtain the projection matrices corresponding to the camera and the projector according to the internal parameter matrix and the external parameter matrix;

[0128] A neural network construction module, which is used to construct a neural network;

[0129] An expansion module: which is used to reconstruct the surface of a flat plate large enough to cover the fields of view of the camera and the projector by using the large-step phase-shifting method, obtain the actual plane point cloud and expand it;

[0130] A fitting module, which is used to obtain the undistorted point cloud P and the point cloud Q after undistortion by OpenCV at multiple different heights according to the expanded actual plane point cloud, linearly fit P and Q respectively by using the least squares method to obtain the linearly fitted point clouds P' and Q' at different heights, and then perform plane fitting on Q' to obtain Q";

[0131] A training module, which is used to use the point cloud P' as the input data set of the neural network and Q' as the label data set of the neural network to train the neural network to obtain a trained neural network;

[0132] A prediction module, which is used to take the actual three-dimensional world coordinates as the input and use the trained neural network to predict the ideal three-dimensional world coordinates.

[0133] In summary, the present invention provides a distortion correction method based on linear fitting and a neural network. The network is used to establish a mapping relationship between the actual three-dimensional world coordinates and the ideal three-dimensional points. The actual plane data is collected by the measurement system, and the plane data at different heights is approximately linearly fitted from multiple actual plane data at different heights. The ideal plane data is obtained by analyzing and fitting the actual plane data after the OpenCV undistortion technology. Since the plane data is obtained in the full field of view of the camera, the distortion correction described in this method is actually carried out pixel by pixel of the camera. The experimental results show that after applying this network for distortion correction, the measurement accuracy reaches 0.1304 mm in terms of the root mean square value of the average point spacing, an increase of 96.66%.

[0134] For the remaining technical features in this embodiment, those skilled in the art can flexibly select them according to the actual situation to meet different specific actual needs. However, it is obvious to those of ordinary skill in the art that: it is not necessary to adopt these specific details to implement the present invention. In other instances, in order to avoid confusing the present invention, the well-known compositions, structures or components are not specifically described, and they are all within the scope of the technical solutions claimed in the claims of the present invention.

[0135] Modifications and variations made by persons skilled in the art without departing from the spirit and scope of the present invention shall fall within the scope of protection of the appended claims of the present invention. In the above description, in order to provide a thorough understanding of the present invention, a large number of specific details are set forth. However, it will be apparent to those of ordinary skill in the art that the present invention may be practiced without these specific details. In other instances, well-known technologies, such as specific construction details, operating conditions, and other technical conditions, are not specifically described in order to avoid obscuring the present invention.

[0136] Specific examples are used herein to illustrate the principles and embodiments of the present invention. The description of the above embodiments is only for helping to understand the method and its core idea of the present invention; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific embodiments and application scopes. In summary, the content of this specification should not be construed as a limitation on the present invention.

Claims

1. A lens distortion correction method based on the least squares method and neural network, characterized in that Including the following steps: S1. Calibration system: For the measurement system where the camera and the projector are located, obtain the internal parameter matrices and external parameter matrices of the camera and the projector, and determine the correspondence between the world three-dimensional coordinates and the camera pixel coordinates; S2. Construct a neural network; S3. Use the large-step phase-shifting method to reconstruct the surface of a flat plate large enough to cover the fields of view of the camera and the projector, obtain the actual plane point cloud and expand it; S4. Obtain the undistorted point cloud P and the point cloud Q that has been undistorted by OpenCV at multiple different heights based on the expanded actual plane point cloud. Based on the correspondence determined in S1, use the least squares method to perform linear fitting on P and Q respectively to obtain the point clouds P' and Q' at different heights after linear fitting, and then perform plane fitting on Q' to obtain Q"; S5. Use the point cloud P' as the input data set of the neural network, and use Q" as the label data set of the neural network to train the neural network to obtain a trained neural network; S6. Use the actual three-dimensional world coordinates as the input, and use the trained neural network to predict the ideal three-dimensional world coordinates.

2. The lens distortion correction method based on the least squares method and neural network according to claim 1, wherein In S3, the method for expanding the actual plane point cloud is: expand a sufficient length on the Z-axis at a fixed step size, and the fixed step size is determined by the average adjacent point spacing.

3. The lens distortion correction method based on the least squares method and neural network according to claim 1, characterized in that In S4, according to the expanded actual plane point cloud, obtain the undistorted point cloud P and the point cloud Q that has been undistorted by OpenCV at multiple different heights. Based on the correspondence determined in S1, use the least squares method to perform linear fitting on P and Q respectively to obtain the point clouds P' and Q' at different heights after linear fitting, and then perform plane fitting on Q' to obtain Q". Specifically, it includes: S401. Take the expanded actual planar point cloud P at multiple heights, respectively perform least-squares linear fitting on the three-dimensional coordinates corresponding to the same pixel point at each different height, and obtain the point cloud P' of each height plane. The coordinates of P' are used as the input data set; S402. Take a large number of expanded actual plane point clouds at different heights and undistort them using OpenCV to obtain the point cloud Q. Use the least squares linear fitting method described in S401 to obtain the coordinates of the point cloud Q', and then perform plane fitting on each plane point cloud Q' to obtain the target point cloud coordinates Q", which serve as the label data set.

4. The lens distortion correction method based on the least squares method and neural network according to claim 3, characterized in that In S401, take the augmented actual planar point cloud P at multiple heights, and perform least-squares linear fitting on the three-dimensional coordinates corresponding to the same pixel point at each different height to obtain the point cloud P' of each height plane. The coordinates of P' are As the input data set, it specifically includes: For the values on the X-axis: Obtain the x mapped by the same pixel point at different heights according to the pixel coordinate index, and then use the least squares linear fitting to obtain new values: Finally, obtain the x value of the same pixel at different heights, that is where h n is the distance between n different planes and the camera, is x mapped by the same pixel coordinate at different distances, and i = 1, 2,.., u, j = 1, 2,..., v, and are the coefficients of the linear fitting relationship of the corresponding pixel points on the x-axis; Obtain the values on the Y-axis in the same way as for processing the values on the X-axis and the values on the Z-axis Finally, obtain the point cloud coordinates of each height plane as the input to the neural network.

5. The lens distortion correction method based on the least squares method and neural network according to claim 1, wherein In S2, the neural network includes an input layer, a hidden layer, and an output layer. Among them, the sizes of the input layer and the output layer are both 3, the size of the hidden layer is 15, and the activation function uses Sigmoid.

6. The lens distortion correction method based on the least squares method and neural network according to claim 1, characterized in that The training of the neural network specifically includes: using MSE as the loss function of the neural network, and the algorithm used to train the neural network is the Bayesian regularization algorithm.

7. A lens distortion correction system based on the least squares method and neural network, which is used to execute the lens distortion correction method based on the least squares method and neural network according to any one of claims 1-6, and is characterized in that, Including: A projection matrix acquisition module, which is used for the calibration system: the measurement system where the camera and the projector are located, to obtain the internal parameter matrices and external parameter matrices of the camera and the projector, and determine the correspondence between the world three-dimensional coordinates and the camera pixel coordinates; A neural network construction module, which is used to construct a neural network; An expansion module, which is used to use the large-step phase-shifting method to reconstruct the surface of a flat plate large enough to cover the fields of view of the camera and the projector, obtain the actual plane point cloud and expand it; A fitting module, which is used to obtain the undistorted point cloud P and the point cloud Q that has been undistorted by OpenCV at multiple different heights according to the augmented actual planar point cloud. Based on the corresponding relationship determined by S1, the least squares method is used to perform linear fitting on P and Q respectively, obtaining the point clouds P' and Q' at different heights after linear fitting, and then performing planar fitting on Q' to obtain Q''; A training module, which is used to use the point cloud P' as the input data set of the neural network, and use Q'' as the label data set of the neural network to train the neural network, obtaining a trained neural network; A prediction module, which is used to take the actual three-dimensional world coordinates as the input and use the trained neural network to predict the ideal three-dimensional world coordinates.

Images