A nonlinear calibration method and device for laser surface of line structured light 3D camera

Abstract

The present invention discloses a nonlinear calibration method and device for a line structured light 3D camera laser surface, including: performing reverse projection according to the camera's intrinsic parameter matrix, lens distortion parameters and extrinsic parameter matrix to obtain the three-dimensional coordinates of the center point of the laser line in the two-dimensional image captured by the line structured light 3D camera in the camera coordinate system; fitting the surface using a trained neural network model according to the three-dimensional coordinates of the center point of the laser line to obtain the surface model of the line structured light 3D camera laser surface in the camera coordinate system; obtaining the ray corresponding to each pixel according to the three-dimensional coordinates of each pixel on the center contour of the laser line in the two-dimensional image captured by the line structured light 3D camera; calculating the intersection of the ray corresponding to each pixel and the surface model of the line structured light 3D camera laser surface to complete the three-dimensional reconstruction of the line structured light 3D camera laser surface. The present invention can effectively improve the accuracy of laser surface calibration and the accuracy of three-dimensional reconstruction results.

Description

Technical Field

[0001] The invention relates to a nonlinear calibration method and device for a line structured light 3D camera laser surface, and belongs to the technical field of line structured light 3D camera laser surface calibration. Background Art

[0002] At present, line structured light 3D cameras are widely used in the field of three-dimensional imaging. Traditional line structured light 3D cameras usually include a laser emitter, a camera and an image processing unit. The laser emitter generates structured line light, which is projected onto the target surface to form a laser line. The camera records the image of the laser line through a photosensitive element to form a two-dimensional projection, and the image processing unit uses a plane fitting algorithm to process the image, trying to use the best fitting plane to represent the projection of the laser surface in the two-dimensional image, thereby inferring the position of the laser plane in three-dimensional space to extract the spatial information of the laser plane.

[0003] In traditional methods, the laser surface calibration of line structured light 3D cameras usually defaults to a plane, and the plane fitting method is used to calculate the position of the laser plane relative to the camera coordinate system. However, in fact, the laser line may appear like a bow or S shape, which makes the laser surface a nonlinear surface. When dealing with complex surfaces, the accuracy of traditional laser plane calibration methods is limited, and it is difficult to accurately restore the true shape of the laser surface. In addition, the shape adaptability of traditional methods is poor, and they perform poorly when facing complex laser surface shapes such as nonlinear and curved surfaces, and cannot meet high-precision requirements. Summary of the invention

[0004] In order to solve the problems existing in the prior art, the present invention proposes a nonlinear calibration method and device for the laser surface of a line structured light 3D camera, which fully takes into account the problem that the laser line may present nonlinear shapes such as bow shape and S shape. The nonlinear calibration method is used to more accurately describe the shape of the laser surface, thereby improving the calibration accuracy, which is conducive to improving the adaptability of line structured light 3D cameras in a wider range of application scenarios.

[0005] In order to solve the above technical problems, the present invention adopts the following technical means:

[0006] In a first aspect, the present invention provides a nonlinear calibration method for a laser surface of a line structured light 3D camera, comprising the following steps:

[0007] Through camera parameter calibration, the intrinsic parameter matrix, lens distortion parameters and extrinsic parameter matrix of the line structured light 3D camera are obtained;

[0008] Performing back projection according to the intrinsic parameter matrix, the lens distortion parameters and the extrinsic parameter matrix to obtain the three-dimensional coordinates of the center point of the laser line in the two-dimensional image captured by the line structured light 3D camera in the camera coordinate system;

[0009] According to the three-dimensional coordinates of the center point of the laser line, the surface is fitted using the trained neural network model to obtain a surface model of the laser surface of the line structured light 3D camera in the camera coordinate system;

[0010] Obtaining a ray corresponding to each pixel point according to the three-dimensional coordinates of each pixel point on the center contour of the laser line in the two-dimensional image captured by the line structured light 3D camera;

[0011] According to the rays corresponding to each pixel point and the surface model of the line structured light 3D camera laser surface, the intersection points of all rays and the surface model are obtained to complete the three-dimensional reconstruction of the line structured light 3D camera laser surface.

[0012] In combination with the first aspect, further, the camera parameters are calibrated using the Zhang Zhengyou calibration method to obtain a calibrated intrinsic parameter matrix, lens distortion parameters, and extrinsic parameter matrix;

[0013] Optimize the calibration internal and external parameter matrices using maximum likelihood estimation;

[0014] The internal parameter matrix and external parameter matrix optimized by maximum likelihood estimation are used as initial values, and further optimized using LM least squares method to obtain the final internal parameter matrix and external parameter matrix.

[0015] In combination with the first aspect, further, performing inverse projection according to the intrinsic parameter matrix, the lens distortion parameter and the extrinsic parameter matrix to obtain the three-dimensional coordinates of the center point of the laser line in the two-dimensional image captured by the line structured light 3D camera in the camera coordinate system includes:

[0016] A flat plate larger than the camera field of view is fixed on the slide rail of the line structured light 3D camera, wherein the angle between the plane of the flat plate and the projection direction of the laser of the line structured light 3D camera is in the range of 85° to 95°;

[0017] Set N sampling points with equal distances within the depth of field of the camera, collect two images at each sampling point, and obtain N groups of sampling images, wherein one sampling image is taken by turning off the laser and attaching the chess calibration plate to the flat plate, and the other sampling image is taken by turning on the laser and removing the calibration plate;

[0018] According to the intrinsic parameter matrix, the lens distortion parameters and the preset calibration plate parameters, the position and posture of the calibration plate in the camera coordinate system are obtained;

[0019] Move the calibration plate posture along the negative direction of the Z axis of the calibration plate coordinate system by the thickness of the calibration plate to obtain the posture of the flat plate plane;

[0020] The Steger algorithm is used to extract the coordinates of the center point of the laser line in N groups of sampled images. The three-dimensional coordinates of the center point of the laser line in the camera coordinate system are calculated by combining the coordinates of the center point with the posture of the flat plane to obtain a three-dimensional coordinate point set containing N coordinate values.

[0021] In combination with the first aspect, further, the three-dimensional coordinates of the center point of the laser line are normalized, and the features of the normalized three-dimensional coordinates are extracted using the trained PointNet network, and the surface is fitted according to the feature information to obtain the surface model of the laser surface of the line structured light 3D camera in the camera coordinate system.

[0022] In combination with the first aspect, further, obtaining the ray corresponding to each pixel point according to the three-dimensional coordinates of each pixel point on the center contour of the laser line in the two-dimensional image captured by the line structured light 3D camera includes:

[0023] Obtain a two-dimensional image taken by a line structured light 3D camera;

[0024] Using the lens distortion parameters, the two-dimensional coordinates of each pixel point on the central contour of the laser line in the two-dimensional image are corrected to obtain the corrected two-dimensional coordinates;

[0025] The corrected two-dimensional coordinates are converted into three-dimensional coordinates in the camera coordinate system using the intrinsic parameter matrix to obtain the three-dimensional coordinates of each pixel point on the center contour of the laser line in the two-dimensional image, where the conversion formula is:

[0026] ;

[0027] ;

[0028] ;

[0029] in, is the three-dimensional coordinate of the pixel point on the center contour of the laser line in the two-dimensional image, ( , ) is the principal point coordinate in the camera intrinsic parameter matrix, is the corrected two-dimensional coordinate of the pixel point on the center contour of the laser line in the two-dimensional image;

[0030] For each pixel point, the ray corresponding to the pixel point is obtained by pointing to the three-dimensional coordinates of the pixel point in the camera coordinate system from the origin of the line structured light 3D camera.

[0031] In combination with the first aspect, further, based on the rays corresponding to each pixel point and the surface model of the line structured light 3D camera laser surface, an optimization problem of minimizing the distance between the point on each ray and the laser surface is constructed. By solving the optimization problem, the three-dimensional coordinates of the intersection of each ray and the surface model are obtained, and then the laser surface is obtained.

[0032] In a second aspect, the present invention provides a nonlinear calibration device for a laser surface of a line structured light 3D camera, comprising:

[0033] The camera parameter calibration module is used to obtain the intrinsic parameter matrix, lens distortion parameters and extrinsic parameter matrix of the line structured light 3D camera through camera parameter calibration;

[0034] A reverse projection module, used to perform reverse projection according to the intrinsic parameter matrix, the lens distortion parameters and the extrinsic parameter matrix to obtain the three-dimensional coordinates of the center point of the laser line in the two-dimensional image taken by the line structured light 3D camera in the camera coordinate system;

[0035] A surface fitting module is used to fit the surface according to the three-dimensional coordinates of the center point of the laser line using a trained neural network model to obtain a surface model of the laser surface of the line structured light 3D camera in the camera coordinate system;

[0036] A ray generation module, used to obtain a ray corresponding to each pixel point according to the three-dimensional coordinates of each pixel point on the center contour of the laser line in the two-dimensional image captured by the line structured light 3D camera;

[0037] The laser surface calibration module is used to obtain the intersection points of all rays and the surface model according to the rays corresponding to each pixel point and the surface model of the line structured light 3D camera laser surface, and complete the three-dimensional reconstruction of the line structured light 3D camera laser surface.

[0038] In combination with the second aspect, further, the back projection module is specifically used for:

[0039] A flat plate larger than the camera field of view is fixed on the slide rail of the line structured light 3D camera, wherein the angle between the plane of the flat plate and the projection direction of the laser of the line structured light 3D camera is in the range of 85° to 95°;

[0040] Set N sampling points with equal distances within the depth of field of the camera, collect two images at each sampling point, and obtain N groups of sampling images, wherein one sampling image is taken by turning off the laser and attaching the chess calibration plate to the flat plate, and the other sampling image is taken by turning on the laser and removing the calibration plate;

[0041] According to the intrinsic parameter matrix, the lens distortion parameters and the preset calibration plate parameters, the position and posture of the calibration plate in the camera coordinate system are obtained;

[0042] Move the calibration plate posture along the negative direction of the Z axis of the calibration plate coordinate system by the thickness of the calibration plate to obtain the posture of the flat plate plane;

[0043] The Steger algorithm is used to extract the coordinates of the center point of the laser line in N groups of sampled images. The three-dimensional coordinates of the center point of the laser line in the camera coordinate system are calculated by combining the coordinates of the center point with the posture of the flat plane to obtain a three-dimensional coordinate point set containing N coordinate values.

[0044] In combination with the second aspect, further, the ray generation module is specifically used to:

[0045] Obtain a two-dimensional image taken by a line structured light 3D camera;

[0046] Using the lens distortion parameters, the two-dimensional coordinates of each pixel point on the central contour of the laser line in the two-dimensional image are corrected to obtain the corrected two-dimensional coordinates;

[0047] The corrected two-dimensional coordinates are converted into three-dimensional coordinates in the camera coordinate system using the intrinsic parameter matrix to obtain the three-dimensional coordinates of each pixel point on the center contour of the laser line in the two-dimensional image, where the conversion formula is:

[0048] ;

[0049] ;

[0050] ;

[0051] in, is the three-dimensional coordinate of the pixel point on the center contour of the laser line in the two-dimensional image, ( , ) is the principal point coordinate in the camera intrinsic parameter matrix, is the corrected two-dimensional coordinate of the pixel point on the center contour of the laser line in the two-dimensional image;

[0052] For each pixel point, the ray corresponding to the pixel point is obtained by pointing to the three-dimensional coordinates of the pixel point in the camera coordinate system from the origin of the line structured light 3D camera.

[0053] In combination with the second aspect, further, the laser surface calibration module is specifically used for:

[0054] According to the surface model of the laser surface of the line structured light 3D camera corresponding to each pixel point, an optimization problem of minimizing the distance between the point on each ray and the laser surface is constructed. By solving the optimization problem, the three-dimensional coordinates of the intersection of each ray and the surface model are obtained, and then the laser surface is obtained.

[0055] The following advantages can be obtained by using the above technical means:

[0056] The present invention proposes a nonlinear calibration method and device for the laser surface of a line structured light 3D camera, which fully considers the problem that the laser line may present nonlinear shapes such as bow shape and S shape, and fits the laser surface through a neural network model. It can be applicable to the laser surface whose parameter model is unknown, and more accurately describes the shape of the laser surface. The present invention combines ray and surface models, calculates the three-dimensional coordinates of points on the laser surface through an optimization problem solving method, and can realize three-dimensional reconstruction of the laser surface and nonlinear calibration of the laser surface, effectively improving the accuracy of laser surface calibration and the accuracy of three-dimensional reconstruction results. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 A schematic diagram of the steps of a nonlinear calibration method for a laser surface of a line structured light 3D camera according to the present invention;

[0058] Figure 2 The present invention is a schematic structural diagram of a nonlinear calibration device for a laser surface of a line structured light 3D camera. DETAILED DESCRIPTION

[0059] The technical solution of the present invention is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations on the technical solution of the present invention. The embodiments of the present invention and the technical features in the embodiments may be combined with each other unless there is a conflict.

[0060] Example 1

[0061] This embodiment introduces a nonlinear calibration method for the laser surface of a line structured light 3D camera, such as Figure 1 As shown, the following steps are included:

[0062] Step S1: obtain the intrinsic parameter matrix, lens distortion parameters and extrinsic parameter matrix of the line structured light 3D camera through camera parameter calibration.

[0063] Under the online structured light 3D camera, the pixel coordinates of the corner points of the chessboard calibration plate in the image taken by the camera are extracted, and the homography matrix is ​​obtained through the correspondence between the pixel coordinates of the two-dimensional image and the three-dimensional world coordinates, and then the camera's intrinsic parameter matrix, lens distortion parameters, and extrinsic parameter matrix are obtained. Finally, the optimization method is used to optimize the parameters and obtain the optimal intrinsic parameter matrix, lens distortion parameters, and extrinsic parameter matrix.

[0064] The correspondence between the pixel coordinates of a two-dimensional image and the three-dimensional world coordinates is as follows:

[0065] (1)

[0066] in, is the distance from the pixel point in the camera coordinate system to the optical center of the camera along the Z axis; u , v ) is the coordinate of the pixel point in the two-dimensional image in the camera coordinate system, in pixels; , are the focal lengths in the X-axis and Y-axis directions in the camera intrinsic parameter matrix, in pixels, reflecting the magnification capability of the camera lens; ( , ) is the principal point coordinate in the camera intrinsic parameter matrix; R ,T are the rotation matrix and translation vector between the camera coordinate system and the world coordinate system respectively; is a pixel ( u , v ) is a three-dimensional coordinate in the world coordinate system, used to describe the position of an object in the real world.

[0067] The lens distortion parameters include radial distortion coefficient and tangential distortion coefficient, which are used to describe the distortion of the image captured by the line structured light 3D camera. In the embodiment of the present invention, the expression of the image distortion model is:

[0068] (2)

[0069] in, is the distorted two-dimensional coordinate, is the radial distortion coefficient, is the tangential distortion coefficient, is the distance from the coordinate to the origin.

[0070] In an embodiment of the present invention, the Zhang Zhengyou calibration method is used to calibrate the camera parameters, and the maximum likelihood estimation is used to optimize the calibration results. Finally, the results after the maximum likelihood estimation optimization are used as initial values, and the LM least squares method is used for further optimization to obtain more accurate intrinsic parameter matrices and extrinsic parameter matrices.

[0071] Step S2: perform back projection according to the intrinsic parameter matrix, the lens distortion parameters and the extrinsic parameter matrix to convert the two-dimensional coordinates of the center point of the laser line in the two-dimensional image captured by the line structured light 3D camera into three-dimensional coordinates in the camera coordinate system.

[0072] Calibration of a structured light system usually involves obtaining the geometric relationship between the camera and the light source, and reflecting the projection characteristics of the light in the scene. This process can usually be achieved using a calibration plate, and the specific operations are as follows:

[0073] Step S21, prepare a plate larger than the camera field of view and a calibration plate of suitable size and known thickness. Fix the plate on the slide rail, and the laser projection direction of the line structured light 3D camera is roughly perpendicular to the plane of the plate, and the angle between the two is 85°~95°; sample N times (N>10) at equal distances within the depth of field of the camera, and take two images at each sampling position to obtain N groups of sampled images.

[0074] In each set of sampling images, the first sampling image is taken with the laser turned off and the checkerboard calibration plate attached to the flat plate. It is necessary to ensure that the surface features of the calibration plate are complete and the corners are clear in the image. The second sampling image is taken with the laser turned on and the checkerboard calibration plate removed. It is necessary to ensure that the laser line can be clearly imaged. Note that two different exposures are generally used for taking pictures.

[0075] In an embodiment of the present invention, the calibration plate has a plane or a two-dimensional pattern with regular features, and the surface of the calibration plate is preferably flat so that the features on the surface can be accurately detected. The present invention can make the light spot easier to detect by attaching a reflective or scattering material to the calibration plate. The calibration plate, slide rail, etc. used in the present invention are relatively common, do not require special processing, are simple to operate, and are easy to implement.

[0076] Step S22, according to the camera's intrinsic parameter matrix, lens distortion parameters and calibration plate parameters, obtain the position and posture of the calibration plate in the camera coordinate system, and move the calibration plate posture along the negative direction of the calibration plate coordinate system by the thickness of the calibration plate to obtain the position and posture of the flat plate plane.

[0077] Step S23, use the Steger algorithm to extract the coordinates of the center point of the laser line in the N groups of sampled images sampled in step S21, and combine the plane posture of the flat plate to calculate the three-dimensional coordinate point set of the laser line center point set in the camera coordinate system. Each group of sampled images can calculate the three-dimensional coordinates of a laser line center point, and the three-dimensional coordinate point set includes N coordinate values.

[0078] Step S3: According to the three-dimensional coordinates of the center point of the laser line, the surface is fitted using the trained neural network model to obtain a surface model of the laser surface of the line structured light 3D camera in the camera coordinate system.

[0079] The present invention considers that the laser surface is a complex surface structure, and its surface model is unknown. In order to obtain the surface model of the laser surface, the present invention uses the non-parametric method in deep learning to fit the surface, and specifically uses the PointNet network to implement surface fitting. PointNet is a deep learning architecture designed specifically for processing unordered point cloud data, and is suitable for the analysis and processing of three-dimensional point sets.

[0080] The specific operations of step S3 include:

[0081] Step S31, performing data preprocessing on the three-dimensional coordinate point set obtained in step 2.

[0082] The 3D coordinates in the 3D coordinate point set are normalized to ensure that the PointNet network can effectively process data of different scales.

[0083] If the set of three-dimensional coordinate points is too large, the amount of data can be reduced by random sampling for easier processing.

[0084] Step S32: Use the trained PointNet network to extract features from the normalized three-dimensional coordinates, and perform surface fitting based on the feature information to obtain a surface model of the laser surface.

[0085] The normalized three-dimensional coordinates are received through the input layer of the PointNet network; the local features of the center point of each laser line in the three-dimensional coordinate point set are extracted through a series of convolutional layers of the PointNet network; and the global features are extracted from the local features of all points using a symmetric function (such as maximum pooling). After feature extraction, the present invention constructs a surface based on the key points in the global features and the Bessel formula. During the feature extraction process, the present invention ensures that the network is insensitive to the order of the input points by applying shared weights.

[0086] In an embodiment of the present invention, the training process of the PointNet network is as follows: obtaining a training sample set, wherein the training sample includes a plurality of calibrated three-dimensional coordinates; initializing the PointNet network parameters; processing the training samples using the PointNet network, extracting key features for surface reconstruction, selecting key points for constructing the Bezier surface from the extracted features, wherein the feature intensity-significant points are selected as key points, constructing the surface using the Bezier formula and the selected key points, and then obtaining z corresponding to x and y based on the constructed surface inference to obtain a predicted point set; calculating the loss value based on the predicted point set and the real point set of the training sample, and adjusting the network weights using an optimization algorithm such as Adam or SGD until the network loss converges and the network training is completed.

[0087] A 3D Bezier surface is defined by a set of control points that affect the shape of the surface. 3D Bezier surface of order, assuming we have Key Points ,in and , each key point is a three-dimensional vector. The surface can be represented as:

[0088] (3)

[0089] in, is a three-dimensional Bezier surface, are parametric coordinates that define points on the surface, , Usually in the interval Internal changes, 0 represents the starting point, 1 represents the end point, by changing , The value of can be moved on the surface to generate different parts of the surface; and are Bessel basis functions, respectively defined as:

[0090] (4)

[0091] In formula (3), each and are all scalars used to weight the corresponding control points . By and By evaluating this formula at different values ​​of , the entire surface can be generated.

[0092] In the embodiment of the present invention, considering that the overall shape of the surface fitting is more important than the details, Chamfer distance is selected as the loss function. Chamfer distance is a commonly used metric that measures the similarity between two point sets. The expression of the loss function is as follows:

[0093] (5)

[0094] in, The prediction point concentration point G Concentrate on the real point H The Chamfer distance between is the real point set, that is, the three-dimensional coordinate point set calculated by the calibration process, is the prediction point set.

[0095] After PointNet network training is completed, the validation set is used to evaluate the performance of the network to ensure that the model not only performs well on the training data, but also generalizes to new data. According to the evaluation results, the network parameters or training strategy are adjusted. Hyperparameter tuning can be considered, such as adjusting the learning rate, batch size, or number of network layers.

[0096] PointNet is a network that does not deform when arranged, can handle disordered point sets, and does not need to worry about the order of points. It can also learn global features from the entire point set, which is crucial for understanding complex surfaces. Compared with image-based or volume-based methods, PointNet operates directly on point sets and is more efficient.

[0097] The present invention uses the PointNet network to perform surface fitting of three-dimensional point sets, which is an efficient and powerful method. It is particularly suitable for processing large-scale point cloud data, and can learn complex spatial structures, and is suitable for a variety of three-dimensional data processing tasks. However, to obtain the best performance, it may be necessary to carefully tune the network structure and training process.

[0098] Step S4, obtaining the three-dimensional coordinates of each pixel point on the center contour of the laser line in the two-dimensional image taken by the line structured light 3D camera according to the intrinsic parameter matrix and the lens distortion parameters, and obtaining the ray corresponding to each pixel point on the center contour of the laser line in the two-dimensional image according to the three-dimensional coordinates.

[0099] Step S401: extract the two-dimensional coordinates (u, v) of each pixel point on the center contour of the laser line in the two-dimensional image according to the two-dimensional image captured by the line structured light 3D camera, and use the lens distortion parameters to perform distortion correction on the two-dimensional coordinates of each pixel point to obtain the corrected two-dimensional coordinates.

[0100] Step S402: Use the internal parameter matrix to convert the corrected two-dimensional coordinates into three-dimensional coordinates in the camera coordinate system to obtain the three-dimensional coordinates of each pixel point on the center contour of the laser line in the two-dimensional image, where the conversion formula is:

[0101] (6)

[0102] (7)

[0103] (8)

[0104] in, is the three-dimensional coordinate of the pixel point on the center contour of the laser line in the two-dimensional image, ( , ) is the principal point coordinate in the camera intrinsic parameter matrix, It is the corrected two-dimensional coordinate of the pixel point on the center contour of the laser line in the two-dimensional image.

[0105] In the embodiment of the present invention, it is assumed that all points in the normalized camera coordinate system are located on the plane of Z=1.

[0106] Step S403: for each pixel point on the center contour of the laser line in the two-dimensional image, point to the three-dimensional coordinates of the pixel point in the camera coordinate system from the origin of the line structured light 3D camera to obtain the ray corresponding to the pixel point.

[0107] Step S5: According to the surface model of the ray and laser surface corresponding to each pixel point, the three-dimensional coordinates of the intersection points of all rays and the surface model are obtained, the three-dimensional reconstruction of the laser surface of the line structured light 3D camera is completed, and the nonlinear calibration of the laser surface of the line structured light 3D camera is realized.

[0108] Mathematically, rays can be expressed as parametric equations, such as:

[0109] (9)

[0110] in, Indicates the first t Points, For the ray t The coordinates of the points, is the starting point of the ray, which is the camera origin (0,0,0), ( ) is the direction vector of the ray, which is the three-dimensional coordinate .

[0111] Assume that the surface function is S ( x , y ), the surface function can return the input ( x , y ) corresponding to the z value. When the ray intersects the surface, the intersection point satisfies .

[0112] Since the surface function is a surface fitted by a neural network and has no mathematical expression, it is impossible to use conventional numerical methods to calculate the intersection of the surface and the ray. The present invention converts the intersection point solving problem into an optimization problem of minimizing the distance between the point on the ray and the surface, and calculates the three-dimensional coordinates of the intersection point through the existing optimization algorithm.

[0113] Example 2

[0114] Based on the same inventive concept as Example 1, this example introduces a nonlinear calibration device for a laser surface of a line structured light 3D camera, such as Figure 2 As shown, it mainly includes a camera parameter calibration module, a back projection module, a surface fitting module, a ray generation module and a laser surface calibration module.

[0115] The camera parameter calibration module is mainly used to obtain the intrinsic parameter matrix, lens distortion parameters and extrinsic parameter matrix of the line structured light 3D camera through camera parameter calibration.

[0116] The inverse projection module is mainly used to perform inverse projection based on the intrinsic parameter matrix, lens distortion parameters and extrinsic parameter matrix to obtain the three-dimensional coordinates of the center point of the laser line in the two-dimensional image taken by the line structured light 3D camera in the camera coordinate system.

[0117] The surface fitting module is mainly used to fit the surface according to the three-dimensional coordinates of the center point of the laser line using the trained neural network model to obtain the surface model of the laser surface of the line structured light 3D camera in the camera coordinate system;

[0118] The ray generation module is mainly used to obtain the ray corresponding to each pixel point according to the three-dimensional coordinates of each pixel point on the center contour of the laser line in the two-dimensional image taken by the line structured light 3D camera;

[0119] The laser surface calibration module is mainly used to obtain the intersection points of all rays and the surface model according to the rays corresponding to each pixel point and the surface model of the line structured light 3D camera laser surface, and complete the three-dimensional reconstruction of the line structured light 3D camera laser surface.

[0120] The specific functions of the above modules are implemented by referring to the relevant contents of the method in Example 1, which will not be described in detail. It is particularly pointed out that:

[0121] The back projection module is specifically used for:

[0122] A flat plate larger than the camera's field of view is fixed on the slide rail of the line structured light 3D camera, so that the laser projection direction of the line structured light 3D camera is perpendicular to the plane of the flat plate.

[0123] N sampling points with equal distances are set within the depth of field of the camera, and two images are collected at each sampling point to obtain N groups of sampling images, wherein one sampling image is taken with the laser turned off and the chess calibration plate attached to the flat plate, and the other sampling image is taken with the laser turned on and the calibration plate removed.

[0124] According to the intrinsic parameter matrix, lens distortion parameters and preset calibration plate parameters, the position and pose of the calibration plate in the camera coordinate system are obtained.

[0125] The calibration plate posture is moved along the negative direction of the Z axis of the calibration plate coordinate system by the thickness of the calibration plate to obtain the posture of the flat plate plane.

[0126] The Steger algorithm is used to extract the coordinates of the center point of the laser line in N groups of sampling images. The three-dimensional coordinates of the center point of the laser line in the camera coordinate system are calculated by combining the coordinates of the center point with the posture of the flat plane, and a three-dimensional coordinate point set containing N coordinate values ​​is obtained.

[0127] The ray generation module is specifically used for:

[0128] Get a 2D image taken by a line structured light 3D camera.

[0129] The lens distortion parameters are used to correct the two-dimensional coordinates of each pixel point on the central contour of the laser line in the two-dimensional image to obtain the corrected two-dimensional coordinates.

[0130] The corrected two-dimensional coordinates are converted into three-dimensional coordinates in the camera coordinate system using the intrinsic parameter matrix to obtain the three-dimensional coordinates of each pixel point on the center contour of the laser line in the two-dimensional image. The conversion formula is:

[0131] (10)

[0132] (11)

[0133] (12)

[0134] in, is the three-dimensional coordinate of the pixel point on the center contour of the laser line in the two-dimensional image, ( , ) is the principal point coordinate in the camera intrinsic parameter matrix, is the corrected two-dimensional coordinate of the pixel point on the center contour of the laser line in the two-dimensional image, , They are the focal lengths in the X-axis and Y-axis directions in the intrinsic parameter matrix respectively.

[0135] For each pixel point, the ray corresponding to the pixel point is obtained by pointing to the three-dimensional coordinates of the pixel point in the camera coordinate system from the origin of the line structured light 3D camera.

[0136] The laser surface calibration module is specifically used for:

[0137] According to the surface model of the laser surface of the line structured light 3D camera corresponding to each pixel point, an optimization problem of minimizing the distance between the point on each ray and the laser surface is constructed. By solving the optimization problem, the three-dimensional coordinates of the intersection of each ray and the surface model are obtained, and then the laser surface is obtained.

[0138] In summary of the above embodiments, the present invention fully considers the problem that the laser line may present nonlinear shapes such as bow shape and S shape, and fits the laser surface through a neural network model. It can be applicable to laser surfaces whose parameter models are unknown, and more accurately describe the shape of the laser surface. The present invention combines ray and surface models, and calculates the three-dimensional coordinates of points on the laser surface through an optimization problem solving method, which can realize three-dimensional reconstruction of the laser surface and nonlinear calibration of the laser surface, effectively improving the accuracy of laser surface calibration and the accuracy of three-dimensional reconstruction results.

[0139] Compared with the traditional plane fitting method, the present invention can better adapt to various complex laser surface shapes, including curved surfaces and nonlinear structures, which is conducive to improving the adaptability of line structured light 3D cameras in a wider range of application scenarios, expanding the application field of line structured light 3D cameras, and making line structured light 3D cameras more suitable for high-precision three-dimensional imaging tasks, such as precision measurement in manufacturing, virtual reality, medical imaging and other fields. In addition, except for the software algorithm, the calibration plate, slide rail, etc. used in the present invention are relatively common and do not require special processing. Therefore, the operation of the present invention is simpler and easy to implement.

[0140] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A nonlinear calibration method for a laser surface of a line structured light 3D camera, characterized in that: The steps include: Through camera parameter calibration, the intrinsic parameter matrix, lens distortion parameters and extrinsic parameter matrix of the line structured light 3D camera are obtained; Performing back projection according to the intrinsic parameter matrix, the lens distortion parameters and the extrinsic parameter matrix to obtain the three-dimensional coordinates of the center point of the laser line in the two-dimensional image captured by the line structured light 3D camera in the camera coordinate system; According to the three-dimensional coordinates of the center point of the laser line, the surface is fitted using the trained neural network model to obtain a surface model of the laser surface of the line structured light 3D camera in the camera coordinate system; Obtaining a ray corresponding to each pixel point according to the three-dimensional coordinates of each pixel point on the center contour of the laser line in the two-dimensional image captured by the line structured light 3D camera; According to the rays corresponding to each pixel point and the surface model of the line structured light 3D camera laser surface, the intersection points of all rays and the surface model are obtained to complete the three-dimensional reconstruction of the line structured light 3D camera laser surface.

2. The nonlinear calibration method for the laser surface of a line structured light 3D camera according to claim 1, characterized in that: The camera parameters are calibrated using Zhang Zhengyou calibration method to obtain the calibrated intrinsic parameter matrix, lens distortion parameters, and extrinsic parameter matrix; Optimize the calibration internal and external parameter matrices using maximum likelihood estimation; The internal parameter matrix and external parameter matrix optimized by maximum likelihood estimation are used as initial values, and further optimized using LM least squares method to obtain the final internal parameter matrix and external parameter matrix.

3. The nonlinear calibration method for the laser surface of a line structured light 3D camera according to claim 1, characterized in that: The back-projection is performed according to the intrinsic parameter matrix, the lens distortion parameter and the extrinsic parameter matrix to obtain the three-dimensional coordinates of the center point of the laser line in the two-dimensional image captured by the line structured light 3D camera in the camera coordinate system, including: A flat plate larger than the camera field of view is fixed on the slide rail of the line structured light 3D camera, wherein the angle between the plane of the flat plate and the projection direction of the laser of the line structured light 3D camera is in the range of 85° to 95°; Set N sampling points with equal distances within the depth of field of the camera, collect two images at each sampling point, and obtain N groups of sampling images, wherein one sampling image is taken by turning off the laser and attaching the chess calibration plate to the flat plate, and the other sampling image is taken by turning on the laser and removing the calibration plate; According to the intrinsic parameter matrix, the lens distortion parameters and the preset calibration plate parameters, the position and posture of the calibration plate in the camera coordinate system are obtained; Move the calibration plate posture along the negative direction of the Z axis of the calibration plate coordinate system by the thickness of the calibration plate to obtain the posture of the flat plate plane; The Steger algorithm is used to extract the coordinates of the center point of the laser line in N groups of sampled images. The three-dimensional coordinates of the center point of the laser line in the camera coordinate system are calculated by combining the coordinates of the center point with the posture of the flat plane to obtain a three-dimensional coordinate point set containing N coordinate values.

4. The nonlinear calibration method for a line structured light 3D camera laser surface according to claim 1, characterized in that: The three-dimensional coordinates of the center point of the laser line are normalized, and features are extracted from the normalized three-dimensional coordinates using the trained PointNet network. Surface fitting is performed based on the feature information to obtain a surface model of the laser surface of the line structured light 3D camera in the camera coordinate system.

5. The nonlinear calibration method for the laser surface of a line structured light 3D camera according to claim 1, characterized in that: The step of obtaining the ray corresponding to each pixel point according to the three-dimensional coordinates of each pixel point on the center contour of the laser line in the two-dimensional image captured by the line structured light 3D camera comprises: Obtain a two-dimensional image taken by a line structured light 3D camera; Using the lens distortion parameters, the two-dimensional coordinates of each pixel point on the central contour of the laser line in the two-dimensional image are corrected to obtain the corrected two-dimensional coordinates; The corrected two-dimensional coordinates are converted into three-dimensional coordinates in the camera coordinate system using the intrinsic parameter matrix to obtain the three-dimensional coordinates of each pixel point on the center contour of the laser line in the two-dimensional image, where the conversion formula is: ； ； ； in, is the three-dimensional coordinate of the pixel point on the center contour of the laser line in the two-dimensional image, ( , ) is the principal point coordinate in the camera intrinsic parameter matrix, is the corrected two-dimensional coordinate of the pixel point on the center contour of the laser line in the two-dimensional image, , They are the focal lengths in the X-axis and Y-axis directions in the camera intrinsic parameter matrix respectively; For each pixel point, the ray corresponding to the pixel point is obtained by pointing to the three-dimensional coordinates of the pixel point in the camera coordinate system from the origin of the line structured light 3D camera.

6. The nonlinear calibration method for a laser surface of a line structured light 3D camera according to claim 1, characterized in that: According to the surface model of the laser surface of the line structured light 3D camera corresponding to each pixel point, an optimization problem of minimizing the distance between the point on each ray and the laser surface is constructed. By solving the optimization problem, the three-dimensional coordinates of the intersection of each ray and the surface model are obtained, and then the laser surface is obtained.

7. A nonlinear calibration device for a laser surface of a line structured light 3D camera, characterized in that: include: The camera parameter calibration module is used to obtain the intrinsic parameter matrix, lens distortion parameters and extrinsic parameter matrix of the line structured light 3D camera through camera parameter calibration; A reverse projection module, used to perform reverse projection according to the intrinsic parameter matrix, the lens distortion parameters and the extrinsic parameter matrix to obtain the three-dimensional coordinates of the center point of the laser line in the two-dimensional image taken by the line structured light 3D camera in the camera coordinate system; A surface fitting module is used to fit the surface according to the three-dimensional coordinates of the center point of the laser line using a trained neural network model to obtain a surface model of the laser surface of the line structured light 3D camera in the camera coordinate system; A ray generation module, used to obtain a ray corresponding to each pixel point according to the three-dimensional coordinates of each pixel point on the center contour of the laser line in the two-dimensional image captured by the line structured light 3D camera; The laser surface calibration module is used to obtain the intersection points of all rays and the surface model according to the rays corresponding to each pixel point and the surface model of the line structured light 3D camera laser surface, and complete the three-dimensional reconstruction of the line structured light 3D camera laser surface.

8. The nonlinear calibration device for a laser surface of a line structured light 3D camera according to claim 7, characterized in that: The reverse projection module is specifically used for: A flat plate larger than the camera field of view is fixed on the slide rail of the line structured light 3D camera, wherein the angle between the plane of the flat plate and the projection direction of the laser of the line structured light 3D camera is in the range of 85° to 95°; Set N sampling points with equal distances within the depth of field of the camera, collect two images at each sampling point, and obtain N groups of sampling images, wherein one sampling image is taken by turning off the laser and attaching the chess calibration plate to the flat plate, and the other sampling image is taken by turning on the laser and removing the calibration plate; According to the intrinsic parameter matrix, the lens distortion parameters and the preset calibration plate parameters, the position and posture of the calibration plate in the camera coordinate system are obtained; Move the calibration plate posture along the negative direction of the Z axis of the calibration plate coordinate system by the thickness of the calibration plate to obtain the posture of the flat plate plane; The Steger algorithm is used to extract the coordinates of the center point of the laser line in N groups of sampled images. The three-dimensional coordinates of the center point of the laser line in the camera coordinate system are calculated by combining the coordinates of the center point with the posture of the flat plane to obtain a three-dimensional coordinate point set containing N coordinate values.

9. The nonlinear calibration device for a laser surface of a line structured light 3D camera according to claim 7, characterized in that: The ray generation module is specifically used for: Obtain a two-dimensional image taken by a line structured light 3D camera; Using the lens distortion parameters, the two-dimensional coordinates of each pixel point on the central contour of the laser line in the two-dimensional image are corrected to obtain the corrected two-dimensional coordinates; The corrected two-dimensional coordinates are converted into three-dimensional coordinates in the camera coordinate system using the intrinsic parameter matrix to obtain the three-dimensional coordinates of each pixel point on the center contour of the laser line in the two-dimensional image, where the conversion formula is: ； ； ； in, is the three-dimensional coordinate of the pixel point on the center contour of the laser line in the two-dimensional image, ( , ) is the principal point coordinate in the camera intrinsic parameter matrix, is the corrected two-dimensional coordinate of the pixel point on the center contour of the laser line in the two-dimensional image, , They are the focal lengths in the X-axis and Y-axis directions in the camera intrinsic parameter matrix respectively; For each pixel point, the ray corresponding to the pixel point is obtained by pointing to the three-dimensional coordinates of the pixel point in the camera coordinate system from the origin of the line structured light 3D camera.

10. The nonlinear calibration device for a laser surface of a line structured light 3D camera according to claim 7, characterized in that: The laser surface calibration module is specifically used for: According to the surface model of the laser surface of the line structured light 3D camera corresponding to each pixel point, an optimization problem of minimizing the distance between the point on each ray and the laser surface is constructed. By solving the optimization problem, the three-dimensional coordinates of the intersection of each ray and the surface model are obtained, and then the laser surface is obtained.

Images