Three-dimensional reconstruction method and device based on line laser galvanometer scanning and four-camera array

Abstract

The invention relates to a three-dimensional reconstruction method and device based on line laser galvanometer scanning and a four-camera array, and the method employs the four-camera array for image collection, and is more complete in visual field compared with a binocular system. The mapping relation between the two-dimensional image coordinates of each camera and the three-dimensional coordinates of the galvanometer can be directly obtained, and multi-view reconstruction is decomposed into multi-monocular reconstruction, so that the stereo matching process of a traditional passive multi-view three-dimensional reconstruction method is avoided; the problems of object detail loss and reconstruction dead angles caused by the fact that partial areas cannot be observed by two cameras at the same time are solved; meanwhile, a large amount of calculation processes required by three-dimensional reconstruction are put into system calibration of an offline part, so that the calculation amount of an algorithm of an online part is reduced, and the reconstruction speed is increased; in addition, line laser is adopted as a light source, compared with a DLP projection device, the cost is lower, the precision is higher, and the problem that the reconstruction effect of passive three-dimensional reconstruction in a weak texture area is poor is solved in the mode that the laser is actively projected to the surface of an object.

Description

Technical Field

[0001] The present invention relates to the technical field related to machine vision, and in particular to a three-dimensional reconstruction method and device based on linear laser galvanometer scanning and a four-camera array. Background Art

[0002] With the continuous advancement of intelligent manufacturing, manual work is being replaced by robot work. Robots often need to perceive the work scene in three dimensions during their work. Three-dimensional reconstruction of robot work scenes through machine vision technology is an important part of robot work. In recent years, with the continuous expansion of robot work scenes, three-dimensional reconstruction technology for large industrial scenes has attracted the attention of more and more scholars and technicians.

[0003] Three-dimensional reconstruction in industry is a technology that uses various sensors and algorithms to obtain three-dimensional spatial information on the surface of an object, which is often reflected in the reconstruction and generation of point clouds on the surface of the object. The commonly used three-dimensional reconstruction methods in industry are mainly monocular line laser scanning reconstruction, binocular coded structured light reconstruction, binocular line laser galvanometer scanning reconstruction and multi-eye passive vision reconstruction. Among them, the monocular line laser scanning reconstruction method uses the principle of triangulation to complete three-dimensional measurement by calibrating the relationship between the fixed camera and the laser plane; due to its reliance on a precise mechanical scanning platform and a limited monocular field of view, this method has a small scanning range and a large imaging blind spot. The binocular coded structured light reconstruction method uses a DLP projector to project modulated coded surface structured light, and then decodes the collected structured light image through a computer to complete the three-dimensional measurement; but due to the low illumination brightness and poor resolution of the projector, this method is easily interfered by ambient light and performs poorly in terms of accuracy, efficiency, and stability. The binocular line laser galvanometer scanning reconstruction method uses a laser galvanometer to reflect line lasers at different angles to complete scanning, replacing the bulky mechanical scanning platform and having a larger scanning range; the application of binocular vision makes this method have a more comprehensive perspective compared with monocular line laser scanning, reducing the blind spots of imaging; but since this method mainly uses line laser feature point matching and epipolar constraint stereo matching to complete 3D reconstruction, some positions that cannot be observed in two cameras at the same time will become the blind spots of reconstruction, resulting in the loss of object details. Multi-eye passive vision reconstruction uses more cameras to shoot the scene at different perspectives, and reconstructs the object through feature point stereo matching, further reducing the blind spots of imaging, but it also has problems such as long stereo matching time, low accuracy, and poor reconstruction effect in weak texture areas. Summary of the invention

[0004] The purpose of the present invention is to solve at least one of the deficiencies of the prior art and to provide a three-dimensional reconstruction method and device based on linear laser galvanometer scanning and a four-camera array.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] Specifically, a three-dimensional reconstruction method based on line laser galvanometer scanning and a four-camera array is proposed, including the following:

[0007] Four CMOS cameras are arranged in an array in advance to collect laser stripe images within the scanning range from four different angles respectively, and the matching galvanometer and line laser are set at the center of the four CMOS cameras, and calibration is completed to establish a mapping relationship between the two-dimensional pixel coordinates of the four CMOS cameras and the three-dimensional coordinates of the galvanometer;

[0008] Step 110, projecting a single line laser of an initial angle to the measured scene through a line laser, and driving four CMOS cameras to synchronously collect laser stripe images in the measured scene;

[0009] Step 120, extracting the laser stripe center in the laser stripe image;

[0010] Step 130, calculating and obtaining the three-dimensional point cloud data of a single line laser at four viewing angles based on the mapping relationship between the two-dimensional pixel coordinates of the four CMOS cameras and the three-dimensional coordinates of the galvanometer;

[0011] Step 140, fusing the three-dimensional point cloud data under four viewing angles to reconstruct the fringe point cloud of this part;

[0012] Step 150, driving the laser galvanometer to continuously change the projection angle of the line laser, and repeating steps 110 to 140 to reconstruct the stripe point cloud of each part of the measured scene;

[0013] Step 160: splice and reconstruct the stripe point clouds of each part to obtain a complete point cloud of the object.

[0014] Further, specifically, completing the calibration to establish a mapping relationship between the two-dimensional pixel coordinates of the four CMOS cameras and the three-dimensional coordinates of the galvanometer includes:

[0015] S101, driving four cameras to synchronously collect images of the checkerboard calibration plate in different postures in the field of view, and collecting line laser stripe images of the calibration plate at several different projection angles in the first three postures through a line laser and a laser galvanometer;

[0016] S102, using the collected laser stripe images of the calibration plate in different postures, the Zhang calibration method is used to calculate the internal parameters of the four cameras and the external parameters corresponding to the calibration plate in each posture;

[0017] S103, using the internal and external parameters of each camera obtained by camera calibration and the collected coplanar laser stripe images to fit the plane equations of several laser planes in four camera coordinate systems;

[0018] S104, based on the obtained plane equation and its corresponding plane normal vector, solve the transformation relationship between the four camera coordinate systems and the galvanometer coordinate system;

[0019] S105 , establishing a mapping relationship between the two-dimensional pixel coordinates of the four cameras and the three-dimensional coordinates of the galvanometer by combining the transformation relationship between the four camera coordinate systems and the galvanometer coordinate system and the light plane constraint equation related to the galvanometer rotation angle.

[0020] Further, specifically, in step S103, the plane equations of several laser planes in four camera coordinate systems are obtained by fitting the internal and external parameters of each camera obtained by camera calibration and several coplanar laser stripe images collected, including:

[0021] S201: Divide the collected calibration plate line laser stripe image into a plurality of groups of coplanar laser stripe images;

[0022] The laser stripe images on the three attitude calibration plates corresponding to the same galvanometer angle are divided into one group and regarded as coplanar laser stripe images, and the number of groups of the final images is the same as the number of projection angles in step S101;

[0023] S202: extracting the center of the laser stripe in each image based on the grayscale centroid method;

[0024] S203: using the camera internal and external parameters obtained by camera calibration, calculating the three-dimensional coordinates of the center of the laser stripe in each camera coordinate system;

[0025] S204: Based on the three-dimensional coordinates of the center of the coplanar laser stripes in each camera coordinate system, plane equations of several laser planes in each camera coordinate system are obtained by fitting using the least square method.

[0026] Further, specifically, in step S104, based on the obtained plane equation and its corresponding plane normal vector, the transformation relationship between the four camera coordinate systems and the galvanometer coordinate system is solved, including:

[0027] S301: Calculate the unit direction vector of each coordinate axis of the galvanometer coordinate system in each camera coordinate system according to the geometric relationship between the unit normal vector of the laser plane and each coordinate axis of the galvanometer coordinate system;

[0028] S302: Using the fitted laser plane equation, solve Y by the least squares method G The point P on the axis with the shortest distance to all light planes i (x 0i ,y 0i ,z 0i ), Y G The axis is the axis in the galvanometer coordinate system that coincides with the galvanometer motor rotation axis;

[0029] S303: According to Y G Axis unit direction vector and P i Click Create Y G The axis equation is used to obtain the coordinates O of the origin of the galvanometer coordinate system in each camera coordinate system. Gi (x Gi ,y Gi ,z Gi );

[0030] S304: Based on the obtained unit direction vectors of each coordinate axis of the galvanometer coordinate system in each camera coordinate system and O Gi Establish the transformation relationship between each camera coordinate system and the galvanometer coordinate system.

[0031] Further, specifically, in step 140, the three-dimensional point cloud data under four viewing angles are fused to reconstruct the fringe point cloud of the part, including:

[0032] S401: point cloud preprocessing to remove noise;

[0033] S402: Setting a reference point cloud according to the point cloud position, and solving the transformation from each point cloud to the reference point cloud using the ICP point cloud registration algorithm;

[0034] S403: transform each point cloud to the reference point cloud position by using the point cloud transformation relationship and splice them.

[0035] Furthermore, specifically, in step S401, outliers and noise in the point cloud are removed by median filtering.

[0036] Further, specifically, in step S403, the point cloud transformation is completed by the following formula:

[0037]

[0038] Where p s and p t They are the point clouds before and after the transformation, and Solve the ICP point cloud registration algorithm to obtain the transformation relationship from other view point clouds to the reference view point cloud, where i is the index of the point cloud to be transformed and j is the index of the reference point cloud.

[0039] The present invention also proposes a three-dimensional reconstruction device based on line laser galvanometer scanning and a four-camera array, comprising the following:

[0040] Support frame;

[0041] Four CMOS cameras are arranged in an array on the top of the support frame to uniformly collect images within the scanning range from four different angles;

[0042] The galvanometer is set at the center of the array of four CMOS cameras;

[0043] A line laser is placed horizontally, with its axis perpendicular to and intersecting with the motor shaft of the galvanometer;

[0044] A control module is used to control the movement of the galvanometer and the synchronization between the CMOS camera and the galvanometer;

[0045] Image acquisition module, used to drive four CMOS cameras for synchronous image acquisition and set camera parameters;

[0046] System calibration module, used to complete system calibration using the collected calibration images, and set and save system parameters;

[0047] The 3D reconstruction module is used to control the overall process of 3D reconstruction of objects, and to display and save the reconstructed point cloud results.

[0048] The beneficial effects of the present invention are:

[0049] The present invention proposes a 3D reconstruction method and device based on line laser galvanometer scanning and four-camera array. The method adopts a four-camera array for image acquisition and has a more complete field of view compared with a binocular system. The system calibration method proposed in the present invention can directly obtain the mapping relationship between the two-dimensional image coordinates of each camera and the three-dimensional coordinates of the galvanometer, so as to decompose the multi-eye reconstruction into multiple monocular reconstructions, thereby avoiding the stereo matching process of the traditional passive multi-eye 3D reconstruction method, and solving the problem of object detail loss and reconstruction blind spot caused by the fact that some areas cannot be observed by two cameras at the same time. At the same time, the method proposed in the present invention puts a large number of calculation processes required for 3D reconstruction into the system calibration of the offline part, reduces the calculation amount of the online part algorithm, and improves the reconstruction speed. In addition, the method adopts line laser as the light source, which has lower cost and higher precision than the DLP projection device, and solves the problem of poor reconstruction effect of passive 3D reconstruction in weak texture areas by actively projecting laser to the surface of the object. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The above and other features of the present disclosure will become more obvious by describing in detail the embodiments shown in the accompanying drawings. The same reference numerals in the accompanying drawings of the present disclosure represent the same or similar elements. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure. For those skilled in the art, other accompanying drawings can be obtained based on these accompanying drawings without creative work. In the accompanying drawings:

[0051] Figure 1 The figure shows the camera arrangement diagram of the three-dimensional reconstruction device based on line laser galvanometer scanning and four-camera array of the present invention;

[0052] Figure 2It is a flow chart of the three-dimensional reconstruction method based on line laser galvanometer scanning and four-camera array of the present invention;

[0053] Figure 3 Shown is a schematic diagram of the system calibration principle in the present invention;

[0054] Figure 4 Shown is a flow chart of step S103 in the present invention;

[0055] Figure 5 Shown is a flow chart of step S104 in the present invention;

[0056] Figure 6 It is a schematic diagram showing the geometric position relationship between each CMOS camera coordinate system and the galvanometer coordinate system in the present invention;

[0057] Figure 7 Shown is a schematic diagram of the object point cloud reconstruction principle in the present invention;

[0058] Figure 8 Shown is a flow chart of step S109 in the present invention;

[0059] Fig. 9 Shown is a block diagram of the software architecture of a three-dimensional reconstruction device based on line laser galvanometer scanning and a four-camera array. DETAILED DESCRIPTION

[0060] The following will be combined with the embodiments and drawings to clearly and completely describe the concept, specific structure and technical effects of the present invention, so as to fully understand the purpose, scheme and effect of the present invention. It should be noted that the embodiments in this application and the features in the embodiments can be combined with each other without conflict. The same reference numerals used throughout the drawings indicate the same or similar parts.

[0061] Example 1, reference Figure 2 The present invention proposes a three-dimensional reconstruction method based on line laser galvanometer scanning and a four-camera array, including the following:

[0062] Four CMOS cameras are arranged in an array in advance to collect laser stripe images within the scanning range from four different angles respectively, and the matching galvanometer and line laser are set at the center of the four CMOS cameras, and calibration is completed to establish a mapping relationship between the two-dimensional pixel coordinates of the four CMOS cameras and the three-dimensional coordinates of the galvanometer;

[0063] Step 110, projecting a single line laser of an initial angle to the measured scene through a line laser, and driving four CMOS cameras to synchronously collect laser stripe images in the measured scene;

[0064] Step 120, extracting the laser stripe center in the laser stripe image;

[0065] Step 130, calculating and obtaining the three-dimensional point cloud data of a single line laser at four viewing angles based on the mapping relationship between the two-dimensional pixel coordinates of the four CMOS cameras and the three-dimensional coordinates of the galvanometer;

[0066] Step 140, fusing the three-dimensional point cloud data under four viewing angles to reconstruct the fringe point cloud of this part;

[0067] Step 150, driving the laser galvanometer to continuously change the projection angle of the line laser, and repeating steps 110 to 140 to reconstruct the stripe point cloud of each part of the measured scene;

[0068] Step 160: splice and reconstruct the stripe point clouds of each part to obtain a complete point cloud of the object.

[0069] As a preferred embodiment of the present invention, specifically, completing the calibration to establish a mapping relationship between the two-dimensional pixel coordinates of the four CMOS cameras and the three-dimensional coordinates of the galvanometer includes:

[0070] S101, driving four cameras to synchronously collect images of the checkerboard calibration plate in different postures in the field of view, and collecting line laser stripe images of the calibration plate at several different projection angles in the first three postures through a line laser and a laser galvanometer;

[0071] S102, using the collected laser stripe images of the calibration plate in different postures, the Zhang calibration method is used to calculate the internal parameters of the four cameras and the external parameters corresponding to the calibration plate in each posture;

[0072] S103, using the internal and external parameters of each camera obtained by camera calibration and the collected coplanar laser stripe images to fit the plane equations of several laser planes in four camera coordinate systems;

[0073] S104, based on the obtained plane equation and its corresponding plane normal vector, solve the transformation relationship between the four camera coordinate systems and the galvanometer coordinate system;

[0074] S105 , establishing a mapping relationship between the two-dimensional pixel coordinates of the four cameras and the three-dimensional coordinates of the galvanometer by combining the transformation relationship between the four camera coordinate systems and the galvanometer coordinate system and the light plane constraint equation related to the galvanometer rotation angle.

[0075] As a preferred embodiment of the present invention, specifically, in step S103, the plane equations of several laser planes in four camera coordinate systems are obtained by fitting the internal and external parameters of each camera obtained by camera calibration and several coplanar laser stripe images collected, including:

[0076] S201: Divide the collected calibration plate line laser stripe image into a plurality of groups of coplanar laser stripe images;

[0077] The laser stripe images on the three attitude calibration plates corresponding to the same galvanometer angle are divided into one group and regarded as coplanar laser stripe images, and the number of groups of the final images is the same as the number of projection angles in step S101;

[0078] S202: extracting the center of the laser stripe in each image based on the grayscale centroid method;

[0079] S203: using the camera internal and external parameters obtained by camera calibration, calculating the three-dimensional coordinates of the center of the laser stripe in each camera coordinate system;

[0080] S204: Based on the three-dimensional coordinates of the center of the coplanar laser stripes in each camera coordinate system, plane equations of several laser planes in each camera coordinate system are obtained by fitting using the least square method.

[0081] As a preferred embodiment of the present invention, specifically, in step S104, based on the obtained plane equation and its corresponding plane normal vector, the transformation relationship between the four camera coordinate systems and the galvanometer coordinate system is solved, including:

[0082] S301: Calculate the unit direction vector of each coordinate axis of the galvanometer coordinate system in each camera coordinate system according to the geometric relationship between the unit normal vector of the laser plane and each coordinate axis of the galvanometer coordinate system;

[0083] S302: Using the fitted laser plane equation, solve Y by the least squares method G The point P on the axis with the shortest distance to all light planes i (x 0i ,y 0i ,z 0i ), Y G The axis is the axis in the galvanometer coordinate system that coincides with the galvanometer motor rotation axis;

[0084] S303: According to Y G Axis unit direction vector and P i Click Create Y G The axis equation is used to obtain the coordinates O of the origin of the galvanometer coordinate system in each camera coordinate system. Gi (x Gi ,y Gi ,z Gi );

[0085] S304: Based on the obtained unit direction vectors of each coordinate axis of the galvanometer coordinate system in each camera coordinate system and O Gi Establish the transformation relationship between each camera coordinate system and the galvanometer coordinate system.

[0086] As a preferred embodiment of the present invention, specifically, in step 140, the three-dimensional point cloud data under four viewing angles are fused to reconstruct the partial fringe point cloud, including:

[0087] S401: point cloud preprocessing to remove noise;

[0088] S402: Setting a reference point cloud according to the point cloud position, and solving the transformation from each point cloud to the reference point cloud using the ICP point cloud registration algorithm;

[0089] S403: transform each point cloud to the reference point cloud position by using the point cloud transformation relationship and splice them.

[0090] As a preferred implementation of the present invention, specifically, in step S401, outliers and noise in the point cloud are removed by median filtering.

[0091] As a preferred implementation of the present invention, specifically, in step S403, point cloud transformation is completed by the following formula:

[0092]

[0093] Where p s and p t They are the point clouds before and after the transformation, and Solve the ICP point cloud registration algorithm to obtain the transformation relationship from other view point clouds to the reference view point cloud, where i is the index of the point cloud to be transformed and j is the index of the reference point cloud.

[0094] A preferred implementation of the overall process of the present invention is introduced as follows:

[0095] like Figure 1 The figure shows a schematic diagram of the hardware architecture of a three-dimensional reconstruction system based on line laser galvanometer scanning and a four-camera array provided by an embodiment of the present invention, which is mainly composed of four cameras, a line laser, a laser galvanometer, a support frame and several connectors. The four cameras are arranged in an array to evenly collect images within the scanning range from four different angles; the galvanometer is located at the center of the camera array, and the line laser is placed horizontally, with its axis perpendicular to and intersecting with the galvanometer motor shaft. This schematic diagram only expresses the relative position design between the various hardware of the system, and its specific layout can be designed separately.

[0096] Figure 2 A schematic flow chart of a three-dimensional reconstruction method based on linear laser galvanometer scanning and a four-camera array provided in an embodiment of the present invention, the method comprising steps S101 to S111.

[0097] (I) System calibration steps:

[0098] The basic principle of system calibration is as follows Figure 3 As shown, O C1 -X C1 Y C1 Z C1, O C2 -X C2 Y C2 Z C2 , O C3 -X C3 Y C3 Z C3 and O C4 -X C4 Y C4 Z C4 are the camera coordinate systems of camera 1, camera 2, camera 3 and camera 4 respectively; G -X G Y G Z G is the galvanometer coordinate system. G The axis coincides with the galvanometer motor axis, Z G The axis is located on the laser plane when the galvanometer angle is 0°, and the coordinate origin O G Located at the center of the camera array, X G The axis is determined by the right-hand rule.

[0099] S101: driving four cameras to synchronously collect images of the checkerboard calibration plate in different postures in the field of view, and collecting line laser stripe images of the calibration plate at several different projection angles in the first three postures through a line laser and a laser galvanometer;

[0100] The main implementation methods of this step are as follows Figure 3 As shown in the left part, the number of line laser projection angles should be greater than 4.

[0101] S102: using the collected images of the calibration plate in different postures, the Zhang calibration method is used to calculate the intrinsic parameters of the four cameras and the extrinsic parameters corresponding to the calibration plate in each posture;

[0102] S103: using the internal and external parameters of each camera obtained by camera calibration and the collected coplanar laser stripe images to fit the plane equations of several laser planes in four camera coordinate systems;

[0103] Specifically, Figure 4 As shown, step S103 includes:

[0104] S201: Divide the collected calibration plate line laser stripe image into a plurality of groups of coplanar laser stripe images;

[0105] The laser stripe images on the three attitude calibration plates corresponding to the same galvanometer angle are divided into one group and regarded as coplanar laser stripe images, and the number of final image groups is the same as the number of projection angles in step S101.

[0106] S202: extracting the center of the laser stripe in each image;

[0107] This step can be performed according to the grayscale centroid method as follows:

[0108] For the laser stripe images captured by camera 1 and camera 3, the stripes are horizontal, so the pixels in each column with grayscale values ​​greater than a certain threshold in the image are substituted into the following formula for calculation:

[0109]

[0110] Where f(i,j) is the gray value of the pixel in the i-th row and j-th column of the image, (u j ,v j ) is the grayscale centroid of the column, which is regarded as the center point of the laser light bar on the column.

[0111] For the laser stripe images captured by camera 2 and camera 4, the stripes are longitudinal, so the pixels in each row with grayscale values ​​greater than a certain threshold in the image are substituted into the following formula:

[0112]

[0113] Where (u i ,v i ) is the grayscale centroid of the row, which is regarded as the center point of the laser light bar on the row.

[0114] S203: using the camera internal and external parameters obtained by camera calibration, calculating the three-dimensional coordinates of the center of the laser stripe in each camera coordinate system;

[0115] In the pixel coordinate system Ou of any camera i (i=1,2,3,4) i v i In the figure, assume that P(u,v) is an arbitrary point at the center of the laser stripe on the calibration plate, which corresponds to the camera coordinate system O Ci -X Ci Y Ci Z Ci The coordinate value in (X C ,Y C ,Z C ), in the world coordinate system O W -X W Y W Z W The coordinate value in (X W ,Y W ,Z W ). According to the definition of the world coordinate system on the calibration plate, Z W = 0, and substituting it into the pinhole imaging model of the camera, we can get:

[0116]

[0117] Where M Ii and MWi are the intrinsic and extrinsic matrix of camera i obtained by camera calibration. The transformation matrix M between the world coordinate system and the pixel coordinate system IWi It can be expressed as:

[0118]

[0119] Furthermore, we can obtain the following solution for the world coordinates of the stripe center (X W ,Y W ,0) equation:

[0120]

[0121] According to the relationship between the world coordinate system and the camera coordinate system, the coordinates of the fringe center in the camera coordinate system (X C ,Y C ,Z C ) can be solved by the following formula:

[0122]

[0123] S204: Based on the three-dimensional coordinates of the center of the coplanar laser stripes in each camera coordinate system, the plane equations of several laser planes in each camera coordinate system are obtained by fitting using the least square method;

[0124] This step uses a series of coplanar laser stripe center sample points (X Ck ,Y Ck ,Z Ck ), k = 0, 1, ..., n-1 (n ≥ 3), and the plane equations of several laser planes in each camera coordinate system are obtained by least squares fitting.

[0125] In the camera coordinate system O C1 -X C1 Y C1 Z C1 With O C3 -X C3 Y C3 Z C3 Since the light plane is always not aligned with Y Ci The axes are parallel, and the plane equation obtained by fitting is expressed as:

[0126] a ij X C -Y C +c ij Z C +d ij =0

[0127] In the camera coordinate system O C2 -X C2 YC2 Z C2 With O C4 -X C4 Y C4 Z C4 Since the light plane is always not aligned with X Ci The axes are parallel, and the plane equation obtained by fitting is expressed as:

[0128] -X C +b ij Y C +c ij Z C +d ij =0

[0129] S104: Solving the transformation relationships between the four camera coordinate systems and the galvanometer coordinate system based on the obtained light plane equation and its corresponding plane normal vector;

[0130] Specifically, Figure 5 As shown, step S104 includes:

[0131] S301: Calculate the unit direction vector of each coordinate axis of the galvanometer coordinate system in each camera coordinate system according to the geometric relationship between the unit normal vector of the laser plane and each coordinate axis of the galvanometer coordinate system;

[0132] According to the laser plane equation obtained in step S103, the unit normal vector of the j-th light plane in the i-th camera coordinate system can be calculated.

[0133] According to Figure 6 The geometric position relationship between the camera coordinate system and the galvanometer coordinate system shown can be obtained:

[0134]

[0135] in, arrive X G Unit direction vector of the axis in the four camera coordinate systems arrive They are the unit normal vectors of the laser plane in each camera coordinate system when the galvanometer rotation angle is 0°.

[0136] Similarly, according to Figure 6 You can get:

[0137]

[0138] In the formula Y G Axis in the camera coordinate system O Ci -X Ci Y Ci ZCi According to this formula, using several laser plane unit normal vector samples, the optimal solution can be obtained by the least squares method.

[0139] In seeking and Finally, Z can be determined based on the perpendicular relationship between the three coordinate axes of the galvanometer coordinate system. G Axis in the camera coordinate system O Ci -X Ci Y Ci Z Ci The unit direction vector

[0140]

[0141] S302: Using the fitted laser plane equation, solve Y by the least squares method G The point P on the axis with the shortest distance to all light planes i (x 0i ,y 0i ,z 0i );

[0142] S303: According to Y G Axis unit direction vector and P i Click Create Y G The axis equation is used to obtain the coordinates O of the origin of the galvanometer coordinate system in each camera coordinate system. Gi (x Gi ,y Gi ,z Gi );

[0143] In obtaining P i After the coordinates of the camera coordinate system O Ci -X Ci Y Ci Z Ci Y G The axis equation can be expressed as:

[0144]

[0145] Because O G In plane O Ci -Y Ci Z Ci So x Gi = 0. Gi (0,y Gi ,z Gi )Substitute into Y G In the axis equation, we can find O Gi The coordinates are:

[0146]

[0147] S304: Based on the obtained unit direction vectors of each coordinate axis of the galvanometer coordinate system in each camera coordinate system and O Gi Establish the transformation relationship between each camera coordinate system and the galvanometer coordinate system;

[0148] According to the definition of the galvanometer coordinate system, the galvanometer coordinate system O G -X G Y G Z G Relative to the camera coordinate system O Ci -X Ci Y Ci Z Ci The rotation matrix R Gi and the translation vector T Gi They are:

[0149]

[0150] S105: establishing a mapping relationship between the two-dimensional pixel coordinates of the four cameras and the three-dimensional coordinates of the galvanometer by combining the transformation relationship between the camera coordinate system and the galvanometer coordinate system and the light plane constraint equation related to the galvanometer rotation angle;

[0151] Specifically, step S105 includes:

[0152] According to the pinhole imaging model of the camera, the mapping relationship between the galvanometer coordinate system and the camera i pixel coordinate system can be expressed as follows:

[0153]

[0154] Where M Ii is the intrinsic parameter matrix of camera i obtained by camera calibration, M Gi is the transformation relationship between the galvanometer coordinate system and the camera i coordinate system. The transformation matrix M between the galvanometer coordinate system and the camera i pixel coordinate system IGi It can be expressed as:

[0155]

[0156] Then the relationship between the pixel coordinate system and the galvanometer coordinate system can be further described by the following set of equations:

[0157]

[0158] Since the coefficient matrix of the system of equations is a matrix with row rank r = 3, the system of equations has infinite solutions. In order to obtain a unique solution, the following definition of the light plane related to the galvanometer rotation angle in the galvanometer coordinate system is introduced as an additional constraint of the system of equations:

[0159] XG cos2ω j -Z G sin2ω j =0

[0160] Where ω j is the current rotation angle of the galvanometer. Adding this constraint to the equation system, we can obtain the following solution for the three-dimensional coordinates (X G ,Y G ,Z G ) is the system of equations:

[0161]

[0162] Through this set of equations, we can know the center coordinates of the laser line (u, v) and the galvanometer rotation angle ω corresponding to the current light plane. j Under the condition of G ,Y G ,Z G ) to achieve 3D point cloud reconstruction of laser stripes. This process does not require stereo matching, and can effectively solve the problem of object detail loss, reconstruction blind spots and mismatching caused by the inability of two cameras to observe some areas at the same time, thereby improving reconstruction accuracy and speed.

[0163] (II) Point cloud reconstruction steps:

[0164] like Figure 7 As shown, this is the basic principle of object point cloud reconstruction.

[0165] S106: Projecting a single line laser of an initial angle to the measured scene through a line laser, and driving four cameras to synchronously collect laser stripe images in the scene;

[0166] S107: extracting the laser stripe center in the laser stripe image;

[0167] The extraction method of this step is the same as that of step S202.

[0168] S108: Calculate the three-dimensional point cloud data of a single laser at four viewing angles using the mapping relationship between the two-dimensional pixel coordinates of the four cameras obtained by system calibration and the three-dimensional coordinates of the galvanometer;

[0169] This step reconstructs a single laser using the mapping relationship obtained in step S105.

[0170] S109: Fusion of laser stripe point clouds from four perspectives;

[0171] like Figure 8 As shown, step S109 includes:

[0172] S401: point cloud preprocessing to remove noise;

[0173] This step uses median filtering to remove outliers and noise in the point cloud;

[0174] S402: Setting a reference point cloud according to the point cloud position, and solving the transformation from each point cloud to the reference point cloud using the ICP point cloud registration algorithm;

[0175] First, the reference view point cloud is determined according to the position of the point cloud in the field of view. The closer the laser stripes are to the camera, the higher the reconstruction accuracy of the stripes. When the X coordinate of the point cloud is less than or equal to 0, the view 1 point cloud is used as the reference; when the X coordinate of the point cloud is greater than 0, the view 3 point cloud is used as the reference.

[0176] Then, the ICP point cloud registration algorithm is used to solve the transformation relationship between other view point clouds and the reference view point cloud. and Where i is the index of the point cloud to be transformed, and j is the index of the reference point cloud.

[0177] S403: transforming each point cloud to a reference point cloud position by using a point cloud transformation relationship and splicing them;

[0178] Specifically, the point cloud transformation is completed using the following formula:

[0179]

[0180] Where p s and p t They are the point clouds before and after the transformation.

[0181] S110: driving the laser galvanometer to continuously change the projection angle of the line laser, and reconstructing the stripe point clouds of each part of the scene according to the above method;

[0182] S111: stitching and reconstructing the partial point clouds to obtain a complete point cloud of the object.

[0183] The present invention directly obtains the conversion relationship from the two-dimensional image collected by the camera to the three-dimensional coordinates of the object through the offline system calibration process, so that the system only needs to perform a small amount of calculation to complete the three-dimensional point cloud reconstruction of the object during the online point cloud reconstruction of the object, thereby improving the speed of three-dimensional reconstruction to a certain extent.

[0184] An embodiment of the present invention also provides a three-dimensional reconstruction system software architecture based on line laser galvanometer scanning and four-camera array, and the software architecture is used to execute any embodiment of the aforementioned three-dimensional reconstruction method based on line laser galvanometer scanning and four-camera array. Fig. 9 A schematic block diagram of the software architecture of a three-dimensional reconstruction system based on line laser galvanometer scanning and a four-camera array provided in an embodiment of the present invention.

[0185] like Fig. 9 As shown, the software architecture 900 includes: a control module 901 , an image acquisition module 902 , a system calibration module 903 and a three-dimensional reconstruction module 904 .

[0186] Reference Figure 1 The present invention also proposes a three-dimensional reconstruction device based on line laser galvanometer scanning and a four-camera array, comprising the following:

[0187] Support frame;

[0188] Four CMOS cameras are arranged in an array on the top of the support frame to uniformly collect images within the scanning range from four different angles;

[0189] The galvanometer is set at the center of the array of four CMOS cameras;

[0190] A line laser is placed horizontally, with its axis perpendicular to and intersecting with the motor shaft of the galvanometer;

[0191] The control module 901 is used to control the movement of the galvanometer and the synchronization between the CMOS camera and the galvanometer;

[0192] Image acquisition module 902, used to drive four CMOS cameras to perform synchronous image acquisition and set camera parameters;

[0193] The system calibration module 903 is used to complete the system calibration using the collected calibration images, and to set and save the system parameters;

[0194] The 3D reconstruction module 904 is used to control the overall process of 3D reconstruction of the object, and to display and save the reconstructed point cloud results.

[0195] In addition, each functional module in each embodiment of the present invention can be integrated into one processing module, each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.

[0196] If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can include: any entity or system that can carry the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0197] Although the description of the present invention has been quite detailed and specifically describes several described embodiments, it is not intended to be limited to any of these details or embodiments or any particular embodiment, but should be regarded as providing a broad possible interpretation of these claims in view of the prior art by reference to the appended claims, thereby effectively covering the intended scope of the present invention. In addition, the above description of the present invention is based on the embodiments foreseeable by the inventor, and its purpose is to provide a useful description, and those non-substantial changes to the present invention that have not yet been foreseen may still represent equivalent changes to the present invention.

[0198] The above is only a preferred embodiment of the present invention. The present invention is not limited to the above implementation. As long as the technical effect of the present invention is achieved by the same means, it should belong to the protection scope of the present invention. Within the protection scope of the present invention, its technical scheme and / or implementation method can have various modifications and changes.

Claims

1. A three-dimensional reconstruction method based on line laser galvanometer scanning and a four-camera array, characterized in that: Includes the following: Four CMOS cameras are arranged in an array in advance to collect laser stripe images within the scanning range from four different angles respectively, and the matching galvanometer and line laser are set at the center of the four CMOS cameras, and calibration is completed to establish a mapping relationship between the two-dimensional pixel coordinates of the four CMOS cameras and the three-dimensional coordinates of the galvanometer; Step 110, projecting a single line laser of an initial angle to the measured scene through a line laser, and driving four CMOS cameras to synchronously collect laser stripe images in the measured scene; Step 120, extracting the laser stripe center in the laser stripe image; Step 130, calculating and obtaining the three-dimensional point cloud data of a single line laser at four viewing angles based on the mapping relationship between the two-dimensional pixel coordinates of the four CMOS cameras and the three-dimensional coordinates of the galvanometer; Step 140, fusing the three-dimensional point cloud data under four viewing angles to reconstruct the fringe point cloud of this part; Step 150, driving the laser galvanometer to continuously change the projection angle of the line laser, and repeating steps 110 to 140 to reconstruct the stripe point cloud of each part of the measured scene; Step 160: splice and reconstruct the stripe point clouds of each part to obtain a complete point cloud of the object.

2. The three-dimensional reconstruction method based on line laser galvanometer scanning and four-camera array according to claim 1, characterized in that: Specifically, completing the calibration to establish a mapping relationship between the two-dimensional pixel coordinates of the four CMOS cameras and the three-dimensional coordinates of the galvanometer includes: S101, driving four cameras to synchronously collect images of the checkerboard calibration plate in different postures in the field of view, and collecting line laser stripe images of the calibration plate at several different projection angles in the first three postures through a line laser and a laser galvanometer; S102, using the collected laser stripe images of the calibration plate in different postures, the Zhang calibration method is used to calculate the internal parameters of the four cameras and the external parameters corresponding to the calibration plate in each posture; S103, using the internal and external parameters of each camera obtained by camera calibration and the collected coplanar laser stripe images to fit the plane equations of several laser planes in four camera coordinate systems; S104, based on the obtained plane equation and its corresponding plane normal vector, solve the transformation relationship between the four camera coordinate systems and the galvanometer coordinate system; S105 , establishing a mapping relationship between the two-dimensional pixel coordinates of the four cameras and the three-dimensional coordinates of the galvanometer by combining the transformation relationship between the four camera coordinate systems and the galvanometer coordinate system and the light plane constraint equation related to the galvanometer rotation angle.

3. The three-dimensional reconstruction method based on line laser galvanometer scanning and four-camera array according to claim 2, characterized in that: Specifically, in step S103, the plane equations of several laser planes in four camera coordinate systems are obtained by fitting the internal and external parameters of each camera obtained by camera calibration and several coplanar laser stripe images collected. include, S201: Divide the collected calibration plate line laser stripe image into a plurality of groups of coplanar laser stripe images; The laser stripe images on the three attitude calibration plates corresponding to the same galvanometer angle are divided into one group and regarded as coplanar laser stripe images, and the number of groups of the final images is the same as the number of projection angles in step S101; S202: extracting the center of the laser stripe in each image based on the grayscale centroid method; S203: using the camera internal and external parameters obtained by camera calibration, calculating the three-dimensional coordinates of the center of the laser stripe in each camera coordinate system; S204: Based on the three-dimensional coordinates of the center of the coplanar laser stripes in each camera coordinate system, plane equations of several laser planes in each camera coordinate system are obtained by fitting using the least square method.

4. The three-dimensional reconstruction method based on line laser galvanometer scanning and four-camera array according to claim 3 is characterized in that: Specifically, in step S104, based on the obtained plane equation and its corresponding plane normal vector, the transformation relationship between the four camera coordinate systems and the galvanometer coordinate system is solved, including: S301: Calculate the unit direction vector of each coordinate axis of the galvanometer coordinate system in each camera coordinate system according to the geometric relationship between the unit normal vector of the laser plane and each coordinate axis of the galvanometer coordinate system; S302: Using the fitted laser plane equation, solve Y by the least squares method G The point P on the axis with the shortest distance to all light planes i (x 0i ,y 0i ,z 0i ), Y G The axis is the axis in the galvanometer coordinate system that coincides with the galvanometer motor rotation axis; S303: According to Y G Axis unit direction vector and P i Click Create Y G The axis equation is used to obtain the coordinates O of the origin of the galvanometer coordinate system in each camera coordinate system. Gi (x Gi ,y Gi ,z Gi ); S304: Based on the obtained unit direction vectors of each coordinate axis of the galvanometer coordinate system in each camera coordinate system and O Gi Establish the transformation relationship between each camera coordinate system and the galvanometer coordinate system.

5. The three-dimensional reconstruction method based on line laser galvanometer scanning and four-camera array according to claim 1, characterized in that: Specifically, in step 140, the three-dimensional point cloud data under four viewing angles are fused to reconstruct the fringe point cloud of this part, including: S401: point cloud preprocessing to remove noise; S402: Setting a reference point cloud according to the point cloud position, and solving the transformation from each point cloud to the reference point cloud using the ICP point cloud registration algorithm; S403: transform each point cloud to the reference point cloud position by using the point cloud transformation relationship and splice them.

6. The three-dimensional reconstruction method based on line laser galvanometer scanning and four-camera array according to claim 5, characterized in that: Specifically, in step S401, outliers and noise in the point cloud are removed by median filtering.

7. The three-dimensional reconstruction method based on line laser galvanometer scanning and four-camera array according to claim 5, characterized in that: Specifically, in step S403, the point cloud transformation is completed by the following formula: Where p s and p t They are the point clouds before and after the transformation, and Solve the ICP point cloud registration algorithm to obtain the transformation relationship from other viewpoint point clouds to the reference viewpoint point cloud, where i is the index of the point cloud to be transformed and j is the index of the reference point cloud.

8. A three-dimensional reconstruction device based on line laser galvanometer scanning and a four-camera array, characterized in that: Includes the following: Support frame; Four CMOS cameras are arranged in an array on the top of the support frame to uniformly collect images within the scanning range from four different angles; The galvanometer is set at the center of the array of four CMOS cameras; A line laser is placed horizontally, with its axis perpendicular to and intersecting with the motor shaft of the galvanometer; A control module is used to control the movement of the galvanometer and the synchronization between the CMOS camera and the galvanometer; Image acquisition module, used to drive four CMOS cameras for synchronous image acquisition and set camera parameters; System calibration module, used to complete system calibration using the collected calibration images, and set and save system parameters; The 3D reconstruction module is used to control the overall process of 3D reconstruction of objects, and to display and save the reconstructed point cloud results.

Images