Three-dimensional reconstruction method and device based on line laser galvanometer scanning and four-camera array

Abstract

This invention relates to a 3D reconstruction method and apparatus based on line laser galvanometer scanning and a four-camera array. The four-camera array is used for image acquisition, providing a more complete field of view compared to a binocular system. Furthermore, it can directly obtain the mapping relationship between the 2D image coordinates of each camera and the 3D coordinates of the galvanometer, thus decomposing multi-view reconstruction into multiple monocular reconstructions. This avoids the stereo matching process of traditional passive multi-view 3D reconstruction methods and solves the problems of lost object details and reconstruction blind spots caused by some areas not being simultaneously observed by two cameras. Simultaneously, the large amount of computation required for 3D reconstruction is incorporated into the offline system calibration, reducing the computational load of the online algorithm and improving reconstruction speed. In addition, using a line laser as the light source is lower in cost and higher in accuracy compared to DLP projection devices, and the active projection of the laser onto the object surface solves the problem of poor reconstruction results in weakly textured areas in passive 3D reconstruction.

Description

Technical Field

[0001] This invention relates to the field of machine vision technology, and in particular to a three-dimensional reconstruction method and apparatus based on line laser galvanometer scanning and a four-camera array. Background Technology

[0002] With the continuous advancement of intelligent manufacturing, manual labor is increasingly being replaced by robotic operations. Robots often need to perceive the 3D working environment during their work, and 3D reconstruction of this environment using machine vision technology is a crucial part of robotic operations. In recent years, as the scenarios in which robots operate have expanded, 3D reconstruction technology for large-scale industrial environments has attracted increasing attention from scholars and engineers.

[0003] Industrial 3D reconstruction utilizes various sensors and algorithms to acquire three-dimensional spatial information of an object's surface, often manifested as the reconstruction and generation of point clouds on the object's surface. Commonly used 3D reconstruction methods in industry include monocular laser scanning reconstruction, binocular coded structured light reconstruction, binocular laser galvanometer scanning reconstruction, and multi-view passive vision reconstruction. Monocular laser scanning reconstruction utilizes the principle of triangulation, completing 3D measurement by calibrating a fixed camera and laser plane relationship; however, due to its reliance on a precise mechanical scanning platform and a limited monocular field of view, this method has a small scanning range and large blind spots. Binocular coded structured light reconstruction uses a DLP projector to project modulated coded structured light onto a surface, and then a computer decodes the acquired structured light image to complete the 3D measurement; however, due to the low brightness and poor resolution of the projector illumination, this method is easily affected by ambient light interference, resulting in poor performance in terms of accuracy, efficiency, and stability. Binocular line laser galvanometer scanning reconstruction uses laser galvanometers to reflect line laser light at different angles to complete the scan, replacing the bulky mechanical scanning platform and providing a larger scanning range. The application of binocular vision gives this method a more comprehensive field of view compared to monocular line laser scanning, reducing imaging blind spots. However, because this method mainly uses line laser feature point matching and epipolar constraint stereo matching to complete 3D reconstruction, some locations that cannot be observed simultaneously in both cameras become reconstruction blind spots, resulting in the loss of object details. Multi-view passive vision reconstruction uses more cameras to capture the scene from different perspectives and reconstructs objects through feature point stereo matching, further reducing imaging blind spots. However, it also suffers from problems such as long stereo matching time, low accuracy, and poor reconstruction results in weak texture areas. Summary of the Invention

[0004] The purpose of this invention is to at least address one of the shortcomings of the prior art by providing a three-dimensional reconstruction method and apparatus based on line laser galvanometer scanning and a four-camera array.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] Specifically, a 3D reconstruction method based on line laser galvanometer scanning and a four-camera array is proposed, including the following:

[0007] Four CMOS cameras are arranged in an array in advance to acquire laser stripe images within the scanning range from four different angles. A galvanometer and a line laser are set at the center of the four CMOS cameras and calibration is performed to establish the mapping relationship between the two-dimensional pixel coordinates of the four CMOS cameras and the three-dimensional coordinates of the galvanometer.

[0008] Step 110: Project a single line laser at an initial angle onto the scene under test using a line laser, and drive four CMOS cameras to simultaneously acquire laser stripe images of the scene under test.

[0009] Step 120: Extract the center of the laser stripe in the laser stripe image;

[0010] Step 130: Calculate the three-dimensional point cloud data of a single line laser under four viewpoints based on the mapping relationship between the two-dimensional pixel coordinates of the four CMOS cameras and the three-dimensional coordinates of the galvanometer.

[0011] Step 140: Reconstruct the striped point cloud by fusing the 3D point cloud data from four perspectives;

[0012] Step 150: Drive the laser galvanometer to continuously change the projection angle of the line laser, and repeat steps 110 to 140 to reconstruct the fringe point cloud of each part of the scene under test.

[0013] Step 160: Combine and reconstruct the striped point clouds of each part to obtain the complete point cloud of the object.

[0014] Furthermore, specifically, calibration is performed to establish the mapping relationship between the two-dimensional pixel coordinates of the four CMOS cameras and the three-dimensional coordinates of the galvanometer, including:

[0015] S101 drives four cameras to simultaneously acquire images of the checkerboard calibration board in different postures in the field of view, and acquires line laser stripe images of the calibration board at several different projection angles in the first three postures through a line laser and a laser galvanometer.

[0016] S102. Using the laser stripe images of the calibration board under different postures, calculate the intrinsic parameters of the four cameras and the extrinsic parameters of the calibration board under each posture using Zhang's calibration method.

[0017] S103. Using the intrinsic and extrinsic parameters of each camera obtained from camera calibration and several coplanar laser stripe images acquired, several plane equations of laser planes in the four camera coordinate systems are obtained by fitting.

[0018] S104. Based on the obtained plane equations and their corresponding plane normal vectors, the transformation relationships between the four camera coordinate systems and the galvanometer coordinate system are obtained respectively.

[0019] S105. By combining the transformation relationship between the four camera coordinate systems and the galvanometer coordinate system, and the light plane constraint equation related to the galvanometer rotation angle, establish the mapping relationship between the two-dimensional pixel coordinates of the four cameras and the three-dimensional coordinates of the galvanometer.

[0020] Furthermore, specifically, in step S103, the intrinsic and extrinsic parameters of each camera obtained from camera calibration and the acquired coplanar laser fringe images are used to fit several plane equations of the laser plane in the four camera coordinate systems, including...

[0021] S201: Divide the acquired calibration plate line laser stripe image into several groups of coplanar laser stripe images;

[0022] Among them, the laser stripe images on the three attitude calibration plates corresponding to the same galvanometer angle are divided into a group and regarded as coplanar laser stripe images. The final number of image groups is the same as the number of projection angles in step S101.

[0023] S202: Extract the center of the laser stripe in each image based on the gray-scale centroid method;

[0024] S203: Using the camera intrinsic and extrinsic parameters obtained from camera calibration, calculate the three-dimensional coordinates of the laser stripe center in each camera coordinate system;

[0025] S204: Based on the three-dimensional coordinates of the coplanar laser fringe center in each camera coordinate system, the plane equations of several laser planes in each camera coordinate system are obtained by fitting using the least squares method.

[0026] Furthermore, specifically, in step S104, based on the obtained plane equations and their corresponding plane normal vectors, the transformation relationships between the four camera coordinate systems and the galvanometer coordinate system are obtained, including:

[0027] S301: Calculate the unit direction vector of each coordinate axis of the galvanometer coordinate system in each camera coordinate system based on the geometric relationship between the unit normal vector of the laser plane and each coordinate axis of the galvanometer coordinate system;

[0028] S302: Using the fitted laser plane equation, solve for Y using the least squares method. G The point P on the axis that has the shortest distance to all light planes i (x 0i ,y 0i ,z 0i ), Y G The axis is the axis in the galvanometer coordinate system that coincides with the galvanometer motor's rotating shaft;

[0029] S303: According to Y G Axial unit direction vector and P i Point to establish Y G The axis equations were calculated, and the coordinates O of the origin of the galvanometer coordinate system in each camera coordinate system were obtained. Gi (x Gi ,y Gi ,z Gi );

[0030] S304: Based on the obtained unit direction vectors of each coordinate axis of the galvanometer coordinate system in each camera coordinate system and O Gi Establish the transformation relationship between each camera coordinate system and the galvanometer coordinate system.

[0031] Furthermore, specifically, in step 140, the 3D point cloud data from four perspectives is fused to reconstruct this part of the striped point cloud, including,

[0032] S401: Point cloud preprocessing to remove noise;

[0033] S402: Set a reference point cloud based on the point cloud location, and use the ICP point cloud registration algorithm to solve the transformation of each point cloud to the reference point cloud;

[0034] S403: Transform each point cloud to the reference point cloud position using point cloud transformation relationships and then stitch them together.

[0035] Furthermore, specifically, in step S401, outliers and noise in the point cloud are removed by median filtering.

[0036] Furthermore, specifically, in step S403, the point cloud transformation is performed using the following formula:

[0037]

[0038] In the formula p s and p t These are the point clouds before and after the transformation, respectively. and The ICP point cloud registration algorithm is used to solve the transformation relationship between point clouds from other viewpoints to the reference viewpoint, where i is the index of the point cloud to be transformed and j is the index of the reference point cloud.

[0039] This invention also proposes a three-dimensional reconstruction device based on line laser galvanometer scanning and a four-camera array, comprising the following:

[0040] Support frame;

[0041] Four CMOS cameras are arrayed on top of the support frame to uniformly acquire images within the scanning range from four different angles;

[0042] The galvanometer is positioned at the center of the array of four CMOS cameras;

[0043] A line laser, placed horizontally, with its axis perpendicular to and intersecting the motor shaft of the galvanometer;

[0044] The control module is used to control the movement of the galvanometer and the synchronization between the CMOS camera and the galvanometer.

[0045] The image acquisition module is used to drive four CMOS cameras to acquire images synchronously and set camera parameters;

[0046] The system calibration module is used to complete system calibration using the acquired calibration images, and to set and save system parameters.

[0047] The 3D reconstruction module controls the overall process of 3D reconstruction of an object, and displays and saves the reconstructed point cloud results.

[0048] The beneficial effects of this invention are as follows:

[0049] This invention proposes a 3D reconstruction method and apparatus based on line laser galvanometer scanning and a four-camera array. This method uses a four-camera array for image acquisition, providing a more complete field of view compared to binocular systems. The system calibration method proposed in this invention directly obtains the mapping relationship between the 2D image coordinates of each camera and the 3D coordinates of the galvanometer, decomposing multi-view reconstruction into multiple monocular reconstructions. This avoids the stereo matching process of traditional passive multi-view 3D reconstruction methods and solves the problems of lost object details and reconstruction blind spots caused by some areas not being simultaneously observed by two cameras. Furthermore, the proposed method incorporates a large amount of computation required for 3D reconstruction into the offline system calibration, reducing the computational load of the online algorithm and improving reconstruction speed. In addition, this method uses a line laser as the light source, which is lower in cost and higher in accuracy compared to DLP projection devices, and solves the problem of poor reconstruction results in weakly textured areas in passive 3D reconstruction by actively projecting the laser onto the object surface. Attached Figure Description

[0050] The above and other features of this disclosure will become more apparent from the detailed description of the embodiments illustrated in conjunction with the accompanying drawings. In the accompanying drawings, the same reference numerals denote the same or similar elements. Obviously, the drawings described below are merely some embodiments of this disclosure. Those skilled in the art can obtain other drawings based on these drawings without any creative effort. In the drawings:

[0051] Figure 1 The diagram shown is a camera arrangement diagram of the three-dimensional reconstruction device based on line laser galvanometer scanning and a four-camera array according to the present invention.

[0052] Figure 2The flowchart shown is a three-dimensional reconstruction method based on line laser galvanometer scanning and a four-camera array according to the present invention.

[0053] Figure 3 The diagram shown is a schematic diagram of the system calibration principle in this invention;

[0054] Figure 4 The diagram shown is a flowchart of step S103 in this invention;

[0055] Figure 5 The diagram shown is a flowchart of step S104 in this invention;

[0056] Figure 6 The diagram shows the geometric positional relationship between the coordinate systems of each CMOS camera and the coordinate system of the galvanometer in this invention.

[0057] Figure 7 The diagram shown illustrates the principle of object point cloud reconstruction in this invention.

[0058] Figure 8 The diagram shown is a flowchart of step S109 in this invention;

[0059] Figure 9 The diagram shows the software architecture principle block diagram of a 3D reconstruction device based on line laser galvanometer scanning and a four-camera array. Detailed Implementation

[0060] The following will provide a clear and complete description of the concept, specific structure, and technical effects of the present invention in conjunction with embodiments and accompanying drawings, so as to fully understand the purpose, solution, and effects of the present invention. It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. The same reference numerals used throughout the accompanying drawings indicate the same or similar parts.

[0061] Example 1, referring to Figure 2 This invention proposes a three-dimensional reconstruction method based on line laser galvanometer scanning and a four-camera array, comprising the following:

[0062] Four CMOS cameras are arranged in an array in advance to acquire laser stripe images within the scanning range from four different angles. A galvanometer and a line laser are set at the center of the four CMOS cameras and calibration is performed to establish the mapping relationship between the two-dimensional pixel coordinates of the four CMOS cameras and the three-dimensional coordinates of the galvanometer.

[0063] Step 110: Project a single line laser at an initial angle onto the scene under test using a line laser, and drive four CMOS cameras to simultaneously acquire laser stripe images of the scene under test.

[0064] Step 120: Extract the center of the laser stripe in the laser stripe image;

[0065] Step 130: Calculate the three-dimensional point cloud data of a single line laser under four viewpoints based on the mapping relationship between the two-dimensional pixel coordinates of the four CMOS cameras and the three-dimensional coordinates of the galvanometer.

[0066] Step 140: Reconstruct the striped point cloud by fusing the 3D point cloud data from four perspectives;

[0067] Step 150: Drive the laser galvanometer to continuously change the projection angle of the line laser, and repeat steps 110 to 140 to reconstruct the fringe point cloud of each part of the scene under test.

[0068] Step 160: Combine and reconstruct the striped point clouds of each part to obtain the complete point cloud of the object.

[0069] In a preferred embodiment of the present invention, specifically, the calibration to establish the mapping relationship between the two-dimensional pixel coordinates of the four CMOS cameras and the three-dimensional coordinates of the galvanometer includes:

[0070] S101 drives four cameras to simultaneously acquire images of the checkerboard calibration board in different postures in the field of view, and acquires line laser stripe images of the calibration board at several different projection angles in the first three postures through a line laser and a laser galvanometer.

[0071] S102. Using the laser stripe images of the calibration board under different postures, calculate the intrinsic parameters of the four cameras and the extrinsic parameters of the calibration board under each posture using Zhang's calibration method.

[0072] S103. Using the intrinsic and extrinsic parameters of each camera obtained from camera calibration and several coplanar laser stripe images acquired, several plane equations of laser planes in the four camera coordinate systems are obtained by fitting.

[0073] S104. Based on the obtained plane equations and their corresponding plane normal vectors, the transformation relationships between the four camera coordinate systems and the galvanometer coordinate system are obtained respectively.

[0074] S105. By combining the transformation relationship between the four camera coordinate systems and the galvanometer coordinate system, and the light plane constraint equation related to the galvanometer rotation angle, establish the mapping relationship between the two-dimensional pixel coordinates of the four cameras and the three-dimensional coordinates of the galvanometer.

[0075] In a preferred embodiment of the present invention, specifically, in step S103, the intrinsic and extrinsic parameters of each camera obtained from camera calibration and the acquired coplanar laser stripe images are used to fit several plane equations of the laser planes in the four camera coordinate systems, including...

[0076] S201: Divide the acquired calibration plate line laser stripe image into several groups of coplanar laser stripe images;

[0077] Among them, the laser stripe images on the three attitude calibration plates corresponding to the same galvanometer angle are divided into a group and regarded as coplanar laser stripe images. The final number of image groups is the same as the number of projection angles in step S101.

[0078] S202: Extract the center of the laser stripe in each image based on the gray-scale centroid method;

[0079] S203: Using the camera intrinsic and extrinsic parameters obtained from camera calibration, calculate the three-dimensional coordinates of the laser stripe center in each camera coordinate system;

[0080] S204: Based on the three-dimensional coordinates of the coplanar laser fringe center in each camera coordinate system, the plane equations of several laser planes in each camera coordinate system are obtained by fitting using the least squares method.

[0081] In a preferred embodiment of the present invention, specifically, in step S104, the transformation relationships between the four camera coordinate systems and the galvanometer coordinate system are obtained based on the obtained plane equations and their corresponding plane normal vectors, including:

[0082] S301: Calculate the unit direction vector of each coordinate axis of the galvanometer coordinate system in each camera coordinate system based on the geometric relationship between the unit normal vector of the laser plane and each coordinate axis of the galvanometer coordinate system;

[0083] S302: Using the fitted laser plane equation, solve for Y using the least squares method. G The point P on the axis that has the shortest distance to all light planes i (x 0i ,y 0i ,z 0i ), Y G The axis is the axis in the galvanometer coordinate system that coincides with the galvanometer motor's rotating shaft;

[0084] S303: According to Y G Axial unit direction vector and P i Point to establish Y G The axis equations were calculated, and the coordinates O of the origin of the galvanometer coordinate system in each camera coordinate system were obtained. Gi (x Gi ,y Gi ,z Gi );

[0085] S304: Based on the obtained unit direction vectors of each coordinate axis of the galvanometer coordinate system in each camera coordinate system and O Gi Establish the transformation relationship between each camera coordinate system and the galvanometer coordinate system.

[0086] In a preferred embodiment of the present invention, specifically, step 140 involves fusing three-dimensional point cloud data from four perspectives to reconstruct the fringe point cloud, including...

[0087] S401: Point cloud preprocessing to remove noise;

[0088] S402: Set a reference point cloud based on the point cloud location, and use the ICP point cloud registration algorithm to solve the transformation of each point cloud to the reference point cloud;

[0089] S403: Transform each point cloud to the reference point cloud position using point cloud transformation relationships and then stitch them together.

[0090] In a preferred embodiment of the present invention, specifically, in step S401, outliers and noise in the point cloud are removed by median filtering.

[0091] In a preferred embodiment of the present invention, specifically, in step S403, the point cloud transformation is performed using the following formula:

[0092]

[0093] In the formula p s and p t These are the point clouds before and after the transformation, respectively. and The ICP point cloud registration algorithm is used to solve the transformation relationship between point clouds from other viewpoints to the reference viewpoint, where i is the index of the point cloud to be transformed and j is the index of the reference point cloud.

[0094] A preferred embodiment of the overall process of the present invention is described below:

[0095] like Figure 1 The diagram shown illustrates the hardware architecture of a 3D reconstruction system based on line laser galvanometer scanning and a four-camera array, as provided in an embodiment of the present invention. It mainly consists of four cameras, a line laser, a laser galvanometer, a support frame, and several connecting components. The four cameras are arranged in an array to uniformly acquire images within the scanning range from four different angles. The galvanometer is located at the center of the camera array, and the line laser is placed horizontally, its axis perpendicular to and intersecting the galvanometer motor's rotation axis. This diagram only illustrates the relative positions of the hardware components; the specific arrangement can be designed separately.

[0096] Figure 2 This is a flowchart illustrating a three-dimensional reconstruction method based on line laser galvanometer scanning and a four-camera array provided in an embodiment of the present invention. The method includes steps S101 to S111.

[0097] (I) System calibration steps:

[0098] The basic principles of system calibration are as follows: Figure 3 As shown, where O C1 -X C1 Y C1 Z C1O C2 -X C2 Y C2 Z C2 O C3 -X C3 Y C3 Z C3 and O C4 -X C4 Y C4 Z C4 These are the camera coordinate systems for camera 1, camera 2, camera 3, and camera 4, respectively; O G -X G Y G Z G Let Y be the coordinate system of the galvanometer. G The shaft coincides with the rotating shaft of the galvanometer motor, Z G The axis lies on the laser plane when the galvanometer angle is 0°, with the origin O. G Located at the center of the camera array, X G The axis is determined by the right-hand rule.

[0099] S101: Drives four cameras to simultaneously acquire images of the checkerboard calibration board in different postures in the field of view, and acquires line laser stripe images of the calibration board at several different projection angles in the first three postures through a line laser and a laser galvanometer.

[0100] The main implementation method of this step is as follows: Figure 3 As shown in the left part, the number of line laser projection angles should be greater than 4.

[0101] S102: Using the collected calibration board images in different postures, the intrinsic parameters of the four cameras and the extrinsic parameters of the calibration board in each posture are calculated by Zhang's calibration method.

[0102] S103: Using the intrinsic and extrinsic parameters of each camera obtained from camera calibration and several coplanar laser stripe images acquired, several plane equations of laser planes in the four camera coordinate systems are obtained by fitting.

[0103] Specifically, such as Figure 4 As shown, step S103 includes:

[0104] S201: Divide the acquired calibration plate line laser stripe image into several groups of coplanar laser stripe images;

[0105] Among them, the laser stripe images on the three attitude calibration plates corresponding to the same galvanometer angle are divided into a group and regarded as coplanar laser stripe images. The final number of image groups is the same as the number of projection angles in step S101.

[0106] S202: Extract the center of the laser stripe in each image;

[0107] This step can be performed using the grayscale centroid method as follows:

[0108] For the laser stripe images captured by camera 1 and camera 3, the stripes are horizontal. Therefore, the pixels with gray values ​​greater than a certain threshold in each column of the image are substituted into the following formula for calculation:

[0109]

[0110] In the formula, f(i,j) is the gray value of the pixel in the i-th row and j-th column of the image, (u j ,v j The grayscale center of this column is considered as the center point of the laser stripe in this column.

[0111] For the laser stripe images captured by cameras 2 and 4, the stripes are vertically oriented. Therefore, the pixels with gray values ​​greater than a certain threshold in each row of the image are substituted into the following formula:

[0112]

[0113] In the formula (u i ,v i The grayscale center of gravity of the line is considered as the center point of the laser stripe on that line.

[0114] S203: Using the camera intrinsic and extrinsic parameters obtained from camera calibration, calculate the three-dimensional coordinates of the laser stripe center in each camera coordinate system;

[0115] In the pixel coordinate system Ou of any camera i (i = 1, 2, 3, 4) i v i In this context, assume P(u,v) is any point at the center of the laser stripe on the calibration plate, and its coordinates in the corresponding camera coordinate system O Ci -X Ci Y Ci Z Ci The coordinates in (X) are (X) C ,Y C Z C In the world coordinate system O W -X W Y W Z W The coordinates in (X) are (X) W ,Y W Z W According to the definition of the world coordinate system on the calibration plate, Z... W =0, substituting it into the pinhole imaging model of the camera, we get:

[0116]

[0117] In the formula M Ii and MWi These are the camera i intrinsic and extrinsic parameter matrices obtained from camera calibration, respectively. M is the transformation matrix between the world coordinate system and the pixel coordinate system. IWi It can be represented as:

[0118]

[0119] Furthermore, the world coordinates (X) of the fringe center can be obtained as follows: W ,Y W The equation for (,0):

[0120]

[0121] Based on the relationship between the world coordinate system and the camera coordinate system, the coordinates of the fringe center in the camera coordinate system are (X... C ,Y C Z C The following formula can be used to solve for it:

[0122]

[0123] S204: Based on the three-dimensional coordinates of the coplanar laser fringe center in each camera coordinate system, the least squares method is used to fit the plane equations of several laser planes in each camera coordinate system.

[0124] This step utilizes a series of coplanar laser fringe center sample points (X) obtained in step S203. Ck ,Y Ck Z Ck ), k=0,1,…,n-1(n≥3), and the plane equations of several laser planes in each camera coordinate system are obtained by fitting using the least squares method.

[0125] In the camera coordinate system O C1 -X C1 Y C1 Z C1 With O C3 -X C3 Y C3 Z C3 Below, because the light plane is never in contact with the Y... Ci With the axes parallel, the fitted plane equation is expressed as:

[0126] a ij X C -Y C +c ij Z C +d ij =0

[0127] In camera coordinate system O C2 -X C2 YC2 Z C2 With O C4 -X C4 Y C4 Z C4 Below, because the light plane is never in contact with X Ci With the axes parallel, the fitted plane equation is expressed as:

[0128] -X C +b ij Y C +c ij Z C +d ij =0

[0129] S104: Based on the obtained light plane equation and its corresponding plane normal vector, the transformation relationship between the four camera coordinate systems and the galvanometer coordinate system is obtained respectively;

[0130] Specifically, such as Figure 5 As shown, step S104 includes:

[0131] S301: Calculate the unit direction vector of each coordinate axis of the galvanometer coordinate system in each camera coordinate system based on the geometric relationship between the unit normal vector of the laser plane and each coordinate axis of the galvanometer coordinate system;

[0132] Based on the laser plane equation obtained in step S103, the unit normal vector of the j-th light plane in the i-th camera coordinate system can be calculated.

[0133] According to such Figure 6 The geometric positional relationships between the camera coordinate systems and the galvanometer coordinate system shown can be obtained as follows:

[0134]

[0135] in, arrive X G The unit direction vector of the axis in the four camera coordinate systems arrive These are the unit normal vectors of the laser plane in each camera coordinate system when the galvanometer rotation angle is 0°.

[0136] Similarly, according to Figure 6 We can obtain:

[0137]

[0138] In the formula For Y G The axis in the camera coordinate system O Ci -X Ci Y Ci ZCi The unit direction vector is given. Based on this formula, the optimal solution can be obtained using several samples of the laser plane's unit normal vectors through the least squares method.

[0139] In seeking and Then, based on the perpendicular relationship between the three coordinate axes of the galvanometer coordinate system, Z can be determined. G The axis in the camera coordinate system O Ci -X Ci Y Ci Z Ci unit direction vector

[0140]

[0141] S302: Using the fitted laser plane equation, solve for Y using the least squares method. G The point P on the axis that has the shortest distance to all light planes i (x 0i ,y 0i ,z 0i );

[0142] S303: According to Y G Axial unit direction vector and P i Point to establish Y G The axis equations were calculated, and the coordinates O of the origin of the galvanometer coordinate system in each camera coordinate system were obtained. Gi (x Gi ,y Gi ,z Gi );

[0143] In obtaining P i After obtaining the coordinates, the camera coordinate system O Ci -X Ci Y Ci Z Ci Y in G The equation of the axis can be expressed as:

[0144]

[0145] Due to O G In plane O Ci -Y Ci Z Ci Above, therefore x Gi =0. Move O Gi (0,y Gi ,z Gi Substitute Y G From the equation of the axis, we obtain O. Gi The coordinates are:

[0146]

[0147] S304: Based on the obtained unit direction vectors of each coordinate axis of the galvanometer coordinate system in each camera coordinate system and O Gi Establish the transformation relationship between each camera coordinate system and the galvanometer coordinate system;

[0148] According to the definition of the galvanometer coordinate system, the galvanometer coordinate system O G -X G Y G Z G Relative to camera coordinate system O Ci -X Ci Y Ci Z Ci rotation matrix R Gi Translation vector T Gi They are respectively:

[0149]

[0150] S105: Combine the transformation relationship between the camera coordinate system and the galvanometer coordinate system with the light plane constraint equation related to the galvanometer rotation angle to establish the mapping relationship between the two-dimensional pixel coordinates of the four cameras and the three-dimensional coordinates of the galvanometer.

[0151] Specifically, step S105 includes:

[0152] Based on the pinhole imaging model of the camera, the mapping relationship between the galvanometer coordinate system and the camera i-pixel coordinate system can be expressed as follows:

[0153]

[0154] In the formula M Ii Let M be the intrinsic parameter matrix of camera i obtained from camera calibration. Gi This represents the transformation relationship between the galvanometer coordinate system and the camera i-coordinate system. The transformation matrix M between the galvanometer coordinate system and the camera i-pixel coordinate system is... IGi It can be represented as:

[0155]

[0156] The relationship between the pixel coordinate system and the galvanometer coordinate system can be further described by the following system of equations:

[0157]

[0158] Since the coefficient matrix of this system of equations is a matrix with row rank r = 3, the system of equations has infinitely many solutions. To obtain a unique solution, the following definition of the optical plane in the galvanometer coordinate system, related to the galvanometer rotation angle, is introduced as an additional constraint on the system of equations:

[0159] XG cos2ω j -Z G sin2ω j =0

[0160] In the formula ω j Let X be the current rotation angle of the galvanometer. Adding this constraint to the system of equations, we obtain the following solution: Solve for the three-dimensional coordinates (X, Y) of the laser fringe points in the galvanometer coordinate system. G ,Y G Z G The system of equations:

[0161]

[0162] This system of equations allows us to determine the coordinates of the laser line center (u, v) and the corresponding mirror rotation angle ω on the current optical plane. j Under the given conditions, the three-dimensional coordinates (X, X, Y) of the fringe center point in the galvanometer coordinate system are obtained. G ,Y G Z G This method enables the reconstruction of 3D point clouds of laser stripes. This process eliminates the need for stereo matching, effectively resolving issues such as loss of object details, reconstruction blind spots, and mismatches caused by some areas not being simultaneously observable by two cameras, thus improving reconstruction accuracy and speed.

[0163] (II) Point Cloud Reconstruction Steps:

[0164] like Figure 7 As shown, this illustrates the basic principle of object point cloud reconstruction.

[0165] S106: A single line laser beam at an initial angle is projected onto the scene under test by a line laser, driving four cameras to simultaneously acquire images of the laser stripes in the scene;

[0166] S107: Extract the center of the laser stripe in the laser stripe image;

[0167] The extraction method in this step is the same as in step S202.

[0168] S108: The mapping relationship between the two-dimensional pixel coordinates of the four cameras and the three-dimensional coordinates of the galvanometer obtained by the system calibration is used to calculate the three-dimensional point cloud data of a single laser in four viewpoints;

[0169] This step uses the mapping relationship obtained in step S105 to reconstruct a single laser beam.

[0170] S109: Fusion of laser stripe point clouds from four perspectives;

[0171] like Figure 8 As shown, step S109 includes:

[0172] S401: Point cloud preprocessing to remove noise;

[0173] This step uses median filtering to remove outliers and noise from the point cloud;

[0174] S402: Set a reference point cloud based on the point cloud location, and use the ICP point cloud registration algorithm to solve the transformation of each point cloud to the reference point cloud;

[0175] First, determine the reference viewpoint point cloud based on the position of the point cloud in the field of view. Since the closer the laser stripe is to the camera, the higher the reconstruction accuracy of the stripe, when the X coordinate of the point cloud is less than or equal to 0, the point cloud at viewpoint 1 is used as the reference; when the X coordinate of the point cloud is greater than 0, the point cloud at viewpoint 3 is used as the reference.

[0176] Next, the ICP point cloud registration algorithm is used to solve for the transformation relationship between point clouds from other viewpoints and the reference viewpoint point cloud. and Where i is the index of the point cloud to be transformed, and j is the index of the reference point cloud.

[0177] S403: Transform each point cloud to the position of the reference point cloud using point cloud transformation relationships and then stitch them together;

[0178] Specifically, the point cloud transformation is performed using the following formula:

[0179]

[0180] In the formula p s and p t These are the point clouds before and after the transformation, respectively.

[0181] S110: Drive the laser galvanometer to continuously change the projection angle of the line laser, and reconstruct the fringe point cloud of each part of the scene according to the above method;

[0182] S111: The point clouds of each part obtained by stitching and reconstruction are used to obtain the complete point cloud of the object.

[0183] This invention directly obtains the conversion relationship between the two-dimensional image captured by the camera and the three-dimensional coordinates of the object through the offline system calibration process. This allows the system to complete the three-dimensional point cloud reconstruction of the object with only a small amount of calculation during the online point cloud reconstruction process, thereby improving the speed of three-dimensional reconstruction to a certain extent.

[0184] The present invention also provides a software architecture for a three-dimensional reconstruction system based on line laser galvanometer scanning and a four-camera array, which is used to execute any of the aforementioned three-dimensional reconstruction methods based on line laser galvanometer scanning and a four-camera array. Figure 9 This is a schematic block diagram illustrating the software architecture of a 3D reconstruction system based on line laser galvanometer scanning and a four-camera array, provided in an embodiment of the present invention.

[0185] like Figure 9 As shown, the software architecture 900 includes: a control module 901, an image acquisition module 902, a system calibration module 903, and a 3D reconstruction module 904.

[0186] Reference Figure 1 The present invention also proposes a three-dimensional reconstruction device based on line laser galvanometer scanning and a four-camera array, comprising the following:

[0187] Support frame;

[0188] Four CMOS cameras are arrayed on top of the support frame to uniformly acquire images within the scanning range from four different angles;

[0189] The galvanometer is positioned at the center of the array of four CMOS cameras;

[0190] A line laser, placed horizontally, with its axis perpendicular to and intersecting the motor shaft of the galvanometer;

[0191] The control module 901 is used to control the movement of the galvanometer and the synchronization between the CMOS camera and the galvanometer;

[0192] The image acquisition module 902 is used to drive four CMOS cameras to acquire images synchronously and set camera parameters;

[0193] The system calibration module 903 is used to complete system calibration using the acquired calibration images, and to set and save system parameters.

[0194] The 3D reconstruction module 904 is used to control the overall process of 3D reconstruction of objects, display and save the reconstructed point cloud results.

[0195] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0196] If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or system capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0197] Although the description of the invention has been quite detailed and particularly of several described embodiments, it is not intended to limit it to any of these details or embodiments or any particular embodiment, but should be considered as providing a broad possible interpretation of the claims by referring to the appended claims and taking into account the prior art, thereby effectively covering the intended scope of the invention. Furthermore, the invention has been described above with respect to embodiments foreseeable by the inventors in order to provide a useful description, and non-substantial modifications to the invention that have not yet been foreseen may still represent equivalent modifications.

[0198] The above description is merely a preferred embodiment of the present invention. The present invention is not limited to the above-described embodiments. Any embodiment that achieves the technical effects of the present invention using the same means should fall within the protection scope of the present invention. Within the protection scope of the present invention, various modifications and variations of the technical solutions and / or embodiments are possible.

Claims

1. A three-dimensional reconstruction method based on line laser galvanometer scanning and a four-camera array, characterized in that, Including the following: Four CMOS cameras are arranged in an array in advance to acquire laser stripe images within the scanning range from four different angles. A galvanometer and a line laser are set at the center of the four CMOS cameras and calibration is performed to establish the mapping relationship between the two-dimensional pixel coordinates of the four CMOS cameras and the three-dimensional coordinates of the galvanometer. Step 110: Project a single line laser at an initial angle onto the scene under test using a line laser, and drive four CMOS cameras to simultaneously acquire laser stripe images of the scene under test. Step 120: Extract the center of the laser stripe in the laser stripe image; Step 130: Calculate the three-dimensional point cloud data of a single line laser under four viewpoints based on the mapping relationship between the two-dimensional pixel coordinates of the four CMOS cameras and the three-dimensional coordinates of the galvanometer. Step 140: Reconstruct a partial striped point cloud by fusing 3D point cloud data from four perspectives; Step 150: Drive the laser galvanometer to continuously change the projection angle of the line laser, and repeat steps 110 to 140 to reconstruct the fringe point cloud of each part of the scene under test. Step 160: Combine the striped point clouds obtained from the reconstruction of each part to obtain the complete point cloud of the object; Specifically, calibration is performed to establish the mapping relationship between the two-dimensional pixel coordinates of the four CMOS cameras and the three-dimensional coordinates of the galvanometer, including: S101 drives four cameras to simultaneously acquire images of the checkerboard calibration board in different postures in the field of view, and acquires line laser stripe images of the calibration board at several different projection angles in the first three postures through a line laser and a laser galvanometer. S102. Using the laser stripe images of the calibration board under different postures, calculate the intrinsic parameters of the four cameras and the extrinsic parameters of the calibration board under each posture using Zhang's calibration method. S103. Using the intrinsic and extrinsic parameters of each camera obtained from camera calibration and several coplanar laser stripe images acquired, several plane equations of laser planes in the four camera coordinate systems are obtained by fitting. S104. Based on the obtained plane equations and their corresponding plane normal vectors, the transformation relationships between the four camera coordinate systems and the galvanometer coordinate system are obtained respectively. S105. By combining the transformation relationship between the four camera coordinate systems and the galvanometer coordinate system, and the light plane constraint equation related to the galvanometer rotation angle, establish the mapping relationship between the two-dimensional pixel coordinates of the four cameras and the three-dimensional coordinates of the galvanometer.

2. The three-dimensional reconstruction method based on line laser galvanometer scanning and a four-camera array according to claim 1, characterized in that, Specifically, in step S103, the intrinsic and extrinsic parameters of each camera obtained from camera calibration and the acquired coplanar laser fringe images are used to fit several plane equations of the laser plane in the four camera coordinate systems. include, S201: Divide the acquired calibration plate line laser stripe image into several groups of coplanar laser stripe images; Among them, the laser stripe images on the three attitude calibration plates corresponding to the same galvanometer angle are divided into a group and regarded as coplanar laser stripe images. The final number of image groups is the same as the number of projection angles in step S101. S202: Extract the center of the laser stripe in each image based on the gray-scale centroid method; S203: Using the camera intrinsic and extrinsic parameters obtained from camera calibration, calculate the three-dimensional coordinates of the laser stripe center in each camera coordinate system; S204: Based on the three-dimensional coordinates of the coplanar laser fringe center in each camera coordinate system, the plane equations of several laser planes in each camera coordinate system are obtained by fitting using the least squares method.

3. The three-dimensional reconstruction method based on line laser galvanometer scanning and a four-camera array according to claim 2, characterized in that, Specifically, in step S104, based on the obtained plane equations and their corresponding plane normal vectors, the transformation relationships between the four camera coordinate systems and the galvanometer coordinate system are solved, including: S301: Calculate the unit direction vector of each coordinate axis of the galvanometer coordinate system in each camera coordinate system based on the geometric relationship between the unit normal vector of the laser plane and each coordinate axis of the galvanometer coordinate system; S302: Solve the laser plane equation obtained by fitting using the least squares method. The point on the axis with the shortest distance to all light planes. , The axis is the axis in the galvanometer coordinate system that coincides with the galvanometer motor's rotating shaft; S303: According to Axial unit direction vector and Point creation The axis equations were calculated, and the coordinates of the origin of the galvanometer coordinate system in each camera coordinate system were obtained. ; S304: Based on the obtained unit direction vectors of each coordinate axis in the galvanometer coordinate system in each camera coordinate system and... Establish the transformation relationship between each camera coordinate system and the galvanometer coordinate system.

4. The three-dimensional reconstruction method based on line laser galvanometer scanning and a four-camera array according to claim 1, characterized in that, Specifically, in step 140, the fringe point cloud for this part is reconstructed by fusing 3D point cloud data from four perspectives, including: S401: Point cloud preprocessing to remove noise; S402: Set a reference point cloud based on the point cloud location, and use the ICP point cloud registration algorithm to solve the transformation of each point cloud to the reference point cloud; S403: Transform each point cloud to the reference point cloud position using point cloud transformation relationships and then stitch them together.

5. The three-dimensional reconstruction method based on line laser galvanometer scanning and a four-camera array according to claim 4, characterized in that, Specifically, in step S401, outliers and noise in the point cloud are removed by median filtering.

6. The three-dimensional reconstruction method based on line laser galvanometer scanning and a four-camera array according to claim 4, characterized in that, Specifically, in step S403, the point cloud transformation is performed using the following formula: ； In the formula and These are the point clouds before and after the transformation, respectively. and The ICP point cloud registration algorithm solves for the transformation relationship from other viewpoint point clouds to the reference viewpoint point cloud, where... Index of the point cloud to be transformed. This serves as the baseline point cloud index.

Images