An automated three-dimensional color imaging and measurement system

Abstract

The present invention provides an automated three-dimensional color imaging and measurement system, comprising: an imaging module, comprising a color three-dimensional sensor for acquiring a first three-dimensional image and a color image of a measured object, and a depth camera for acquiring a second three-dimensional image of the measured object; a posture adjustment module, for fixing the imaging module and performing posture adjustment on the imaging module; a processor, connecting and controlling the imaging module and the posture adjustment module, so that the imaging module performs multi-angle image acquisition on the measured object, and calculates a color three-dimensional model of the measured object based on the first three-dimensional image, the color image and the second three-dimensional image. The present invention controls the multi-viewing posture adjustment of the three-dimensional sensor by using a mechanical arm and a rotating motor to meet the overlap and integrity of all scanning fields of view; obtains the morphological data of the object in real time through a depth camera to estimate the optimal scanning posture of the three-dimensional sensor, so as to ensure that the surface to be measured of the object is within the measurement range of the three-dimensional sensor.

Description

Technical Field

[0001] The invention belongs to the field of electronic technology, and more specifically, relates to an automatic three-dimensional color imaging and measuring system. Background Art

[0002] Among many optical three-dimensional measurement technologies, phase-based active binocular vision 3D imaging technology is considered to be the most effective technology for accurately detecting and reconstructing the three-dimensional shape of objects due to its non-contact, fast and high-precision characteristics.

[0003] However, in the process of optical 3D measurement and imaging, due to the limitation of the measurement range of the 3D sensor, the size change and topological change of the object to be measured will have different degrees of impact on the complete 3D measurement and imaging, especially bringing huge challenges to automated scanning: it is necessary to meet the integrity requirements of 3D scanning, and coordinate and control the posture relationship between the 3D sensor and the measurement surface to ensure the accuracy and efficiency of 3D digital measurement.

[0004] In view of the huge challenges in current three-dimensional measurement, the present invention provides a robot-assisted fully automatic three-dimensional color imaging and measurement system. Summary of the invention

[0005] To solve the above problems, the present invention provides an automated three-dimensional color imaging and measurement system, comprising: an imaging module, including a color three-dimensional sensor for acquiring a first three-dimensional image and a color image of a measured object, and a depth camera for acquiring a second three-dimensional image of the measured object; a posture adjustment module, for fixing the imaging module and performing posture adjustment on the imaging module; a processor, connecting and controlling the imaging module and the posture adjustment module, so that the imaging module performs multi-angle image acquisition on the measured object, and calculates a color three-dimensional model of the measured object based on the first three-dimensional image, the color image and the second three-dimensional image. In one embodiment, the color three-dimensional sensor includes an active binocular vision camera and a color camera composed of a left camera, a right camera, and a projector.

[0006] In one embodiment, the posture adjustment module includes: a robotic arm connected to the imaging module, including multiple rotating axes to adjust the posture of the imaging model; rotating axes, respectively connected to the robotic arm and the base, for making the imaging module rotate around the base.

[0007] In one embodiment, the color three-dimensional sensor includes an active binocular vision camera consisting of a left camera, a right camera, a projector, and a color camera.

[0008] In one embodiment, the depth camera includes a TOF depth camera, a structured light depth camera, or a binocular vision depth camera.

[0009] In one embodiment, at least one of the resolution, accuracy, and frame rate of the first three-dimensional image is higher than that of the second three-dimensional image.

[0010] In one embodiment, calculating the color three-dimensional model of the object under test based on the first three-dimensional image, the color image and the second three-dimensional image includes the following steps: generating a global scanning viewpoint using the second three-dimensional image; acquiring the first three-dimensional image and the color image at the global scanning viewpoint, and calculating the color three-dimensional model of the object under test based on the first three-dimensional image and the color image.

[0011] In one embodiment, the global scan viewpoint is generated based on constraint conditions, wherein the constraint conditions include but are not limited to at least one of a visibility constraint, a measurement space constraint, an overlap constraint, and an occlusion constraint.

[0012] In one embodiment, the global scan viewpoint is calculated based on the next best viewpoint (NBVs) algorithm; the NBVs algorithm includes the following steps: constructing a minimum bounding box containing the rough model and the scan space, and dividing the scan space into 3D voxel grids at a certain distance interval; finding voxels at the rough model sampling points, and using the voxels as search seeds to perform an expansion search on the domain voxels to obtain the valid voxels; solving the labeling scores of the valid voxels and selecting the voxels with the largest score value for viewpoint calculation.

[0013] In one embodiment, the three-dimensional scanning is performed based on the global scanning viewpoint and shortest path planning is performed to perform the three-dimensional scanning along the shortest path.

[0014] In one embodiment, the shortest path planning algorithm includes but is not limited to one of an ant colony algorithm, a neural network algorithm, a particle swarm algorithm, and a genetic algorithm.

[0015] The beneficial effects of the present invention are as follows: a robot 3D scanning strategy based on the assistance of a depth camera is proposed, which adopts a robotic arm and a rotary motor to control the multi-view posture adjustment of a 3D sensor to meet the overlap and integrity of all scanning fields of view; the morphological data of the object is acquired in real time by the depth camera to estimate the optimal scanning posture of the 3D sensor to ensure that the surface to be measured of the object is within the measurement range of the 3D sensor. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 Schematic diagram of an automated three-dimensional color imaging and measurement system according to an embodiment of the present invention.

[0017] Figure 2 Schematic diagram of the distribution and transformation relationship of the system coordinate system according to an embodiment of the present invention.

[0018] Figure 3 Schematic diagram of a low-cost three-dimensional target based on non-coding markers according to an embodiment of the present invention.

[0019] Figure 4 Schematic diagram of the constraint relationship of a binocular vision three-dimensional sensor according to an embodiment of the present invention.

[0020] Figure 5 Schematic diagram of ISO point visible range (a) and voxel inclusion viewpoint (b) according to an embodiment of the present invention.

[0021] Figure 6 is a statistical diagram of a voxel intra-vector histogram according to an embodiment of the present invention.

[0022] Figure 7 is a flow chart of an NBVs algorithm according to an embodiment of the present invention. DETAILED DESCRIPTION

[0023] The present invention will be further described in detail below in conjunction with specific implementations and with reference to the accompanying drawings. It should be emphasized that the following description is merely exemplary and is not intended to limit the scope of the present invention and its application.

[0024] System Description

[0025] Figure 1 Schematic diagram of an automated three-dimensional color imaging and measurement system according to an embodiment of the present invention. The system 10 includes a base 101, a mechanical arm 102, an imaging module 103, a rotating shaft 105, and a processor (not shown in the figure).

[0026] The base 101 is used to place the object to be measured 104 , and the base may not be a necessary configuration of the system, for example, it may be other planes or structures.

[0027] The imaging module 103 includes a color three-dimensional sensor and a depth camera 1035. The color three-dimensional sensor includes an active binocular vision camera composed of a left camera 1031, a right camera 1032, and a projector 1033, and a color camera 1034, which are respectively used to collect a first three-dimensional image and a color image of the object to be measured 104. By the relative position information between each camera (obtained by calibration), the first three-dimensional image can be further aligned with the color to obtain a three-dimensional color image of the object to be measured, or the color image collected by the color camera is texture mapped to achieve coloring of the three-dimensional image to obtain a three-dimensional color image. In one embodiment, the left camera 1031 and the right camera 1032 are high-resolution black and white cameras, and the projector can be a digital stripe projector for projecting coded structured light images. The left camera 1031 and the right camera 1032 collect phase structured light images and perform high-precision three-dimensional imaging based on phase-assisted active stereo vision (PAAS) technology. In one embodiment, the left and right cameras can also be infrared cameras, etc., and the parameters of the left and right cameras, such as focal length, resolution, depth of field, etc., can be the same or different. The first three-dimensional image refers to a three-dimensional image of the object 104 collected by the color three-dimensional sensor.

[0028] The depth camera 1035 is used to collect a second three-dimensional image of the object under test. The depth camera 1035 can be a depth camera based on time-of-flight (TOF), structured light or passive binocular vision technology. Generally, at least one of the resolution, accuracy and frame rate of the collected second three-dimensional image is lower than that of the first three-dimensional image. Generally speaking, the resolution, accuracy and frame rate of the second three-dimensional image are all lower than those of the first three-dimensional image. For ease of description, in the following description, the first three-dimensional image of the object is referred to as a high-precision fine three-dimensional model, and the second three-dimensional image of the object is referred to as a low-precision rough three-dimensional model. The second three-dimensional image refers to the three-dimensional image of the object under test 104 collected by the depth camera 1035.

[0029] The mechanical arm 102 and the rotating shaft 105 constitute a posture adjustment module, which is used to fix the imaging module 103 and adjust its posture. Among them, the mechanical arm 102 is connected to the imaging module 103 and the rotating shaft 105, and the rotating shaft 105 is installed on the base 101 for rotating around the base 101. The mechanical arm 102 is a multi-axis linkage mechanical arm for corresponding posture adjustment. Through the joint adjustment of the rotating shaft 105 and the mechanical arm 102, the imaging module 103 can be transformed in multiple directions to facilitate multi-directional measurement of the object to be measured 104. In some embodiments, the rotating shaft 105 includes a rotating motor, and the mechanical arm will rotate around the base driven by the rotating shaft under the drive of the rotating motor to measure the object to be measured.

[0030] The processor is connected to the robot arm 102, the imaging module 103, and the rotating shaft 105, and is used to perform control and corresponding data processing or three-dimensional scanning tasks, such as three-dimensional color image extraction, rough three-dimensional model establishment, fine three-dimensional model establishment, etc. It is understandable that the processor can be a single processor or multiple independent processors. For example, the imaging module can include multiple special processors for executing algorithms such as three-dimensional imaging. The system also includes a memory for storing algorithm programs executed by the processor, such as various algorithms and methods mentioned in the present invention (calibration methods, reconstruction methods, viewpoint generation algorithms, and scanning methods, etc.). The memory can be various computer-readable media, such as non-transitory storage media, including magnetic media and optical media, such as disks, tapes, CDROMs, RAMs, ROMs, etc.

[0031] It can be understood that the three-dimensional image mentioned above may refer to a depth image, or may refer to point cloud data, mesh data or three-dimensional model data obtained by further processing based on the depth image.

[0032] When the system 10 is used to perform a three-dimensional scan on the object 104, the overall scanning process is performed by the processor and is divided into the following steps:

[0033] Step 1: calibrate the depth camera 1035 and the color 3D sensor to obtain the internal and external parameters of the depth camera 103 and the color 3D sensor. The specific process is described below.

[0034] Step 2: Use the depth camera 1035 to collect a low-precision rough three-dimensional model of the object 104, such as using the rotating shaft 105 and the mechanical arm 102 to control the depth camera 1035 to circle the object 104 to quickly generate a low-precision rough three-dimensional model of the object. It is understandable that the object 104 needs to be placed on the base 101 in advance. In one embodiment, the object 104 is placed at the center of the base 101;

[0035] Step 3: Generate global scanning viewpoints based on the low-precision rough three-dimensional model calculation, specifically, automatically generate global scanning viewpoints according to the NBVs algorithm proposed in the present invention.

[0036] Step 4: Perform high-precision three-dimensional scanning on the object 104 using an active binocular vision camera based on the generated global scanning viewpoint and the shortest path planning to obtain a first high-precision fine three-dimensional model;

[0037] In some embodiments, it is also necessary to calculate a confidence map of the first high-precision fine three-dimensional model, determine areas with missing data and missing details, and perform supplementary scanning to obtain a second high-precision fine three-dimensional model with higher accuracy;

[0038] In some embodiments, during the acquisition process of the first and / or second high-precision fine three-dimensional models, a color camera is simultaneously used to acquire color images, and the color images are texture mapped to achieve coloring of the fine three-dimensional models to obtain three-dimensional color digital images, ultimately achieving high-fidelity three-dimensional color digitization of the complete object.

[0039] System calibration

[0040] Before using the system 10 to perform a three-dimensional scan on the object 104 to be measured, it is necessary to calibrate each component in the system to obtain the relative position relationship between the coordinate systems of each component. Only based on the relative position relationship can corresponding operations be performed, such as coloring, generating a global scanning viewpoint based on a rough three-dimensional model, etc.

[0041] Figure 2 Schematic diagram of the distribution and transformation relationship of the system coordinate system according to an embodiment of the present invention. The world coordinate system is established on the base coordinate system, and the color 3D sensor coordinate system is established on the left camera S l The depth camera coordinate system is built on the infrared camera S i On. It is necessary to determine the internal and external parameters of the color three-dimensional sensor through system calibration, as well as the transformation matrix of the color three-dimensional sensor / depth camera coordinate system-mechanical arm coordinate system-mechanical arm base coordinate system-base coordinate system. The difficulty of system calibration in the present invention lies in that there are sensors with different resolutions and different fields of view (for example, the color camera has 20 million pixels, the left and right cameras have 5 million pixels, and the lens FOV is H39.8°, V27.60; the depth camera has 300,000 pixels, and the FOV is H 58.40, V 45.50), and sensors with different spectral response ranges (such as the spectral response range of the color camera and the black and white camera is in the visible light band; the response range of the infrared camera is in the infrared band). At the same time, the calibration accuracy of the color three-dimensional sensor must be guaranteed. Therefore, designing and making high-precision stereo targets is the key to high-precision calibration.

[0042] Figure 3This is a schematic diagram of a low-cost stereo target based on non-coding marker points according to an embodiment of the present invention. The stereo target consists of a first sub-target A and a second sub-target B. The first sub-target A is composed of a plane, and the surface of the plane is non-coding marker points arranged regularly (for example, 11×9). The accurate spatial coordinates of these marker points can be determined by bundle adjustment technology. The marker points include reference points and positioning points, where the point locations include at least four. In order to improve the accuracy of marker point extraction of low-resolution depth cameras, the reference points and positioning points are both designed with large circles. The interior of the positioning point and the reference point includes a small black concentric marker point (for example, concentric circles), and the positioning point and the reference point are distinguished by the grayscale of the center of the marker point (for example, if the grayscale of the center is greater than 125, it is a reference point, and if it is less than 125, it is a positioning point, that is, the grayscale of the center of the reference point is different from that of the positioning point). Figure 3 As shown in (c), this design greatly increases the size of the reference point and improves the positioning accuracy of the positioning point; the second sub-target B is composed of multiple planes, and non-coding markers are randomly pasted on the surface for rotation axis calibration. The non-coding markers of the second sub-target B are designed with small circles. The small circles here are relative to the size of the markers in the first sub-target A, that is, smaller than the markers in the first sub-target A. The calibration process is to reconstruct the spatial coordinates of random markers under multiple perspectives by surrounding the stereo target with a color three-dimensional sensor, and determine the base coordinate system by optimizing the marker matching. Therefore, there is no need to predetermine the spatial coordinates of the random markers of the second sub-target B, which greatly reduces the difficulty and cost of target production.

[0043] The calibration process is divided into two steps: (1) The rotating shaft (rotating motor) remains stationary, and the robotic arm carries the color 3D sensor to collect multi-view images of the first sub-target A and calculate the internal and external parameters of the color 3D sensor, H lm and H im Since the left and right cameras, the color camera and the infrared camera work under light sources of different spectral bands, in each acquisition, the left and right cameras and the color camera first collect the target image under visible light illumination, and then illuminate with infrared light source, and the infrared camera collects the target image; (2) The robot arm keeps the posture unchanged, the motor rotates at different angles, and the left and right cameras use the binocular stereo vision principle to reconstruct the three-dimensional coordinates of the random marking points on the target B part at each viewing angle. The rotation angle is determined by matching the marking points, thereby constructing the base coordinate system and calculating H ba .

[0044] In one embodiment, when calibrating the color 3D sensor, three cameras (left, right, and infrared cameras) simultaneously acquire target patterns under different viewing angles, and construct the objective function of the single camera calibration model:

[0045]

[0046] in represents the spatial homogeneous coordinates of the jth marker among the M markers in the target coordinate system, x ij (i=1,...N) represents the image coordinates of the jth landmark point in the image captured by the camera at the i-th viewing angle, K is the camera's internal parameter matrix, including focal length, principal point position and tilt factor, ε is the lens distortion, and this paper only considers the typical fifth-order lens distortion {Fryer, 1986#114}, Represents the transformation matrix from the target coordinate system to the camera coordinate system at the i-th viewing angle.

[0047] Generally, assuming that the left camera coordinate system is the three-dimensional sensor coordinate system, the structural parameters of the three cameras are:

[0048]

[0049] in, and The left camera S l To the right camera S r The rotation matrix and translation vector, and For the left camera S l To color camera S c In order to obtain higher-precision structural parameters, we add the transformation matrix to the nonlinear objective function of the three cameras and use the Gauss-Newton or Levenberg-Marquardt method to minimize the objective function to estimate the camera parameters:

[0050]

[0051] where τ = {ε l ,ε r ,ε c ,K l ,K r ,K c ,H lr ,H lc}, In this way, the internal and external parameters of the color 3D sensor can be obtained. The parameters of the infrared camera are solved in a similar way.

[0052] After the color 3D sensor calibration is completed, the transformation matrix of the left camera at each acquisition angle can be obtained. Directly given by the robot control system, according to the mathematical model of hand-eye calibration, the following relationship is established:

[0053]

[0054] Among them, i,k=1,2,...,N, and i≠k, N is the number of scans, and N motion postures can be established According to Tsai's method

[30] , the linear least squares method can be used to solve H sg and H cb .

[0055] In one embodiment, in order to further improve the accuracy, we use it as the initial value to establish a nonlinear objective function:

[0056]

[0057] in It can be obtained from the robot arm in real time. Using the Levenberg-Marquardt method to minimize the objective function can obtain a higher precision H lm With H bt .H im The solution of is similar and will not be discussed further.

[0058] In one embodiment, during the calibration of the rotation axis, the robot arm maintains the same posture, and the transformation matrix from the robot arm to the base is recorded as H′ gb The 3D sensor moves in a circle around the stereo target and reconstructs the random landmarks of part B of the target at different rotation angles. m = 1, 2, ..., T, T is the number of rotations, j is the number of landmarks, and the landmarks reconstructed in all fields of view are Perform global matching optimization to obtain the transformation relationship of the target landmark at each rotation angle [R (m) |T (m) ], then the rotation axis direction vector can be calculated under the constraint of the distance between each two closed circular trajectory planes, and the center of each circular trajectory can be obtained by the global least squares optimization method, thereby determining the transformation relationship H from the three-dimensional sensor coordinate system to the base coordinate system rl According to the transformation relationship (H rl ) -1 =H br H′ mb H lm The base coordinate system to the base coordinate system H can be obtained br .

[0059] Global Scan Viewpoint Generation

[0060] According to the stereoscopic vision imaging model, the measurement space of the three-dimensional sensor is limited by the binocular camera angle (FOV), the focal length and depth of field (DOF) of the camera lens and the digital projection lens, and the quality of the point cloud of three-dimensional reconstruction is also affected by many constraints. The present invention is based on certain constraints. By analyzing the rough three-dimensional model (Roughmodel), a series of scanning viewpoints are automatically generated to achieve three-dimensional digital color imaging of the complete object with the least number of viewpoints. Next, the constraints and viewpoint generation methods are introduced respectively.

[0061] Figure 4 Schematic diagram of the constraint relationship of a binocular vision 3D sensor according to an embodiment of the present invention. Figure 4 (a) is a schematic diagram of the basic structure and measurement space of the binocular sensor. Figure 4 (b) is the measurement space constraint of the 3D sensor, Figure 4 (c) is the point cloud visibility constraint. For the sake of simplicity, the present invention does not describe the calculation of the specific view volume, and the measurement space is simplified to Figure 4 As shown in (b), assuming that the working distance range of the 3D sensor is [d n ,d f ], maximum field of view The viewpoint position is v i (x,y,z),v i (α, β, γ) represents the unit vector of the 3D sensor optical axis direction, v ik =d(v i ,s k ) represents the viewpoint position v i Point to the measurement target point position s k The viewpoint planning process is affected by the object surface space, viewpoint space and imaging work space, and its constraints mainly include but are not limited to at least one of the following aspects:

[0062] 1) Visibility constraint: indicates the angle range within which the target point is allowed to be captured by the sensor. Suppose the target point p is k The normal vector is n k , then the visibility constraint

[0063]

[0064] in Indicates the maximum viewing angle range of the measurement target point, such as Figure 4 (c) as shown.

[0065] 2) Measurement space constraints: including field of view (FOV) constraints and depth of field (DOF) constraints, representing the measurable range of the 3D sensor. The constraints are:

[0066]

[0067] where φ max Indicates the maximum field of view of the 3D sensor, such as Figure 4 (b) as shown.

[0068] 3) Overlap constraint: For the subsequent ICP matching and grid fusion (registration and integration) of multi-view depth data, there needs to be a certain degree of overlap between adjacent scan fields of view. The field of view overlap is defined as W and W cover Represent the total area of ​​the field of view and the area of ​​the overlapping part respectively, and the constraints are

[0069] ξ≥ξ min (8)

[0070] where ξ min is the minimum field of view overlap.

[0071] 4) Occlusion constraint: When the viewpoint v i To the measurement target point s k The line segment d(v i ,s k ) intersects with the object entity, indicating that the viewpoint v i At the target point s k The viewpoint direction v ik is obscured.

[0072] For an object of unknown shape, first use the depth camera to perform an initial scan around the object to generate a rough 3D model. The purpose of this step is to use the model to generate a global scanning perspective, so the rough 3D model does not require too high accuracy and resolution, nor does it require the scanning data to be particularly complete. In addition, since depth cameras generally have the characteristics of a wide scanning perspective, a large depth range of the measurement space, and good real-time performance, for most objects of different sizes and different surface materials, a set of scanning postures can be simply preset to achieve an initial scan of the object's morphology.

[0073] In one embodiment, during the initial scanning process, a matching and fusion algorithm, such as the kinectFusion algorithm, is used to match and integrate the data in real time. After the initial scanning is completed, the original point cloud is preprocessed by noise filtering, smoothing, edge removal, and normalization estimation, and then an initial closed triangular mesh model is generated. The model is sampled by Poisson-disc sampling to obtain the so-called ISO points, such as Figure 4 As shown in (b), let the model sampling point be

[0074] According to the initial model size and the maximum working distance d of the scanner f , construct the minimum bounding box S that includes the model and the scan space, and divide the space into 3D voxel grids according to a certain distance interval ΔD (for example, divided into 100×100×100 voxels). For any spatial point (p x ,p y ,p z ), according to formula (9), we can quickly find out which voxel grid the point belongs to.

[0075]

[0076] Among them (p x-min ,p y-min ,p z-min ) is the minimum coordinate value of the bounding box S, v i =(n x ,n y ,n z ) is the voxel number value, the center point of the voxel It will be used as a spatial 3D point to participate in the calculation of the next best views (NBVs). The NBVs algorithm in this paper is mainly divided into three steps:

[0077] Step 1: For the initial model sampling point s k , along its normal n k , the distance is d0=(d n +d f ) / 2, we can find the voxel v according to formula (9): i . With v i To search for seeds, a greedy algorithm is used to perform an expansion search on the neighborhood voxels. According to the visibility constraints described above, the voxel numbers that satisfy equation (10) are recorded at the sampling point s k The associated collection Among them, such as Figure 5 (a) shown.

[0078]

[0079] where v ik =d(v i ,s k ) represents point v i To point k The vector, w ik (v i ,s k )=1 means s k v i It can be seen that when w ik (v i ,s k )=0 means s k to v i There is occlusion between them. At the same time, (s k ,v ik ) is recorded in all v that satisfy formula (10) i The associated collection In For all ISO points k}Execute step 1 to obtain all valid voxels that record ISO points {v i}, and the voxels that are not recorded are considered invalid and no longer participate in the calculation.

[0080] Step 2: For the valid voxel v i , according to its collection Element s k The labeling function g(s k ), solve for the labeling score of the voxel

[0081]

[0082] g(s k )Marks k The usage of s k When it has not been confirmed to belong to a certain scanning viewpoint, it is marked as 1, and when it has been confirmed to belong to a certain scanning viewpoint, it is marked as 0, that is,

[0083]

[0084] Step 3: Select the voxel with the largest label score value for viewpoint calculation. The ISO points recorded by a voxel may not be covered by the same scanning range, such as Figure 5 (b), so we use the histogram statistics method to All the s k The vector d(v i ,s k ) for statistics and selection. According to the Cartesian coordinate system (x, y, z) and the spherical coordinate system The transformation relationship converts the vector d(v i ,s k ) is converted to the spherical coordinate system, the X-axis and Y-axis of the histogram are θ and The Z axis is the statistical number of iso points, such as Figure 6 As shown. According to the scanning field of view constraint of the 3D sensor φ max and overlap constraint ξ min , determines the size of the filter window in the XY plane φ filter =φ max (1-ξ min ), the filter traverses all elements (x, y) of the histogram XY plane and calculates the sum of the number of iso points in the filter. When the statistic in the filter window is the largest, the iso points contained in the filter are {s′ k} k∈N , N is the label weight g(s′ k )=1 k The number of s′ k The vector d(v i ,s k ) Find the mean value as the scanning viewpoint direction

[0085]

[0086] At this point, the spatial position of the viewpoint and the viewpoint direction vector can be obtained

[0087] Step 4: k The labeling function g(s′) in k ) is set to 0.

[0088] Repeat Step 2-Step 4 until the labeling scores of all voxels are below the threshold. The flowchart of the NBVs algorithm is shown in the figure. From the above algorithm flow, it can be seen that the valid voxels contain all iso points that meet the constraints. The higher the labeling score of the voxel, the more the viewpoint calculated by the voxel can cover the surface range of the object, that is, the more important the viewpoint is. The viewpoint in this article is calculated by selecting the voxel with the largest labeling score, and the final viewpoint list is also sorted from most to least according to the number of iso points covered by the viewpoint.

[0089] Automatic 3D scanning and complementary scanning

[0090] The spatial positions and directions of a series of viewpoints are obtained through the NBVs algorithm described above. How to scan all viewpoints in the shortest path is a path planning problem. Algorithms for solving path planning problems include but are not limited to ant colony algorithms, neural network algorithms, particle swarm algorithms, genetic algorithms, etc., each with its own advantages and disadvantages. For example, in one embodiment, the shortest path can be obtained by solving the viewpoint set using the ant colony algorithm. Next, a color three-dimensional sensor is used to perform three-dimensional scanning along the shortest path. The high-precision depth data (left camera coordinate system) collected at each viewing angle is converted to the world coordinate system through the coordinate system transformation relationship, and finally the real-time matching of multi-view depth data is achieved to calculate a high-precision and fine three-dimensional model of the object.

[0091] In one embodiment, the 3D sensor performs 3D scanning along the shortest path. In each viewpoint transformation process, the joint control of the rotating motor and the robot arm is involved, which is essentially a transformation problem of the coordinate system of each sensor. Suppose two adjacent scanning viewpoints are V i and V j , the transformation matrix of these two viewpoints Indicates the infrared depth sensor coordinate system in the world coordinate system from the viewpoint V i To V j In order to make the three-dimensional sensor from the viewpoint V i Adjust to V j , respectively find the viewpoint V i and V j In the rotation axis coordinate system (i.e. world coordinate system) to X a Y a Projection coordinates v of the plane i⊥ and v j⊥ , thus obtaining the rotation angle θ of the rotating motor ij and the transformation matrix

[0092]

[0093] Viewpoint after motor rotation Then the viewpoint V′ i To V j The transformation matrix Since the transformation matrix H im and the robot arm at viewpoint V′ i The transformation matrix Known, the following transformation relationship can be established:

[0094]

[0095] Therefore, we can get the position of the robot arm at the viewpoint V j The transformation matrix is ​​as follows. By combining the rotation angle θ of the rotating motor ijand the rotation matrix of the robot The posture adjustment of the 3D sensor at different viewpoints can be realized. The high-precision depth data at each viewpoint (in the left camera coordinate system) is converted to the world coordinate system through the transformation matrix of the infrared depth sensor coordinate system and the viewpoint transformation matrix, realizing real-time matching of multi-view depth data.

[0096] From formula (10), we can see that the viewpoint planning algorithm in this paper has taken into account the self-occlusion of the object. However, in the actual scanning process, due to the influence of factors such as the surface material of the object, some data missing or sparse point cloud data and other low quality problems are inevitable. More importantly, since the rough 3D model used for viewpoint planning loses the detailed information of the object, the generated viewpoint does not take into account the fine scanning of the geometric details.

[0097] To this end, in one embodiment, a method of constructing a model confidence map is used to reflect the missing parts of the original data and the missing areas of details, and a viewpoint planning algorithm is used to generate a viewpoint for supplementary scanning. The original point cloud data obtained in the previous high-precision scanning stage is sampled by Poisson-disc sampling to generate IS0 sampling points. Generate iso point s according to formula (16) k Confidence map of :

[0098] f(s k ) = f g (s k ,n k )f s (s k ,n k ) (16)

[0099] where f g (s k ,n k )=Γ(s k )·n k Defined as the completeness confidence score, Γ(s k ) is point s k The scalar field gradient at n k is the normal vector. g (s k ,n k ) is already obtained during the Poisson-disk sampling process, so no additional calculation is required; f s (s k ,n k ) is the smoothness confidence score K (smoothnessconfidencescore), satisfying

[0100]

[0101] where ||g|| is the l2-norm, For point s k The K neighborhood range Ω k The original point cloud inside, The spatial weight function θ(||s k -q j ||) in Ω k The range decays sharply with the increase of radius; the orthogonal weight function φ(n k ,q j -s k ) reflects the K neighborhood range Ω k The original point q j The distance to the tangent plane at the iso point. When the smoothness confidence score is high, the surface point s k The local area is relatively smooth and the scanning quality is relatively high; when the smooth confidence score value is low, it indicates that point s k If the local raw scan data is sparse, or the raw scan data contains more high-frequency components, such as point cloud noise or rich geometric details, more supplementary scans are required.

[0102] The confidence score effectively reflects the quality and fidelity of the scanned model point cloud data. We use the model confidence score to guide the viewpoint planning of the supplementary scanning phase. Set the confidence score threshold ε and find the range of iso points of the missing part and the part rich in geometric details S′={s′ k |f(s′ k )≤ε}, and the viewpoint of S′ is calculated by the above algorithm. The difference from the NBVs algorithm mentioned above is that g(s′ k ) According to s′ k Assign confidence scores

[0103]

[0104] Therefore, the voxel score no longer reflects the number of iso points it contains, but the sum of the confidence scores of the iso points. Calculating the viewpoint for the voxel with the highest confidence score will make the viewpoint focus more on scanning the missing parts and parts rich in geometric details.

[0105] The above contents are further detailed descriptions of the present invention in combination with specific / preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, they can also make several substitutions or modifications to these described embodiments without departing from the concept of the present invention, and these substitutions or modifications should be regarded as belonging to the protection scope of the present invention.

Claims

1. An automated three-dimensional color imaging and measurement system, characterized in that: include: An imaging module, comprising a color three-dimensional sensor for acquiring a first three-dimensional image and a color image of the object under test, and a depth camera for acquiring a second three-dimensional image of the object under test; A posture adjustment module, used for fixing the imaging module and adjusting the posture of the imaging module; A processor is connected to and controls the imaging module and the posture adjustment module, so that the imaging module performs multi-angle image acquisition on the object to be measured, and calculates the color three-dimensional model of the object to be measured based on the first three-dimensional image, the color image and the second three-dimensional image; the color three-dimensional sensor includes an active binocular vision camera and a color camera composed of a left camera, a right camera and a projector; the first three-dimensional image of the object is a high-precision fine three-dimensional model, and the second three-dimensional image of the object is a low-precision rough three-dimensional model; The depth camera includes a TOF depth camera, a structured light depth camera, or a binocular vision depth camera; The posture adjustment module comprises: A mechanical arm connected to the imaging module, comprising a plurality of rotation axes for adjusting the position and posture of the imaging module; A rotating shaft, connected to the mechanical arm and the base respectively, and used to make the imaging module rotate around the base; The step of calculating the color three-dimensional model of the object under test based on the first three-dimensional image, the color image and the second three-dimensional image comprises the following steps: generating a global scanning viewpoint using the second three-dimensional image; Acquire the first three-dimensional image and the color image at the global scanning viewpoint, and calculate the color three-dimensional model of the object under test based on the first three-dimensional image and the color image; When the system is used to perform a three-dimensional scan of the object being measured, the overall scanning process is performed by the processor and is divided into the following steps: Step 1: Calibrate the depth camera and the color 3D sensor to obtain the internal and external parameters of the depth camera and the color 3D sensor; Step 2: Use the depth camera to collect a low-precision rough 3D model of the object to be measured; use the rotation axis and the robotic arm to control the depth camera to circle the object to be measured to generate a low-precision rough 3D model of the object; Step 3: Generate global scanning viewpoints based on low-precision rough 3D model calculations; Step 4: Perform high-precision three-dimensional scanning of the object to be measured using an active binocular vision camera according to the shortest path planning based on the generated global scanning viewpoint to obtain a first high-precision fine three-dimensional model; Calculating a confidence map of the first high-precision fine three-dimensional model, determining areas with missing data and missing details, and performing supplementary scanning to obtain a second high-precision fine three-dimensional model with higher accuracy; During the acquisition process of the first and / or second high-precision fine three-dimensional models, synchronously using a color camera to acquire a color image; The three-dimensional target consists of a first sub-target A and a second sub-target B. The calibration process is divided into two steps: (1) The rotation axis remains stationary, and the robotic arm carries a color 3D sensor to collect multi-view images of the first sub-target A and calculate the internal and external parameters of the color 3D sensor, H lm and H im ; Since the left and right cameras, the color camera and the infrared camera work under light sources of different spectral bands, in each acquisition, the left and right cameras and the color camera first acquire the target image under visible light illumination, and then illuminate the target with an infrared light source, and the infrared camera acquires the target image; (2) The robot arm keeps its posture unchanged, and the motor rotates at different angles. The left and right cameras use the principle of binocular stereo vision to reconstruct the three-dimensional coordinates of the random marking points of the second sub-target B part at each viewing angle, and determine the rotation angle by matching the marking points, thereby constructing the base coordinate system and calculating H ba ; Among them, H lm For S l The transformation matrix to M, S l is the color 3D sensor coordinate system, M is the robot arm coordinate system; H im For S i The transformation matrix to M, S i is the depth camera coordinate system; H ba is the transformation matrix from B to A, B is the robot base coordinate system, and A is the base coordinate system.

2. The automated three-dimensional color imaging and measurement system according to claim 1, characterized in that: At least one of the resolution, accuracy and frame rate of the first three-dimensional image is higher than that of the second three-dimensional image.

3. The automated three-dimensional color imaging and measurement system according to claim 1, characterized in that: The global scan viewpoint is generated based on constraint conditions, where the constraint conditions include at least one of a visibility constraint, a measurement space constraint, an overlap constraint, and an occlusion constraint.

4. The automated three-dimensional color imaging and measurement system according to claim 3, characterized in that: The global scanning viewpoint calculates the global best viewpoint NBVs based on a global optimization algorithm, and the NBVs algorithm includes the following steps: Constructing a minimum bounding box including the low-precision rough three-dimensional model and the scanning space, and dividing the scanning space into 3D voxel grids according to a certain distance interval; Find voxels for the sampling points of the low-precision rough three-dimensional model, and use the voxels as search seeds to perform expansion search on the domain voxels to obtain valid voxels; The labeling scores of the valid voxels are solved and the voxel with the largest score value is selected for viewpoint calculation.

5. The automated three-dimensional color imaging and measurement system according to claim 4, characterized in that: The three-dimensional scanning is performed based on the global scanning viewpoint and the shortest path planning is performed to perform the three-dimensional scanning along the shortest path.

6. The automated three-dimensional color imaging and measurement system according to claim 5, characterized in that: The shortest path planning algorithm is one of an ant colony algorithm, a neural network algorithm, a particle swarm algorithm, and a genetic algorithm.

Images