An automated three-dimensional color imaging and measurement method

Abstract

The present invention provides an automated three-dimensional color imaging and measurement method, including: calibrating a depth camera and a color three-dimensional sensor to obtain the internal parameters and external parameters of the depth camera and the color three-dimensional sensor; using the depth camera to obtain a rough three-dimensional model of an object, and generating a global scanning viewpoint based on the rough three-dimensional model; performing path planning on the global scanning viewpoint to obtain the shortest path, and using the color three-dimensional sensor to perform three-dimensional scanning on the object based on the shortest path to obtain a first high-precision fine three-dimensional model; calculating and determining the missing areas of the first high-precision fine three-dimensional model, and using the color three-dimensional sensor to perform supplementary scanning on the missing areas to obtain a second high-precision fine three-dimensional model. By proposing a complete automated three-dimensional scanning method, the present invention ensures the overlap and integrity of the overall data, as well as the accuracy of data acquisition under a single viewing angle.

Description

Technical Field

[0001] The present invention belongs to the field of electronic technology, and more specifically, relates to an automated three-dimensional color imaging and measurement method. Background Art

[0002] Among many optical three-dimensional measurement technologies, the active binocular vision 3D imaging technology based on phase is considered to be one of the most effective technologies for accurately detecting and reconstructing the three-dimensional shape of an object due to its non-contact, fast, and high-precision characteristics.

[0003] However, in the process of optical three-dimensional measurement and imaging, due to the limitation of the measurement range of the three-dimensional sensor, the size change and topological change of the object to be measured will both have varying degrees of impact on the complete three-dimensional measurement and imaging, especially posing a huge challenge to automated scanning: it is necessary to meet the integrity requirements of three-dimensional scanning and coordinate and control the attitude relationship between the three-dimensional sensor and the measurement surface to ensure the accuracy and efficiency of three-dimensional digital measurement.

[0004] Aiming at the huge challenges existing in current three-dimensional measurement, the present invention provides a robot-assisted fully automated three-dimensional color imaging and measurement method. Summary of the Invention

[0005] To solve the above problems, the present invention provides an automated three-dimensional color imaging and measurement method, including: calibrating a depth camera and a color three-dimensional sensor to obtain the internal parameters and external parameters of the depth camera and the color three-dimensional sensor; using the depth camera to obtain a rough three-dimensional model of an object, and generating a global scanning viewpoint based on the rough three-dimensional model; performing path planning on the global scanning viewpoint to obtain the shortest path, and using the color three-dimensional sensor to perform three-dimensional scanning on the object based on the shortest path to obtain a first high-precision fine three-dimensional model; calculating and determining the missing area of the first high-precision fine three-dimensional model, and using the color three-dimensional sensor to perform supplementary scanning on the missing area to obtain a second high-precision fine three-dimensional model.

[0006] In one embodiment, it further includes synchronously using the color three-dimensional sensor to collect color images during the acquisition process of the first and / or second high-precision fine three-dimensional models, and performing texture mapping on the color images to color the first and / or second high-precision fine three-dimensional models to obtain a three-dimensional color digital image of the object.

[0007] In one embodiment, the supplementary scanning refers to generating supplementary scanning viewpoints by constructing a model confidence map and combining a viewpoint planning algorithm, and scanning the object at the supplementary scanning viewpoints.

[0008] In one embodiment, the global scanning viewpoints are calculated based on the three-dimensional information of the rough model and the intrinsic spatial constraints of the three-dimensional sensor.

[0009] In one embodiment, the global scanning viewpoints are generated based on constraint conditions, which include but are not limited to at least one of visibility constraint, measurement space constraint, overlap degree constraint, and occlusion constraint.

[0010] In one embodiment, the global scanning viewpoints are used to calculate the globally optimal viewpoints (NBVs) based on a global optimization algorithm.

[0011] In one embodiment, the NBVs algorithm includes the following steps: constructing a minimum bounding box containing the rough three-dimensional model and the scanning space, and performing 3D voxel grid division on the scanning space at a certain distance interval; finding voxels for the sampling points of the rough three-dimensional model, and using the voxels as search seeds to perform dilation search on their neighboring voxels to obtain the effective voxels; solving the label scores of the effective voxels and selecting the voxel with the maximum score value for viewpoint calculation.

[0012] In one embodiment, the three-dimensional scanning is based on the global scanning viewpoints and performs the shortest path planning to perform three-dimensional scanning along the shortest path.

[0013] In one embodiment, the shortest path planning algorithm includes but is not limited to one of ant colony algorithm, neural network algorithm, particle swarm algorithm, and genetic algorithm.

[0014] In one embodiment, the rough three-dimensional model includes a three-dimensional point cloud model.

[0015] In one embodiment, the fine three-dimensional model includes a three-dimensional color point cloud model.

[0016] The beneficial effects of the present invention: A robot three-dimensional scanning strategy based on depth camera assistance is proposed, and a complete automated three-dimensional scanning method is proposed, which ensures the overlap degree and integrity of the overall data, as well as the accuracy of data acquisition under a single view. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a schematic diagram of a three-dimensional color digitization system according to an embodiment of the present invention.

[0018] Figure 2 is a schematic diagram of the system coordinate system distribution and transformation relationship according to an embodiment of the present invention.

[0019] Figure 3 is a schematic diagram of a low-cost stereo target based on non-coded fiducial points according to an embodiment of the present invention.

[0020] Figure 4 Schematic diagram of constraint relationships of a binocular vision 3D sensor according to an embodiment of the present invention.

[0021] Figure 5 Schematic diagram of the visible range of ISO points (a) and the voxels containing viewpoints (b) according to an embodiment of the present invention.

[0022] Figure 6 Schematic diagram of the histogram statistics of in-voxel vectors according to an embodiment of the present invention.

[0023] Figure 7 Flowchart of the NBVs algorithm according to an embodiment of the present invention. Detailed implementation manners

[0024] The present invention will be further described in detail below in conjunction with the detailed implementation manners and with reference to the accompanying drawings. It should be emphasized that the following description is merely exemplary and is not intended to limit the scope of the present invention and its applications.

[0025] System description

[0026] Figure 1 Schematic diagram of a 3D color digitization system according to an embodiment of the present invention. The system 10 includes a base 101, a robotic arm 102, an imaging module 103, a rotating shaft 105, and a processor (not shown in the figure).

[0027] The base 101 is used to place the object to be measured 104. The base may not be an essential configuration of the system. For example, it may be other planes or structures.

[0028] The imaging module 103 includes a color three-dimensional sensor and a depth camera 1035. The color three-dimensional sensor includes an active binocular vision camera composed of a left camera 1031, a right camera 1032, and a projector 1033, as well as a color camera 1034, which are respectively used to collect the first three-dimensional image and the color image of the object to be measured 104. By using the relative position information between each camera (obtained through calibration), the first three-dimensional image can be further aligned with the color to obtain the three-dimensional color image of the object to be measured. Or, the color image collected by the color camera is texture-mapped to color the three-dimensional image to obtain the three-dimensional color image. In one embodiment, the left camera 1031 and the right camera 1032 are high-resolution black-and-white cameras, and the projector can be a digital stripe projector for projecting a coded structured light image. The left camera 1031 and the right camera 1032 collect phase-structured light images and perform high-precision three-dimensional imaging based on the phase-assisted active stereo vision (PAAS) technology. In one embodiment, the left and right cameras can also be infrared cameras, etc. The parameters of the left and right cameras, such as focal length, resolution, depth of field, etc., can be the same or different. The first three-dimensional image refers to the three-dimensional image of the object to be measured 104 collected by the color three-dimensional sensor.

[0029] The depth camera 1035 is used to collect the second three-dimensional image of the object to be measured. The depth camera 1035 can be a depth camera based on the time-of-flight (TOF) method, structured light, or passive binocular vision technology. Generally, at least one of the resolution, accuracy, and frame rate of the collected second three-dimensional image is lower than that of the first three-dimensional image. Generally speaking, the resolution, accuracy, and frame rate of the second three-dimensional image are all lower than those of the first three-dimensional image. For the convenience of description, in the following description, the first three-dimensional image of the object is called a high-precision fine three-dimensional model, and the second three-dimensional image of the object is called a low-precision rough three-dimensional model. The second three-dimensional image refers to the three-dimensional image of the object to be measured 104 collected by the depth camera 1035.

[0030] The robotic arm 102 and the rotating shaft 105 form a pose adjustment module, which is used to fix the imaging module 103 and adjust its pose. Among them, the robotic arm 102 is connected to the imaging module 103 and the rotating shaft 105, and the rotating shaft 105 is installed on the base 101 for rotating around the base 101. The robotic arm 102 is a multi-axis linkage robotic arm for performing corresponding pose adjustments. Through the combined adjustment of the rotating shaft 105 and the robotic arm 102, the imaging module 103 can be transformed in multiple azimuth perspectives to facilitate multi-azimuth measurement of the object to be measured 104. In some embodiments, the rotating shaft 105 includes a rotating motor. Driven by the rotating motor, the robotic arm will rotate around the base under the drive of the rotating shaft to measure the object.

[0031] The processor is connected to the robotic arm 102, the imaging module 103, and the rotating shaft 105, and is used to execute control and corresponding data processing or 3D scanning tasks, such as 3D color image extraction, rough 3D model establishment, fine 3D model establishment, etc. It can be understood that the processor can be a single processor or multiple independent processors. For example, the imaging module can include multiple dedicated processors for executing algorithms such as 3D imaging. The system also includes a memory for storing the algorithm programs executed by the processor, such as various algorithms and methods mentioned in the present invention (calibration method, reconstruction method, view point generation algorithm, scanning method, etc.). The memory can be various computer-readable media, such as non-transitory storage media, including magnetic media and optical media, such as disks, tapes, CD-ROMs, RAM, ROM, etc.

[0032] It can be understood that the above-mentioned 3D image can refer to a depth image, or point cloud data, mesh data, or 3D model data, etc. obtained by further processing the depth image.

[0033] When using the system 10 to perform 3D scanning on the object to be measured 104, the overall scanning process is executed by the processor and is divided into the following steps:

[0034] The first step: Calibrate the depth camera 1035 and the color 3D sensor to obtain the internal parameters and external parameters of the depth camera 103 and the color 3D sensor. The specific process will be described in detail later.

[0035] The second step: Use the depth camera 1035 to collect a low-precision rough 3D model of the object to be measured 104. For example, use the rotating shaft 105 and the robotic arm 102 to control the depth camera 1035 to surround the object to be measured 104 for one week to quickly generate a low-precision rough 3D model of the object. It can be understood that the object to be measured 104 needs to be placed on the base 101 in advance. In one embodiment, the object to be measured 104 is placed at the center of the base 101.

[0036] The third step: Calculate and generate global scanning viewpoints based on the low-precision rough 3D model. Specifically, the global scanning viewpoints will be automatically generated according to the NBVs algorithm proposed in the present invention.

[0037] The fourth step: Perform high-precision 3D scanning on the object to be measured 104 using the active binocular vision camera for the generated global scanning viewpoints according to the shortest path planning to obtain the first high-precision fine 3D model.

[0038] In some embodiments, it is also necessary to calculate the confidence map for the first high-precision fine 3D model, determine the areas with missing data and missing details, and perform supplementary scanning to obtain a second high-precision fine 3D model with higher accuracy.

[0039] In some embodiments, during the acquisition process of the first and / or second high-precision fine three-dimensional model, a color camera is synchronously used to acquire color images, and the color images are texture-mapped to color the fine three-dimensional model to obtain a three-dimensional color digital image, ultimately realizing the high-fidelity complete object three-dimensional color digitization.

[0040] System calibration

[0041] Before using the system 10 to perform three-dimensional scanning on the object to be measured 104, it is necessary to calibrate each component in the system to obtain the relative position relationship between the coordinate systems where each component is located. Based on the relative position relationship, corresponding operations can be performed, such as color coloring, generating a global scanning viewpoint based on the rough three-dimensional model, etc.

[0042] Figure 2 It is a schematic diagram of the system coordinate system distribution and transformation relationship according to an embodiment of the present invention. Among them, the world coordinate system is established on the base coordinate system, and the color three-dimensional sensor coordinate system is established on the left camera S l above, and the depth camera coordinate system is established on the internal infrared camera S i above. It is necessary to determine the internal and external parameters of the color three-dimensional sensor and the transformation matrix of the color three-dimensional sensor / depth camera coordinate system - robotic arm coordinate system - robotic arm base coordinate system - base coordinate system through system calibration. The difficulty in system calibration in the present invention is that there are both sensors with different resolutions and different field-of-view ranges (such as a color camera with 20 million pixels, left and right cameras with 5 million pixels, the FOV of the lens is H39.8°, V27.6°; a depth camera with 300,000 pixels, FOV is H58.4°, V45.5°), and sensors with different spectral response ranges (such as the spectral response ranges of color cameras and black-and-white cameras are in the visible light band; the response range of infrared cameras is in the infrared band). At the same time, to ensure the calibration accuracy of the color three-dimensional sensor, designing and manufacturing a high-precision stereo target is the key to performing high-precision calibration.

[0043] Figure 3Schematic diagram of a low-cost stereo target based on non-coded fiducial points according to an embodiment of the present invention. The stereo target consists of a first sub-target A and a second sub-target B. The first sub-target A is partially composed of a plane, and the surface of the plane is regularly arranged (such as 11×9) non-coded fiducial points. The accurate spatial coordinates of these fiducial points can be determined by bundle adjustment technology. The fiducial points include reference points and positioning points, and there are at least four positioning points. In order to improve the fiducial point extraction accuracy of a low-resolution depth camera, both the reference points and the positioning points adopt a large circle design. Inside the positioning points and the reference points, there is a small black concentric fiducial point (such as a concentric circle). The center gray level of the fiducial point is used to distinguish the positioning points and the reference points (for example, if the center gray level is greater than 125, it is a reference point, and if it is less than 125, it is a positioning point, that is, the center gray levels of the reference point and the positioning point are different), as shown in Figure 3 (c). Such a design greatly increases the size of the reference point and improves the positioning accuracy of the positioning point at the same time; the second sub-target B is composed of multiple planes, and non-coded fiducial points are randomly pasted on the surface for rotation axis calibration. The calibration process is to reconstruct the spatial coordinates of the random fiducial points under multiple perspectives by a color three-dimensional sensor surrounding the stereo target for one week, and determine the base coordinate system through fiducial point matching optimization. Therefore, it is not necessary to pre-determine the spatial coordinates of the random fiducial points of the second sub-target B, which greatly reduces the difficulty and cost of target production.

[0044] The calibration process is divided into two steps: (1) The rotation axis (rotation motor) remains stationary, and the robotic arm carries the color three-dimensional sensor to collect multi-perspective images of the first sub-target A, and calculates the internal and external parameters of the color three-dimensional sensor, H lm and H im . Since the left and right cameras, color camera, and infrared camera work under light sources in different spectral bands, in each acquisition, first, under visible light illumination, the left and right cameras and the color camera collect target images, and then illuminate with an infrared light source, and the infrared camera collects target images; (2) The robotic arm keeps its posture unchanged, the motor rotates at different angles, and the left and right cameras use the binocular stereo vision principle to reconstruct the three-dimensional coordinates of the random marked points of the target B part in each perspective, determine the rotation angle through marked point matching, thereby constructing the base coordinate system, and calculate H ba .

[0045] In one embodiment, when the color three-dimensional sensor is calibrated, the three cameras (left, right, and infrared cameras) simultaneously acquire the target patterns under different perspectives respectively, and construct the objective function of the single camera calibration model:

[0046]

[0047] where represents the homogeneous spatial coordinate of the j-th fiducial point among the M fiducial points in the target coordinate system, x ij(i = 1,...N) represents the image coordinates of the j-th fiducial point in the image collected by the camera at the i-th viewing angle. K is the internal parameter matrix of the camera, including the focal length, the position of the principal point, and the skew factor. ε is the lens distortion. In this paper, only the typical fifth-order lens distortion is considered. represents the transformation matrix from the target coordinate system to the camera coordinate system at the i-th viewing angle.

[0048] Generally, let the left camera coordinate system be the three-dimensional sensor coordinate system. Then the structural parameters of the three cameras are:

[0049]

[0050] Among them, and are the rotation matrix and translation vector from the left camera S l to the right camera S r respectively. and are the rotation matrix and translation vector between the left camera S l and the color camera S c respectively. To obtain higher-precision structural parameters, we add the transformation matrix to the non-linear objective function of the three cameras and minimize the objective function through the Gauss-Newton or Levenberg-Marquardt method to achieve camera parameter estimation:

[0051]

[0052] where τ = {ε l , ε r , ε c , K l , K r , K c , H lr , H lc}, From this, the internal and external parameters of the color three-dimensional sensor can be obtained. The parameter solution of the infrared camera is similar.

[0053] After the calibration of the color three-dimensional sensor is completed, the transformation matrix of the left camera at each acquisition viewing angle can be obtained is directly given by the robotic arm control system. According to the mathematical model of hand-eye calibration, the following relationship is established:

[0054]

[0055] where i, k = 1, 2,..., N, and i ≠ k. N is the number of scans. equations can be established for N motion postures. According to Tsai's method

[30] , the linear least squares solution method can be used to solve H sg and Hcb 。

[0056] In one embodiment, to further improve the accuracy, we use it as the initial value to establish a non - linear objective function:

[0057]

[0058] where can be obtained in real - time from the robotic arm. By using the Levenberg - Marquardt method to minimize the objective function, a higher - accuracy H can be obtained lm and H bt 。H im The solution of which is similar and will not be elaborated here.

[0059] In one embodiment, during the calibration process of the rotation axis, the robotic arm maintains a constant pose. Denote the transformation matrix from the robotic arm to the base at this time as H′ gb , and the 3D sensor moves in a circular motion around the stereo target. Random fiducial points of part B of the target are reconstructed at different rotation angles T is the number of rotations, j is the fiducial point serial number. For the fiducial points reconstructed in all fields of view perform global matching optimization to obtain the transformation relationship [R (m) |T (m) of the target fiducial points at each rotation angle. Then, the rotation axis direction vector can be calculated under the constraint of the distance between every two closed - circle trajectory planes, and the center of each circle trajectory can be obtained through the global least - squares optimization method. Thus, the transformation relationship H rl from the 3D sensor coordinate system to the base coordinate system can be determined. According to the transformation relationship (H rl ) -1 = H br H′ mb H lm the transformation relationship H br from the base coordinate system to the base coordinate system can be obtained.

[0060] Global scanning viewpoint generation

[0061] According to the stereo vision imaging model, limited by the binocular camera angle (FOV), the focal lengths and depths of field (DOF) of the camera lens and the digital projection lens, the measurement space of the 3D sensor is limited, and the quality of the 3D - reconstructed point cloud is also affected by many constraint conditions. Based on certain constraint conditions, the present invention automatically generates a series of scanning viewpoints by analyzing the rough 3D model (Roughmodel) to achieve complete 3D digital color imaging of the object with the least number of viewpoints. Next, the constraint conditions and the viewpoint generation method will be introduced respectively.

[0062] Figure 4Schematic diagram of the constraint relationship of a binocular vision 3D sensor according to an embodiment of the present invention. Among them Figure 4 (a) is a schematic diagram of the basic structure of a binocular sensor and the measurement space, Figure 4 (b) is the measurement space constraint of the 3D sensor, Figure 4 (c) is the point cloud visibility constraint. For simplicity of description, the present invention does not expand the description of the calculation of the specific view volume, and the measurement space is simplified to Figure 4 (b) shown. Let the working distance range of the 3D sensor be [d n , d f . The maximum field of view The viewpoint position is v i (x, y, z), and v i (α, β, γ) represents the unit vector of the optical axis direction of the 3D sensor. v ik = d(v i , s k ) represents the vector from the viewpoint position v i pointing to the position s k of the measurement target point. The process of viewpoint planning is affected by the object surface space, the viewpoint space, and the imaging work space, and its constraint conditions mainly include but are not limited to at least one of the following aspects:

[0063] 1) Visibility constraint: It represents the angular range within which the measurement target point is allowed to be collected by the sensor. Let the normal vector of the measurement target point p k be n k , then the visibility constraint condition

[0064]

[0065] where represents the maximum visible angular range of the measurement target point, as shown in ​ (c).

[0066] 2) Measurement space constraint: It includes the field of view (FOV) constraint and the depth of field (DOF) constraint, representing the measurable range of the 3D sensor. Its constraint condition is

[0067]

[0068] where φ max represents the maximum field of view angle of the 3D sensor, as shown in ​ (b).

[0069] 3) Overlap Constraint: For subsequent ICP matching and grid integration of multi-view depth data, there needs to be a certain field of view overlap between adjacent scanning fields of view. Define the field of view overlap as W and W cover represent the total area of the field of view and the overlapping area respectively, and its constraint condition is

[0070] ξ≥ξ min (8)

[0071] where ξ min is the minimum field of view overlap.

[0072] 4) Occlusion Constraint: When the line segment d(v i to the measurement target point s k , s i ) intersects with the object entity, it means that the viewing direction v k of the view point v i at the target point s k is occluded. ik

[0073] For an object with an unknown shape, first use a depth camera to perform an initial scan around the object to be measured to generate a rough 3D model. The purpose of this step is to generate a global scanning perspective using this model. Therefore, the rough 3D model does not require too high precision and resolution, nor does it require particularly complete scan data. In addition, since depth cameras generally have characteristics such as a wide scanning field of view, a large depth range in the measurement space, and good real-time performance, for most objects of different sizes and different surface materials, a simple set of scanning postures can be preset to achieve the initial scan of the object's morphology.

[0074] In one embodiment, during the initial scan process, use a matching and fusion algorithm, such as the kinectFusion algorithm, to perform real-time matching and integration of the data. After the initial scan is completed, perform preprocessing on the original point cloud, such as noise filtering, smoothing, edge removal, and normalization estimation, and then generate an initial closed triangular mesh model. Perform Poisson-disk sampling on this model to obtain the so-called ISO points, as ​ (b) shown. Let the model sampling points be

[0075] According to the size of the initial model and the maximum working distance d of the scanner f, construct the minimum bounding box S that contains the model and the scanning space, and divide this space into a 3D voxel grid at a certain distance interval ΔD (for example, divided into 100×100×100 voxels). For any spatial point (p x , p y , p z ) in S, it can be quickly solved which voxel grid this point belongs to according to Equation (9).

[0076]

[0077] Among them, (p x-min , p y-min , p z-min ) is the minimum coordinate value of the bounding box S, and v i = (n x , n y , n z ) is the voxel number value. The center point of the voxel will be used as a three-dimensional spatial point to participate in the calculation of the next best views (NBVs) below. The NBVs algorithm in this article mainly consists of three steps:

[0078] Step1: For the initial model sampling point s k , along its normal direction n k , at a distance of d0 = (d n + d f ) / 2, according to Equation (9), the voxel v i can be found. Using v i as the search seed, use the greedy algorithm to perform an expanding search on the neighboring voxels. According to the visibility constraint described above, record the voxel numbers that satisfy Equation (10) in the associated set k of the sampling point s , as shown in ​ (a).

[0079]

[0080] Among them, v ik = d(v i , s k ) represents the vector from point v i to point s k , and w ik (v i , s k ) = 1 means that s k is visible to v i . When w ik (v i , s k ) = 0, it means that s k to vi There is occlusion between them. When recording , (s k , v ik ) is also recorded in all v i that satisfy Equation (10) , that is For all ISO points {s k}, perform step1 to obtain all valid voxels {v i} that record the ISO points, and the voxels that are not recorded are regarded as invalid and no longer participate in the operation.

[0081] Step2: For the valid voxel v i , according to the marking function g(s ) of the element s k in its set k , solve the marking score of this voxel

[0082]

[0083] g(s k ) marks the usage of s k . When s k has not been confirmed to belong to a certain scanning view point, it is marked as 1, and when it has been confirmed to belong to a certain scanning view point, it is marked as 0, that is

[0084]

[0085] Step3: Select the voxel with the largest marking score value for view point calculation. The ISO points recorded by a voxel may not be covered by the same scanning range. As shown in ​ (b), so we use the method of histogram statistics to all s k vector d(v i , s k ) in is statistically analyzed and selected. According to the conversion relationship between the Cartesian coordinate system (x, y, z) and the spherical coordinate system , the vector d(v i , s k ) is converted to the spherical coordinate system. The X-axis and Y-axis in the histogram are θ and respectively, and the Z-axis is the statistical quantity of the iso point. As shown in ​ . According to the scanning field of view angle constraint φ max and the overlap degree constraint ξ min of the three-dimensional sensor, determine the size φ filter of the filter window in the XY plane = φ max (1 - ξ min) The filter traverses all elements (x, y) of the histogram XY plane and sums the number of iso points within the filter. When the statistic within the filtering window is the largest, the iso points contained within the filter are {s′ k} k∈N , where N is the number of s k ′ for which the marking weight g(s′ k ) = 1. For the vector d(v k s i k , s k ) of s′

[0086]

[0087] At this point, the spatial position of the viewing point and the viewing direction vector can be obtained

[0088] Step4: Set the marking function g(s′ k ) in {s′ k} to 0.

[0089] Repeat Step2 - Step4 until the marking scores of all voxels are lower than the threshold. The flowchart of the NBVs algorithm is shown in the figure. It can be seen from the above algorithm process that the valid voxels contain all iso points that meet the constraint conditions. The higher the marking score of a voxel, the more object surface range can be covered by the viewing point calculated from this voxel, that is, the more important this viewing point is. In this paper, the viewing point is calculated by selecting the voxel with the largest marking score, and the finally generated list of viewing points is also sorted in descending order according to the number of iso points covered by the viewing point.

[0090] Automatic 3D scanning and supplementary scanning

[0091] Through the NBVs algorithm described above, the spatial positions and directions of a series of viewing points are obtained. How to achieve scanning of all viewing points with the shortest path belongs to the path planning problem. Algorithms for solving the path planning problem include but are not limited to ant colony algorithm, neural network algorithm, particle swarm algorithm, genetic algorithm, etc., each with its own advantages and disadvantages. For example, in one embodiment, the ant colony algorithm is used to solve the viewing point set to obtain the shortest path. Next, a color 3D sensor is used to perform 3D scanning along the shortest path. The high-precision depth data (left camera coordinate system) collected under each viewing angle is transformed to the world coordinate system through the coordinate transformation relationship, and finally, real-time matching of multi-view depth data is achieved to calculate the high-precision fine 3D model of the object.

[0092] In one embodiment, the 3D sensor performs 3D scanning along the shortest path. During each viewpoint transformation process, it involves the joint control of the rotating motor and the robotic arm, which is essentially a problem of the transformation of each sensor coordinate system. Let two adjacent scanning viewpoints be V i and V j . The transformation matrix of these two viewpoints represents the transformation relationship of the infrared depth sensor coordinate system from viewpoint V i to V j in the world coordinate system. In order to adjust the 3D sensor from viewpoint V i to V j , the projection coordinates v i and v j of viewpoints V a Y a plane in the rotation axis coordinate system (i.e., the world coordinate system) are respectively obtained, so as to obtain the rotation angle θ i⊥ and v j⊥ of the rotating motor, and thus obtain the transformation matrix ij and the transformation matrix

[0093]

[0094] The viewpoint after the motor rotates Then the transformation matrix from viewpoint V i ′ to V j is Since the transformation matrix H im and the transformation matrix i ′ of the robotic arm at viewpoint V [[ID=False]]are known, the following transformation relation can be established:

[0095]

[0096] Thus, the transformation matrix of the robotic arm at viewpoint V j is obtained. By combining the rotation angle θ ij of the rotating motor and the rotation matrix of the robotic arm, the attitude adjustment of the 3D sensor at different viewpoints can be realized. The high-precision depth data (in the left camera coordinate system) at each viewpoint is converted to the world coordinate system through the transformation matrix of the infrared depth sensor coordinate system and the transformation matrix of the viewpoint, so as to realize the real-time matching of multi-view depth data.

[0097] As can be seen from Equation (10), the viewpoint planning algorithm in this paper has considered the self-occlusion of objects. However, during the actual scanning process, due to factors such as the surface material of the object, there will inevitably be some data missing, or the point cloud data is sparse and of low quality. More importantly, since the rough three-dimensional model used for viewpoint planning loses the detailed information of the object, the generated viewpoints do not take into account the fine scanning of the geometric details.

[0098] Therefore, in one embodiment, a method of constructing a model confidence map is used to reflect the missing part of the original data and the missing detail area, and combined with the viewpoint planning algorithm to generate viewpoints for supplementary scanning. Poisson-disk sampling is performed on the original point cloud data obtained in the previous high-precision scanning stage to generate IS0 sampling points Generate the iso point s according to Equation (16) k Confidence map of:

[0099] f(s k ) = f g (s k , n k ) f s (s k , n k ) (16)

[0100] Where f g (s k , n k ) = Γ(s k ) · n k Is defined as the completeness confidence score, Γ(s k ) is the scalar field gradient at point s k , n k Is the normal vector. f g (s k , n k ) has been obtained during the Poisson-disk sampling process, so no additional computational effort is required; f s (s k , n k ) is the smoothness confidence score K(smoothness confidence score), satisfying

[0101]

[0102]

[0103]

[0104] Where || g || is the l2-norm is the point s k The K - neighborhood range Ω k of the original point cloud within it, The spatial weight function θ(||s k - q j ||) decays rapidly with the increase of the radius within Ω k ; the orthogonal weight function φ(n k , q j - s k ) reflects the distance from the original point q k within the K - neighborhood range Ω j to the tangent plane at the iso - point. When the smoothing confidence score value is high, the surface at point s k is locally smooth and the scanning quality is relatively high; when the smoothing confidence score value is low, it indicates that the local original scanning data at point s k is sparse, or there are more high - frequency components in the original scanning data, such as point cloud noise or rich in geometric details, etc., and more supplementary scans are needed.

[0105] The confidence score effectively reflects the quality and fidelity of the scanned model's point cloud data. We use the model confidence score to guide the view - point planning in the supplementary scanning process. Set the confidence score threshold ε, and find the range S′ = {s′ k |f(s′ k ) ≤ ε}, and calculate the view - points for S′ through the previous algorithm. Different from the NBVs algorithm mentioned above, g(s′ k ) is assigned according to the confidence score of s′ k

[0106]

[0107] Therefore, the score of the voxel no longer reflects the number of iso - points it contains, but the sum of the confidence scores of the iso - points. Calculating the view - point for the voxel with the highest confidence score will make the view - point focus more on scanning the missing parts and the parts rich in geometric details.

[0108] The above content is a further detailed description of the present invention in combination with specific / preferred embodiments. It cannot be determined that the specific implementation of the present invention is only limited to these descriptions. For those of ordinary skill in the technical field to which the present invention belongs, without departing from the concept of the present invention, they can make several substitutions or modifications to these described embodiments, and these substitution or modification methods should all be regarded as belonging to the protection scope of the present invention.​

Claims

1. An automated three-dimensional color imaging and measurement method, characterized in that, Including: Calibrating a depth camera and a color three-dimensional sensor to obtain the internal and external parameters of the depth camera and the color three-dimensional sensor; the color three-dimensional sensor includes an active binocular vision camera composed of a left camera, a right camera, and a projector, and a color camera; Using the depth camera to obtain a rough three-dimensional model of an object and generating global scanning viewpoints based on the rough three-dimensional model; Using a rotating axis and a robotic arm to control the depth camera to circle around the object to be measured once to generate a rough three-dimensional model of the object; Performing path planning on the global scanning viewpoints to obtain the shortest path, and using the active binocular vision camera in the color three-dimensional sensor to perform three-dimensional scanning on the object based on the shortest path to obtain a first high-precision fine three-dimensional model; Calculating and determining the missing areas of the first high-precision fine three-dimensional model, and using the color three-dimensional sensor to perform supplementary scanning on the missing areas to obtain a second high-precision fine three-dimensional model; The method further includes synchronously using the color camera in the color three-dimensional sensor to collect color images during the acquisition process of the first and / or second high-precision fine three-dimensional models, and performing texture mapping on the color images to color the first and / or second high-precision fine three-dimensional models to obtain a three-dimensional color digital image of the object.

2. The automated three-dimensional color imaging and measurement method according to claim 1, wherein The supplementary scanning refers to: generating supplementary scanning viewpoints by constructing a model confidence map and combining a viewpoint planning algorithm, and scanning the object at the supplementary scanning viewpoints.

3. The automated three-dimensional color imaging and measurement method according to claim 1, characterized in that The point cloud spacing, model details, and geometric accuracy of the fine three-dimensional model are higher than those of the rough three-dimensional model.

4. The automated three-dimensional color imaging and measurement method according to claim 1, characterized in that The global scanning viewpoints are generated based on constraint conditions, and the constraint conditions include at least one of visibility constraint, measurement space constraint, overlap degree constraint, and occlusion constraint.

5. The automated three-dimensional color imaging and measurement method according to claim 1, characterized in that The global scanning viewpoints are calculated based on a global optimization algorithm to obtain the global best viewpoints NBVs.

6. The automated three-dimensional color imaging and measurement method according to claim 5, wherein The NBVs algorithm includes the following steps: Constructing a minimum bounding box including the rough three-dimensional model and the scanning space, and performing 3D voxel grid division on the scanning space at a certain distance interval; Finding the voxels for the sampling points of the rough three-dimensional model, and performing dilation search on the neighboring voxels with the voxels as search seeds to obtain effective voxels; Solving the label scores of the effective voxels and selecting the voxel with the maximum score value for viewpoint calculation.

7. The automated three-dimensional color imaging and measurement method according to claim 1, characterized in that The three-dimensional scanning is based on the global scanning viewpoints and performs the shortest path planning to perform three-dimensional scanning along the shortest path, and the shortest path planning algorithm includes one of an ant colony algorithm, a neural network algorithm, a particle swarm algorithm, and a genetic algorithm.

8. The automated three-dimensional color imaging and measurement method according to claim 1, characterized in that, The rough three-dimensional model includes a three-dimensional point cloud model.

9. The automated three-dimensional color imaging and measurement method according to claim 1, wherein The fine three-dimensional model includes a three-dimensional color point cloud model.

Images