A viewpoint planning method based on three-dimensional model

Abstract

The present invention provides a viewpoint planning method based on a three-dimensional model, comprising the steps of: using the initial sampling point s of the three-dimensional model k Find the voxel v i , and the voxel v i Search for a valid voxel set {v i}; S2, using the marking function g(s k ) solves the voxel set {v i}; S3, select the voxel with the largest labeling score to perform viewpoint calculation; S4, set the labeling function to 0, and repeat steps S2 to S4 until the labeling scores of all voxels are lower than the threshold. The present invention automatically generates a series of scanning viewpoints based on a rough three-dimensional model under certain constraints, and can achieve three-dimensional digital color imaging of a complete object with a minimum number of viewpoints in three-dimensional fine scanning.

Description

Technical Field

[0001] The invention belongs to the field of electronic technology, and more specifically, relates to a viewpoint planning method based on a three-dimensional model. Background Art

[0002] Among many optical three-dimensional measurement technologies, phase-based active binocular vision 3D imaging technology is considered to be the most effective technology for accurately detecting and reconstructing the three-dimensional shape of objects due to its non-contact, fast and high-precision characteristics.

[0003] However, in the process of optical 3D measurement and imaging, due to the limitation of the measurement range of the 3D sensor, the size change and topological change of the object to be measured will have different degrees of impact on the complete 3D measurement and imaging, especially posing a huge challenge to automated scanning: it is necessary to meet the integrity requirements of 3D scanning, and coordinate and control the posture relationship between the 3D sensor and the measurement surface to ensure the accuracy and efficiency of 3D digital measurement.

[0004] In view of the huge challenges in current three-dimensional measurement, the present invention provides a viewpoint planning method based on a three-dimensional model, which provides a prerequisite for high-precision three-dimensional color digital imaging. Summary of the invention

[0005] To solve the above problems, the present invention provides a viewpoint planning method based on a three-dimensional model, which is characterized by comprising the following steps:

[0006] S1, using the initial sampling point s of the three-dimensional model k Find the voxel v i , and the voxel v i Search for a valid voxel set {v i};

[0007] S2, using the labeling function g(s k ) solves the voxel set {v i The labeling score of each voxel in of;

[0008] S3, selecting the voxel with the largest labeling score to perform viewpoint calculation;

[0009] S4, set the labeling function to 0, and repeat steps S2 to S4 until the labeling scores of all the voxels are lower than the threshold.

[0010] In one embodiment, the step S1 further comprises:

[0011] For the initial sampling point s k , along its normal n k , the distance is d0=(d n +df ) / 2, the voxel v can be found according to the following formula i :

[0012]

[0013] Wherein, ΔD refers to the 3D voxel grid division of the three-dimensional model according to the distance interval of ΔD; (p x ,p y ,p z ) refers to the coordinates of a point in three-dimensional space, (p x-min,py-min,pz-min ) is the minimum coordinate value of the space bounding box S, v i =(n x ,n y ,n z ) is the voxel number value.

[0014] In one embodiment, the search in step S1 is performed by dilating the neighborhood voxels using a greedy algorithm and calculating according to visibility constraints.

[0015] In one embodiment, the visibility constraint indicates the angle range within which the measurement target point is allowed to be collected. Suppose the measurement target point p k The normal vector is n k , then the visibility constraint is:

[0016]

[0017] in Indicates the maximum viewing angle range of the measurement target point.

[0018] In one embodiment, the marking function g(s k ) is used to mark the sampling point s k The usage of s k When it has not been confirmed to belong to a certain scanning viewpoint, it is marked as 1, and when it has been confirmed to belong to a certain scanning viewpoint, it is marked as 0.

[0019] In one embodiment, the viewpoint calculation includes calculating the spatial position of the viewpoint.

[0020] In one embodiment, the viewpoint calculation further includes calculating a direction vector of the viewpoint.

[0021] In one embodiment, the calculation of the direction vector comprises the following steps:

[0022] Using histogram statistics for all sampling points s k The loss d(v i ,s k ) to conduct statistics and selection;

[0023] The loss d(v i ,s k ) Convert from Cartesian coordinates (x, y, z) to spherical coordinates Down;

[0024] Determine the size of the filter window in the XY plane φ filter =φ max (1-ξ min ), and use the filter to traverse all elements (x, y) of the histogram XY plane and calculate the sum of the number of sampling points in the filter. When the statistic in the filter window is the largest, the sampling points contained in the filter are {s′ k} k∈N , N is the label weight g(s′ k )=1 k The number of s′ k The vector d(v i ,s k ) find the mean value as the scanning viewpoint direction

[0025]

[0026] In one embodiment, the filter window is based on the measurement space constraint φ max and overlap constraint ξ min The measurement space constraints include field of view (FOV) constraints and depth of field (DOF) constraints, and the constraint conditions are:

[0027]

[0028] where φ max Indicates the maximum field of view of the 3D sensor;

[0029] The overlap constraint means that there must be a certain overlap between adjacent scanning fields of view. The field overlap is defined as W and W cover Represent the total area of ​​the field of view and the area of ​​the overlapping part respectively, and the constraints are

[0030] ξ≥ξ min

[0031] where ξ min is the minimum field of view overlap.

[0032] Beneficial effects of the present invention: A global NBVs algorithm is proposed, which can automatically generate a series of scanning viewpoints under certain constraints based on a rough three-dimensional model, and can achieve three-dimensional digital color imaging of a complete object with a minimum number of viewpoints in three-dimensional fine scanning. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 Schematic diagram of a three-dimensional color digitization system according to an embodiment of the present invention.

[0034] Figure 2 Schematic diagram of the distribution and transformation relationship of the system coordinate system according to an embodiment of the present invention.

[0035] Figure 3 Schematic diagram of a low-cost three-dimensional target based on non-coding markers according to an embodiment of the present invention.

[0036] Figure 4 Schematic diagram of the constraint relationship of a binocular vision three-dimensional sensor according to an embodiment of the present invention.

[0037] Figure 5 Schematic diagram of ISO point visible range (a) and voxel inclusion viewpoint (b) according to an embodiment of the present invention.

[0038] Figure 6 is a statistical diagram of a voxel intra-vector histogram according to an embodiment of the present invention.

[0039] Figure 7 is a flow chart of an NBVs algorithm according to an embodiment of the present invention. DETAILED DESCRIPTION

[0040] The present invention will be further described in detail below in conjunction with specific implementations and with reference to the accompanying drawings. It should be emphasized that the following description is merely exemplary and is not intended to limit the scope of the present invention and its application.

[0041] System Description

[0042] Figure 1 Schematic diagram of a three-dimensional color digitization system according to an embodiment of the present invention. The system 10 includes a base 101, a mechanical arm 102, an imaging module 103, a rotating shaft 105, and a processor (not shown in the figure).

[0043] The base 101 is used to place the object to be measured 104 , and the base may not be a necessary configuration of the system, for example, it may be other planes or structures.

[0044] The imaging module 103 includes a color three-dimensional sensor and a depth camera 1035. The color three-dimensional sensor includes an active binocular vision camera composed of a left camera 1031, a right camera 1032, and a projector 1033, and a color camera 1034, which are respectively used to collect a first three-dimensional image and a color image of the object to be measured 104. By the relative position information between each camera (obtained by calibration), the first three-dimensional image can be further aligned with the color to obtain a three-dimensional color image of the object to be measured, or the color image collected by the color camera is texture mapped to achieve coloring of the three-dimensional image to obtain a three-dimensional color image. In one embodiment, the left camera 1031 and the right camera 1032 are high-resolution black and white cameras, and the projector can be a digital stripe projector for projecting coded structured light images. The left camera 1031 and the right camera 1032 collect phase structured light images and perform high-precision three-dimensional imaging based on phase-assisted active stereo vision (PAAS) technology. In one embodiment, the left and right cameras can also be infrared cameras, etc., and the parameters of the left and right cameras, such as focal length, resolution, depth of field, etc., can be the same or different. The first three-dimensional image refers to a three-dimensional image of the object 104 collected by the color three-dimensional sensor.

[0045] The depth camera 1035 is used to collect a second three-dimensional image of the object under test. The depth camera 1035 can be a depth camera based on time-of-flight (TOF), structured light or passive binocular vision technology. Generally, at least one of the resolution, accuracy and frame rate of the collected second three-dimensional image is lower than that of the first three-dimensional image. Generally speaking, the resolution, accuracy and frame rate of the second three-dimensional image are all lower than those of the first three-dimensional image. For ease of description, in the following description, the first three-dimensional image of the object is referred to as a high-precision fine three-dimensional model, and the second three-dimensional image of the object is referred to as a low-precision rough three-dimensional model. The second three-dimensional image refers to the three-dimensional image of the object under test 104 collected by the depth camera 1035.

[0046] The mechanical arm 102 and the rotating shaft 105 constitute a posture adjustment module, which is used to fix the imaging module 103 and adjust its posture. Among them, the mechanical arm 102 is connected to the imaging module 103 and the rotating shaft 105, and the rotating shaft 105 is installed on the base 101 for rotating around the base 101. The mechanical arm 102 is a multi-axis linkage mechanical arm for corresponding posture adjustment. Through the joint adjustment of the rotating shaft 105 and the mechanical arm 102, the imaging module 103 can be transformed in multiple directions to facilitate multi-directional measurement of the object to be measured 104. In some embodiments, the rotating shaft 105 includes a rotating motor, and the mechanical arm will rotate around the base driven by the rotating shaft under the drive of the rotating motor to measure the object to be measured.

[0047] The processor is connected to the robot arm 102, the imaging module 103, and the rotating shaft 105, and is used to perform control and corresponding data processing or three-dimensional scanning tasks, such as three-dimensional color image extraction, rough three-dimensional model establishment, fine three-dimensional model establishment, etc. It is understandable that the processor can be a single processor or multiple independent processors. For example, the imaging module can include multiple special processors for executing algorithms such as three-dimensional imaging. The system also includes a memory for storing algorithm programs executed by the processor, such as various algorithms and methods mentioned in the present invention (calibration methods, reconstruction methods, viewpoint generation algorithms, and scanning methods, etc.). The memory can be various computer-readable media, such as non-transitory storage media, including magnetic media and optical media, such as disks, tapes, CDROMs, RAMs, ROMs, etc.

[0048] It can be understood that the three-dimensional image mentioned above may refer to a depth image, or may refer to point cloud data, mesh data or three-dimensional model data obtained by further processing based on the depth image.

[0049] When the system 10 is used to perform a three-dimensional scan on the object 104, the overall scanning process is performed by the processor and is divided into the following steps:

[0050] Step 1: calibrate the depth camera 1035 and the color 3D sensor to obtain the internal and external parameters of the depth camera 1035 and the color 3D sensor. The specific process is described in detail below.

[0051] Step 2: Use the depth camera 103 to collect a low-precision rough three-dimensional model of the object 104, such as using the rotation axis 105 and the mechanical arm 102 to control the depth camera 1035 to circle the object 104 to quickly generate a low-precision rough three-dimensional model of the object. It is understandable that the object 104 needs to be placed on the base 101 in advance. In one embodiment, the object 104 is placed at the center of the base 101;

[0052] Step 3: Generate global scanning viewpoints based on the low-precision rough three-dimensional model calculation, specifically, automatically generate global scanning viewpoints according to the NBVs algorithm proposed in the present invention.

[0053] Step 4: Perform high-precision three-dimensional scanning on the object 104 using an active binocular vision camera based on the generated global scanning viewpoint and the shortest path planning to obtain a first high-precision fine three-dimensional model;

[0054] In some embodiments, it is also necessary to calculate a confidence map of the first high-precision fine three-dimensional model, determine areas with missing data and missing details, and perform supplementary scanning to obtain a second high-precision fine three-dimensional model with higher accuracy;

[0055] In some embodiments, during the acquisition process of the first and / or second high-precision fine three-dimensional models, a color camera is simultaneously used to acquire color images, and the color images are texture mapped to achieve coloring of the fine three-dimensional models to obtain three-dimensional color digital images, ultimately achieving high-fidelity three-dimensional color digitization of the complete object.

[0056] System calibration

[0057] Before using the system 10 to perform a three-dimensional scan on the object 104 to be measured, it is necessary to calibrate each component in the system to obtain the relative position relationship between the coordinate systems of each component. Only based on the relative position relationship can corresponding operations be performed, such as coloring, generating a global scanning viewpoint based on a rough three-dimensional model, etc.

[0058] Figure 2 Schematic diagram of the distribution and transformation relationship of the system coordinate system according to an embodiment of the present invention. The world coordinate system is established on the base coordinate system, and the color 3D sensor coordinate system is established on the left camera S l The depth camera coordinate system is built on the infrared camera S i On. It is necessary to determine the internal and external parameters of the color 3D sensor through system calibration, as well as the transformation matrix of the color 3D sensor / depth camera coordinate system-mechanical arm coordinate system-mechanical arm base coordinate system-base coordinate system. The difficulty of system calibration in the present invention lies in that there are sensors with different resolutions and different fields of view (for example, the color camera has 20 million pixels, the left and right cameras have 5 million pixels, and the lens FOV is H39.8°, V27.6°; the depth camera has 300,000 pixels, and the FOV is H 58.4°, V 45.5°), and sensors with different spectral response ranges (such as the spectral response range of the color camera and the black and white camera is in the visible light band; the response range of the infrared camera is in the infrared band). At the same time, the calibration accuracy of the color 3D sensor must be guaranteed. Therefore, designing and making high-precision stereo targets is the key to high-precision calibration.

[0059] Figure 3This is a schematic diagram of a low-cost stereo target based on non-coding marker points according to an embodiment of the present invention. The stereo target consists of a first sub-target A and a second sub-target B. The first sub-target A is composed of a plane, and the surface of the plane is non-coding marker points arranged regularly (for example, 11×9). The accurate spatial coordinates of these marker points can be determined by bundle adjustment technology. The marker points include reference points and positioning points, where the point locations include at least four. In order to improve the accuracy of marker point extraction of low-resolution depth cameras, the reference points and positioning points are both designed with large circles. The interior of the positioning point and the reference point includes a small black concentric marker point (for example, concentric circles), and the positioning point and the reference point are distinguished by the grayscale of the center of the marker point (for example, if the grayscale of the center is greater than 125, it is a reference point, and if it is less than 125, it is a positioning point, that is, the grayscale of the center of the reference point is different from that of the positioning point). Figure 3 As shown in (c), this design greatly increases the size of the reference point and improves the positioning accuracy of the positioning point; the second sub-target B is composed of multiple planes, and non-coding markers are randomly pasted on the surface for rotation axis calibration. The non-coding markers of the second sub-target B are designed with small circles. The small circles here are relative to the size of the markers in the first sub-target A, that is, smaller than the markers in the first sub-target A. The calibration process is to reconstruct the spatial coordinates of random markers under multiple perspectives by surrounding the stereo target with a color three-dimensional sensor, and determine the base coordinate system by optimizing the marker matching. Therefore, there is no need to predetermine the spatial coordinates of the random markers of the second sub-target B, which greatly reduces the difficulty and cost of target production.

[0060] The calibration process is divided into two steps: (1) The rotating shaft (rotating motor) remains stationary, and the robotic arm carries a color 3D sensor to collect multi-view images of the first sub-target A and calculate the internal and external parameters of the color 3D sensor, H lm and H im Since the left and right cameras, the color camera and the infrared camera work under light sources of different spectral bands, in each acquisition, the left and right cameras and the color camera first collect the target image under visible light illumination, and then illuminate the target with an infrared light source, and the infrared camera collects the target image; (2) The robot arm keeps its posture unchanged, and the motor rotates at different angles. The left and right cameras use the principle of binocular stereo vision to reconstruct the three-dimensional coordinates of the random marking points on the target B part at each viewing angle, and determine the rotation angle by matching the marking points, thereby constructing the base coordinate system and calculating H. ba .

[0061] In one embodiment, when calibrating the color 3D sensor, three cameras (left, right, and infrared cameras) simultaneously acquire target patterns under different viewing angles, and construct the objective function of the single camera calibration model:

[0062]

[0063] in represents the spatial homogeneous coordinates of the jth marker among the M markers in the target coordinate system, x ij (i=1,…N) represents the image coordinates of the jth landmark point in the image captured by the camera at the i-th viewing angle, K is the camera's internal parameter matrix, including focal length, principal point position and tilt factor, ε is the lens distortion, and this paper only considers the typical fifth-order lens distortion {Fryer, 1986#114}, Represents the transformation matrix from the target coordinate system to the camera coordinate system at the i-th viewing angle.

[0064] Generally, assuming that the left camera coordinate system is the three-dimensional sensor coordinate system, the structural parameters of the three cameras are:

[0065]

[0066] in, and The left camera S l To the right camera S r The rotation matrix and translation vector, and For the left camera S l To color camera S c In order to obtain higher-precision structural parameters, we add the transformation matrix to the nonlinear objective function of the three cameras and use the Gauss-Newton or Levenberg-Marquardt method to minimize the objective function to estimate the camera parameters:

[0067]

[0068] where τ = {ε l ,ε r ,ε c ,K l ,K r ,K c ,H lr ,H lc}, In this way, the internal and external parameters of the color 3D sensor can be obtained. The parameters of the infrared camera are solved in a similar way.

[0069] After the color 3D sensor calibration is completed, the transformation matrix of the left camera at each acquisition angle can be obtained. Directly given by the robot control system, according to the mathematical model of hand-eye calibration, the following relationship is established:

[0070]

[0071] Among them, i,k=1,2,...,N, and i≠k, N is the number of scans, and N motion postures can be established According to Tsai's method

[30] , the linear least squares method can be used to solve H sg and H cb .

[0072] In one embodiment, in order to further improve the accuracy, we use it as the initial value to establish a nonlinear objective function:

[0073]

[0074] in It can be obtained from the robot arm in real time. Using the Levenberg-Marquardt method to minimize the objective function can obtain a higher precision H lm With H bt .H im The solution is similar and will not be discussed further.

[0075] In one embodiment, during the calibration of the rotation axis, the robot arm maintains the same posture, and the transformation matrix from the robot arm to the base is recorded as H′ gb The 3D sensor moves in a circle around the stereo target and reconstructs the random landmarks of part B of the target at different rotation angles. T is the number of rotations, j is the number of landmarks, and the landmarks reconstructed under all fields of view Perform global matching optimization to obtain the transformation relationship of the target landmark at each rotation angle [R (m) |T (m) ], then the rotation axis direction vector can be calculated under the constraint of the distance between each two closed circular trajectory planes, and the center of each circular trajectory can be obtained by the global least squares optimization method, thereby determining the transformation relationship H from the three-dimensional sensor coordinate system to the base coordinate system rl According to the transformation relationship (H rl ) -1 =H br H′ mb H lm The base coordinate system to the base coordinate system H can be obtained br .

[0076] Global Scan Viewpoint Generation

[0077] According to the stereoscopic vision imaging model, the measurement space of the three-dimensional sensor is limited by the binocular camera angle (FOV), the focal length and depth of field (DOF) of the camera lens and the digital projection lens, and the quality of the point cloud of three-dimensional reconstruction is also affected by many constraints. The present invention is based on certain constraints. By analyzing the rough three-dimensional model (Roughmodel), a series of scanning viewpoints are automatically generated to achieve three-dimensional digital color imaging of the complete object with the least number of viewpoints. Next, the constraints and viewpoint generation methods are introduced respectively.

[0078] Figure 4 Schematic diagram of the constraint relationship of a binocular vision 3D sensor according to an embodiment of the present invention. Figure 4 (a) is a schematic diagram of the basic structure and measurement space of the binocular sensor. Figure 4 (b) is the measurement space constraint of the 3D sensor, Figure 4 (c) is the point cloud visibility constraint. For the sake of simplicity, the present invention does not describe the calculation of the specific view volume, and the measurement space is simplified to Figure 4 As shown in (b), assuming that the working distance range of the 3D sensor is [d n ,d f ], maximum field of view The viewpoint position is v i (x,y,z),v i (α, β, γ) represents the unit vector of the 3D sensor optical axis direction, v ik =d(v i ,s k ) represents the viewpoint position v i Point to the measurement target point position s k The viewpoint planning process is affected by the object surface space, viewpoint space and imaging work space, and its constraints mainly include but are not limited to at least one of the following aspects:

[0079] 1) Visibility constraint: indicates the angle range within which the target point is allowed to be captured by the sensor. Suppose the target point p is k The normal vector is n k , then the visibility constraint

[0080]

[0081] in Indicates the maximum viewing angle range of the measurement target point, such as Figure 4 (c) as shown.

[0082] 2) Measurement space constraints: including field of view (FOV) constraints and depth of field (DOF) constraints, representing the measurable range of the 3D sensor. The constraints are:

[0083]

[0084] where φ max Indicates the maximum field of view of the 3D sensor, such as Figure 4 (b) as shown.

[0085] 3) Overlap constraint: For the subsequent ICP matching and grid fusion (registration and integration) of multi-view depth data, there needs to be a certain degree of overlap between adjacent scan fields of view. The field of view overlap is defined as W and W cover Represent the total area of ​​the field of view and the area of ​​the overlapping part respectively, and the constraints are

[0086] ξ≥ξ min (8)

[0087] where ξ min is the minimum field of view overlap.

[0088] 4) Occlusion constraint: When the viewpoint v i To the measurement target point s k The line segment d(v i ,s k ) intersects with the object entity, indicating that the viewpoint v i At the target point s k The viewpoint direction v ik is obscured.

[0089] For an object of unknown shape, first use the depth camera to perform an initial scan around the object to generate a rough 3D model. The purpose of this step is to use the model to generate a global scanning perspective, so the rough 3D model does not require too high accuracy and resolution, nor does it require the scanning data to be particularly complete. In addition, since depth cameras generally have the characteristics of a wide scanning perspective, a large depth range of the measurement space, and good real-time performance, for most objects of different sizes and different surface materials, a set of scanning postures can be simply preset to achieve an initial scan of the object's morphology.

[0090] In one embodiment, during the initial scanning process, a matching and fusion algorithm, such as the kinectFusion algorithm, is used to match and integrate the data in real time. After the initial scanning is completed, the original point cloud is preprocessed by noise filtering, smoothing, edge removal, and normalization estimation, and then an initial closed triangular mesh model is generated. The model is sampled by Poisson-disc sampling to obtain the so-called ISO points, such as Figure 4 As shown in (b), let the model sampling point be

[0091] According to the initial model size and the maximum working distance d of the scanner f , construct the minimum bounding box S that includes the model and the scan space, and divide the space into 3D voxel grids according to a certain distance interval ΔD (for example, divided into 100×100×100 voxels). For any spatial point (p x ,p y ,p z ), according to formula (9), we can quickly find out which voxel grid the point belongs to.

[0092]

[0093] Among them (p x-min ,p y-min ,p z-min ) is the minimum coordinate value of the bounding box S, v i =(n x ,n y ,n z ) is the voxel number value, the center point of the voxel It will be used as a spatial 3D point to participate in the calculation of the next best views (NBVs). The NBVs algorithm in this paper is mainly divided into three steps:

[0094] Step 1: For the initial model sampling point s k , along its normal n k , the distance is d0=(d n +d f ) / 2, we can find the voxel v according to formula (9): i . With v i To search for seeds, a greedy algorithm is used to perform an expansion search on the neighborhood voxels. According to the visibility constraints described above, the voxel numbers that satisfy equation (10) are recorded at the sampling point s k The associated collection Among them, such as Figure 5 (a) shown.

[0095]

[0096] where v ik =d(v i ,s k ) represents point v i To point k The vector, w ik (v i ,s k )=1 means s k v i It can be seen that when w ik (v i ,s k )=0 means s k to v i There is occlusion between them. At the same time, (s k ,v ik ) is recorded in all v that satisfy formula (10) i The associated collection In For all ISO points k}Execute step 1 to obtain all valid voxels that record ISO points {v i}, and the voxels that are not recorded are considered invalid and no longer participate in the calculation.

[0097] Step 2: For the valid voxel v i , according to its collection Element s k The labeling function g(s k ), solve for the labeling score of the voxel

[0098]

[0099] g(s k )Marks k The usage of s k When it has not been confirmed to belong to a certain scanning viewpoint, it is marked as 1, and when it has been confirmed to belong to a certain scanning viewpoint, it is marked as 0, that is,

[0100]

[0101] Step 3: Select the voxel with the largest label score value for viewpoint calculation. The ISO points recorded by a voxel may not be covered by the same scanning range, such as Figure 5 (b), so we use the histogram statistics method to All the s k The vector d(v i ,s k ) for statistics and selection. According to the Cartesian coordinate system (x, y, z) and the spherical coordinate system The transformation relationship converts the vector d(v i ,s k ) is converted to the spherical coordinate system, the X-axis and Y-axis of the histogram are θ and The Z axis is the statistical number of iso points, such as Figure 6 As shown. According to the scanning field of view constraint of the 3D sensor φ max and overlap constraint ξ min , determines the size of the filter window in the XY plane φ filter =φ max (1-ξ min ), the filter traverses all elements (x, y) of the histogram XY plane and calculates the sum of the number of iso points in the filter. When the statistic in the filter window is the largest, the iso points contained in the filter are {s′ k} k∈N , N is the label weight g(s′ k )=1 k The number of s′ k The vector d(v i ,s k ) Find the mean value as the scanning viewpoint direction

[0102]

[0103] At this point, the spatial position of the viewpoint and the viewpoint direction vector can be obtained

[0104] Step 4: k The labeling function g(s′) in k ) is set to 0.

[0105] Repeat Step 2-Step 4 until the labeling scores of all voxels are lower than the threshold. The flowchart of the NBVs algorithm is shown in Figure 7. From the above algorithm flow, it can be seen that the valid voxels contain all iso points that meet the constraints. The higher the labeling score of the voxel, the more the viewpoint calculated by the voxel can cover the surface range of the object, that is, the more important the viewpoint is. The viewpoint in this paper is calculated by selecting the voxel with the largest labeling score, and the final viewpoint list is also sorted from most to least according to the number of iso points covered by the viewpoint.

[0106] Automatic 3D scanning and complementary scanning

[0107] The spatial position and direction of a series of viewpoints are obtained through the NBVs algorithm described above. How to scan all viewpoints in the shortest path belongs to the path planning problem. The main algorithms for solving path planning problems include ant colony algorithm, neural network algorithm, particle swarm algorithm, genetic algorithm, etc., each with its own advantages and disadvantages. For example, in one embodiment, the shortest path can be obtained by solving the viewpoint set using the ant colony algorithm. Next, a color three-dimensional sensor is used to perform three-dimensional scanning along the shortest path. The high-precision depth data (left camera coordinate system) collected at each viewing angle is converted to the world coordinate system through the coordinate system transformation relationship, and finally the real-time matching of multi-view depth data is realized to calculate a high-precision and fine three-dimensional model of the object.

[0108] In one embodiment, the 3D sensor performs 3D scanning along the shortest path. In each viewpoint transformation process, the joint control of the rotating motor and the robot arm is involved, which is essentially a transformation problem of the coordinate system of each sensor. Suppose two adjacent scanning viewpoints are V i and V j , the transformation matrix of these two viewpoints Indicates the infrared depth sensor coordinate system in the world coordinate system from the viewpoint V i To V j In order to make the three-dimensional sensor from the viewpoint V i Adjust to V j , respectively find the viewpoint V i and V j In the rotation axis coordinate system (i.e. the world coordinate system), a Y a Projection coordinates of the plane and Thus, the rotation angle θ of the rotating motor is obtained ij and the transformation matrix

[0109]

[0110] Viewpoint after motor rotation Then the viewpoint V i ′ to V j The transformation matrix Since the transformation matrix H im and the robot arm at viewpoint V i The transformation matrix under Known, the following transformation relationship can be established:

[0111]

[0112] Therefore, we can get the position of the robot arm at the viewpoint V j The transformation matrix is ​​as follows. By combining the rotation angle θ of the rotating motor ij and the rotation matrix of the robot The posture adjustment of the 3D sensor at different viewpoints can be realized. The high-precision depth data at each viewpoint (in the left camera coordinate system) is converted to the world coordinate system through the transformation matrix of the infrared depth sensor coordinate system and the viewpoint transformation matrix, realizing real-time matching of multi-view depth data.

[0113] From formula (10), we can see that the viewpoint planning algorithm in this paper has taken into account the self-occlusion of the object. However, in the actual scanning process, due to the influence of factors such as the surface material of the object, some data missing or sparse point cloud data and other low quality problems are inevitable. More importantly, since the rough 3D model used for viewpoint planning loses the detailed information of the object, the generated viewpoint does not take into account the fine scanning of the geometric details.

[0114] To this end, in one embodiment, a method of constructing a model confidence map is used to reflect the missing parts of the original data and the missing areas of details, and a viewpoint planning algorithm is used to generate a viewpoint for supplementary scanning. The original point cloud data obtained in the previous high-precision scanning stage is sampled by Poisson-disc sampling to generate IS0 sampling points. Generate iso point s according to formula (16) k Confidence map of :

[0115] f(s k )=f g (s k ,n k )f s (s k ,n k ) (16)

[0116] where f g (s k ,n k )=Γ(s k )·n k Defined as the completeness confidence score, Γ(s k ) is point s k The scalar field gradient at n k is the normal vector. g (s k ,n k ) is already obtained during the Poisson-disk sampling process, so no additional calculation is required; f s (s k ,n k ) is the smoothness confidence score K (smoothness confidence score), satisfying

[0117]

[0118] where ||g|| is the l2-norm, For point s k The K neighborhood range Ω k The original point cloud inside, The spatial weight function θ(||s k -q j ||) in Ω k The range decays sharply with the increase of radius; the orthogonal weight function φ(n k ,q j -s k ) reflects the K neighborhood range Ω k The original point q j The distance to the tangent plane at point iso. When the smoothing confidence score is high, the surface point s k The local area is relatively smooth and the scanning quality is relatively high; when the smooth confidence score value is low, it indicates that point s k If the local raw scan data is sparse, or the raw scan data contains more high-frequency components, such as point cloud noise or rich geometric details, more supplementary scans are required.

[0119] The confidence score effectively reflects the quality and fidelity of the scanned model point cloud data. We use the model confidence score to guide the viewpoint planning of the supplementary scanning phase. Set the confidence score threshold ε and find the range of iso points of the missing part and the part rich in geometric details S′={s′ k |f(s′ k )≤ε}, and the viewpoint of S′ is calculated by the above algorithm. The difference from the NBVs algorithm mentioned above is that g(s′ k ) According to s′ k Assign confidence scores

[0120]

[0121] Therefore, the voxel score no longer reflects the number of iso points it contains, but the sum of the confidence scores of the iso points. Calculating the viewpoint for the voxel with the highest confidence score will make the viewpoint focus more on scanning the missing parts and parts rich in geometric details.

[0122] The above contents are further detailed descriptions of the present invention in combination with specific / preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, they can also make several substitutions or modifications to these described embodiments without departing from the concept of the present invention, and these substitutions or modifications should be regarded as belonging to the protection scope of the present invention.

Claims

1. A viewpoint planning method based on a three-dimensional model, characterized in that: The following steps are involved: S1, using the initial sampling point s of the three-dimensional model k Find the voxel v i , and the voxel v i Search for a valid voxel set {v i }; S2, using the labeling function g(s k ) solves the voxel set {v i The labeling score of each voxel in The marking function g(s k ) is used to mark the sampling point s k The usage of s k When it has not been confirmed to belong to a certain scanning viewpoint, it is marked as 1, and when it has been confirmed to belong to a certain scanning viewpoint, it is marked as 0; S3, selecting the voxel with the largest labeling score to perform viewpoint calculation; S4, setting the labeling function to 0, and repeating steps S2 to S4 until the labeling scores of all the voxels are lower than the threshold; The viewpoint calculation includes calculating the spatial position of the viewpoint; The viewpoint calculation further includes calculating the direction vector of the viewpoint; the calculation of the direction vector includes the following steps: Using histogram statistics for all sampling points s k The vector d(v i ,s k ) to conduct statistics and selection; The vector d(v i ,s k ) Convert from Cartesian coordinates (x, y, z) to spherical coordinates Down; Determine the size of the filter window in the XY plane Φ filter =Φ max (1-ξ min ), and use the filter to traverse all elements (x, y) of the histogram XY plane and calculate the sum of the number of sampling points in the filter. When the statistic in the filter window is the largest, the sampling points contained in the filter are {s′ k } k∈N , N is the label weight g(s′ k )=1 k The number of s′ k The vector d(v i ,s k ) find the mean value as the scanning viewpoint direction Among them, Φ max Indicates the maximum field of view of the 3D sensor; ξ min is the minimum field of view overlap.

2. The viewpoint planning method based on a three-dimensional model according to claim 1, characterized in that: The step S1 further comprises: For the initial sampling point s k , along its normal n k , the distance is d0=(d n +d f ) / 2, the voxel v can be found according to the following formula i : Wherein, ΔD refers to the 3D voxel grid division of the three-dimensional model according to the distance interval of ΔD; (p x ,p y ,p z ) refers to the coordinates of a point in three-dimensional space, (p x-min ,p y-min ,p z-min ) is the minimum coordinate value of the space bounding box S, v i =(n x ,n y ,n z ) is the voxel number value.

3. The viewpoint planning method based on a three-dimensional model according to claim 1, characterized in that: The search in step S1 is performed by dilating the neighborhood voxels using a greedy algorithm and is calculated based on visibility constraints.

4. The viewpoint planning method based on a three-dimensional model according to claim 3, characterized in that: The visibility constraint indicates the angle range within which the measurement target point is allowed to be collected. Suppose the measurement target point p k The normal vector is n k , then the visibility constraint is: in, Indicates the maximum viewing angle range of the measurement target point, v ik =d(v i ,s k ) represents the viewpoint position v i Point to the measurement target point position s k Vector.

5. The viewpoint planning method based on a three-dimensional model according to claim 1, characterized in that: The filter window is determined according to the measurement space constraint and the overlap constraint.

6. The viewpoint planning method based on a three-dimensional model according to claim 5, characterized in that: The measurement space constraints include field of view (FOV) constraints and depth of field (DOF) constraints, and the constraint conditions are: The overlap constraint means that there must be a certain overlap between adjacent scanning fields of view. The field overlap is defined as W and W cover Represent the total area of ​​the field of view and the area of ​​the overlapping part, respectively, and the constraint condition is ξ≥ξ min .

Images