Multi-view vision algorithm based on apriltags

Abstract

The application discloses a multi-vision algorithm based on an Apriltags label and relates to the technical field of three-dimensional reconstruction. The algorithm shoots a photo containing the same Apriltags label through a mobile phone camera, constructs a virtual multi-vision stereo vision system based on a label external parameter, and optimizes the pose precision through multi-label external parameter averaging; real point clouds are obtained through rough extraction (ideal label area) and precise extraction (PCA clipping); point cloud splicing is realized through Apriltags label coordinate conversion, and an ICP algorithm is not needed. The application does not need a real binocular camera, reduces hardware cost, and is suitable for mobile phones and other portable devices; automatic focusing is disabled, the photo size is fixed to ensure parameter stability, three-dimensional reconstruction precision and efficiency are effectively improved, and the application is suitable for scenes such as facility and equipment installation and building site installation inspection.

Description

Technical Field

[0001] This invention relates to the field of 3D reconstruction technology, and in particular to a multi-view vision algorithm based on Apriltags. Background Technology

[0002] The installation of large-scale facilities and equipment (such as modular water tanks) and construction industry installations (such as wall panel and brick installation, prefabricated buildings, etc.) are very common in the engineering industry, resulting in a large market for testing the accuracy of installation or stacking. Three-dimensional reconstruction technology, as a core means of rapidly quantifying installation errors, has become a research hotspot in this field.

[0003] Existing 3D reconstruction methods mainly include monocular camera reconstruction, binocular / multi-view camera reconstruction, RGB-D camera reconstruction, and point cloud stitching techniques, but all of them have significant drawbacks: 1. Highly dependent on hardware, unsuitable for on-site portability requirements. Existing technologies generally rely on dedicated hardware, making them difficult to adapt to the complex environments of facilities, equipment, and on-site construction installations. For example, Chinese patent CN116486020A discloses a 3D reconstruction method based on an RGB-D camera and a rotating platform, which uses Apriltag tags on a turntable to obtain transformation matrices and achieve point cloud fusion. However, this method requires a dedicated RGB-D camera and a rotating platform, making it unsuitable for on-site deployment—large components cannot be placed on a rotating platform, rendering it unusable for 3D reconstruction in actual engineering projects.

[0004] 2. High point cloud acquisition costs reduce project application benefits. Currently, wearable laser scanners, handheld laser scanners, and 3D point cloud cameras are convenient in terms of portability, but the equipment cost and usage cost are relatively high.

[0005] 3. Point cloud stitching has low efficiency and is prone to getting trapped in local optima. Existing point cloud stitching technologies largely rely on the ICP (Iterative Closest Point) algorithm, but this algorithm has inherent flaws. Chinese patent CN114283046B optimizes the parallel computing efficiency of ICP, but its core still depends on point cloud feature matching. In facility and equipment installation at construction sites, due to the high repetition of components and complex backgrounds, the ICP algorithm is sensitive to initial pose and easily gets trapped in local optima, resulting in low point cloud registration accuracy and poor efficiency. For example, when stitching point clouds of large components captured from multiple perspectives, the feature discrimination of similar components is low, making it difficult for the ICP algorithm to accurately match them and failing to meet assembly error requirements.

[0006] 4. Poor scene adaptability; not optimized for facility equipment and on-site construction installation. Existing technologies do not fully consider characteristics such as large volume, multiple components, and limitations of on-site photography: Large components cannot be captured in a single photograph, and existing methods lack efficient multi-view point cloud stitching solutions; During on-site shooting, the autofocus of mobile devices such as smartphones can cause changes in intrinsic parameters, and existing technologies have not proposed a stable parameter control strategy. The failure to utilize Apriltag tags on the component surface for pose constraints resulted in insufficient point cloud reconstruction accuracy.

[0007] In summary, existing technologies cannot meet the demands of facilities, equipment, and on-site construction for low-cost, highly portable, and high-precision 3D reconstruction. There is an urgent need for a 3D reconstruction algorithm based on common mobile devices (such as smartphones) to address these shortcomings. Summary of the Invention

[0008] To address the aforementioned technical problems, the present invention aims to provide a multi-view vision algorithm based on ordinary mobile devices (such as mobile phones).

[0009] To achieve the above objectives, this invention provides a technical solution for a multi-view vision algorithm based on Apriltags, comprising the following steps: Step S1. Use a single moving camera device to capture at least two photos of the structure to be detected that contain the same Apriltags, where the same Apriltags refer to the same physical tag; Step S2. Calculate the relative position of the camera based on the extrinsic parameters of Apriltags in the photo, and construct a virtual binocular stereo vision system; if there are multiple identical Apriltags, optimize the camera pose accuracy by averaging the extrinsic parameters of multiple tags. Step S3. Use the virtual binocular stereo vision system to generate a three-dimensional point cloud, and perform coarse extraction and fine extraction processing on the point cloud in sequence to obtain the real point cloud of the structure to be detected; Step S4. Based on the coordinate transformation relationship of Apriltags, align the point clouds generated from different photos to complete the point cloud stitching.

[0010] Furthermore, step S1 also includes camera intrinsic parameter calibration, which includes: a. Perform camera intrinsic parameter calibration on a single moving camera device and obtain the camera intrinsic parameter matrix K; b. Take at least two photographs of the target scene containing the same Apriltags, where the same Apriltags refer to the same physical tag.

[0011] Furthermore, the camera intrinsic parameter calibration in step S1 includes: a1. Device settings: Turn off the autofocus and auto exposure functions of the single-moving camera device, and fix the focal length; a2. Take calibration photos: Using the standard calibration board as a reference, take photos from multiple angles around the calibration board to collect 15-20 valid photos; a3. Intrinsic parameter calculation: Using the Zhang Zhengyou calibration method, the camera intrinsic parameter matrix K and distortion coefficients are solved by detecting feature points in the calibration photos. The distortion coefficients include radial distortion coefficients k1, k2, k3 and tangential distortion coefficients p1, p2. a4. Verification: Calculate the reprojection error of feature points and ensure that the average reprojection error is less than 1 pixel. If the error exceeds the standard, recalibrate.

[0012] Furthermore, the shooting described in step S1 satisfies the following conditions: the autofocus function is disabled, and the image resolution remains unchanged.

[0013] Furthermore, the construction of the virtual binocular stereo vision system in step S2 includes: a. Epipolar correction: The rotation matrix R and translation vector T are obtained by decomposing the fundamental matrix F. The image of the photograph is corrected to a parallel camera model so that the row coordinates of corresponding pixels are the same and the average row alignment error is <1 pixel. b. Disparity Calculation: The disparity of the corrected image pixels is calculated using the semi-global matching (SGM) method. The SGM parameters include: Parallax range: Calculated using formula (11): Formula (11), In formula (11): d is the disparity value; f is the camera focal length; t is the distance the camera moves; Z represents the average depth of the structure to be tested; c. Point cloud generation: Generate a 3D point cloud based on the disparity map and camera parameters.

[0014] Furthermore, the specific method for averaging the multi-label extrinsic parameters in step S2 is as follows: if n identical Apriltags labels are detected, the averaged camera extrinsic parameters are calculated using formula (8): Formula (8), In formula (8), R once : The averaged camera extrinsic parameter rotation matrix; T once : The averaged translation vector of the camera's extrinsic parameters; n: The total number of identical Apriltags detected; R S The camera relative rotation matrix corresponding to a single label; T SThe camera's relative translation vector corresponding to a single label; R S ,T S : The camera extrinsic rotation matrix and translation vector corresponding to the i-th label; : Perform a summation operation on the extrinsic parameters of n labels.

[0015] Furthermore, the coarse extraction in step S3 includes: combining the ideal regions corresponding to each Apriltags label to determine the overall ideal space covering the structure to be detected, cropping the point cloud to remove background noise, and obtaining the cropped region.

[0016] Furthermore, the ideal region refers to the theoretical three-dimensional pixel region calculated based on Apriltags tags, and the cropping region refers to the spatial range expanded by 10% based on the ideal region.

[0017] Further, the fine extraction in step S3 includes: a. Using the principal component analysis (PCA) algorithm, calculate the covariance matrix of the point cloud and perform eigenvalue decomposition to determine the principal axis direction; b. The PCA principal axis increment is 5% of the design model size. The best-fit rectangle is determined based on the principal axis direction and increment. The point cloud is cropped to remove background noise and obtain the region to be detected.

[0018] Furthermore, the point cloud stitching described in step S4 does not require the use of the Iterative Nearest Point (ICP) algorithm; instead, alignment is achieved directly by calculating the relative positions of the point clouds using the extrinsic parameters of the Apriltags labels.

[0019] Compared with existing technologies, the beneficial effects achieved by this invention are as follows: 1. Low cost and portability: No real binocular camera is required. A single moving camera is used to simulate binocular and multi-view systems, reducing hardware costs and making it compatible with mobile phones. 2. High-precision pose: The accuracy is optimized by averaging the extrinsic parameters of multiple labels, avoiding single-label errors; 3. Robust point cloud stitching: Utilizing Apriltags coordinate transformation instead of the ICP algorithm, it is unaffected by noise and is more efficient; 4. Realistic Model Extraction: The coarse and fine extraction process effectively removes noise and obtains realistic point clouds that reflect assembly quality; 5. Stable parameters: Disable autofocus and fix the image size to ensure that the camera's intrinsic parameters remain unchanged, thus improving the reliability of the results. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the process steps of the multi-view vision algorithm based on Apriltags in Embodiment 1 of the present invention.

[0021] Figure 2 This is a schematic diagram of a multi-view shooting scene for the multi-view vision algorithm of Embodiment 1 of the present invention.

[0022] Figure 3 This is a schematic diagram of the coordinate relationship of the virtual binocular system in the multi-view vision algorithm of Embodiment 1 of the present invention.

[0023] Figure 4 This is a schematic diagram of the camera relative position calculation and point cloud stitching of the multi-view vision algorithm in Embodiment 1 of the present invention.

[0024] Figure 5 This is a schematic diagram showing the construction result of the virtual binocular stereo vision system based on the multi-view vision algorithm in Embodiment 1 of the present invention.

[0025] Figure 6 This is a schematic diagram of the coarse extraction of point cloud of the structure to be detected by the multi-view vision algorithm in Embodiment 1 of the present invention.

[0026] Figure 7 This is a schematic diagram of the point cloud extraction of the structure to be detected by the multi-view vision algorithm in Embodiment 1 of the present invention.

[0027] Figure 8a In the camera intrinsic parameter calibration of the multi-view vision algorithm in Embodiment 1 of the present invention, a standard chessboard is selected as the calibration board.

[0028] Figure 8b This is a bar chart showing the error analysis in the verification process of camera intrinsic parameter calibration for the multi-view vision algorithm in Embodiment 1 of the present invention.

[0029] Figure 9 This is the error distribution of the inverse surface and the reference surface in the effect verification of the multi-view vision algorithm in Embodiment 1 of the present invention.

[0030] Figure 10 In the effect verification of the multi-view vision algorithm in Embodiment 1 of the present invention, the inverse surface containing deformation is obtained by NURBS surface fitting.

[0031] Figure 2-7 middle: 1. Single moving camera device; 2. Apriltags; 3. Structure to be detected; 4. Camera movement direction; 5. Initial phone position; 6. Phone position after movement; 7. Phone position after n movements; 8. Initial shot; 9. Shot after movement; 10. Pixel; 11. Background noise; 12. Ideal region; 13. Phone coordinate system; 14. Noise-reduced point cloud; 15. Cropped region; 16. Region to be detected.

[0032] Figure 8a middle: (0,0) is the origin of the coordinate system; X is the horizontal direction; Y is the vertical direction; and the marker point at the intersection of the chessboard is used as a reference for the calibration algorithm to detect the corner points of the chessboard.

[0033] Figure 8b middle: The horizontal axis (X-axis) represents different image numbers, from 0 to 15, for a total of 16 sets of data; Vertical axis (Y-axis): Represents the error value in pixels; This figure shows the average reprojection error or corner detection error of each image after calibrating multiple different images during the camera calibration process; the lower the error value, the more accurate the calibration result. Detailed Implementation

[0034] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, the specific embodiments of this application will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, it should be noted that, for ease of description, only the parts relevant to this application are shown in the accompanying drawings, not the entire structure. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of this application.

[0035] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0036] This invention provides a multi-view vision algorithm based on Apriltags, implemented in Python 3.6. The algorithm includes the following steps: Step 1: Single mobile device shooting and camera calibration

[0037] like Figure 1 As shown, multi-view photos are taken using a single moving camera device 1, and camera intrinsic parameter calibration is completed.

[0038] Generally speaking, photos taken with a single camera (monocular) usually cannot provide distance information. For example... Figure 2As shown, photos containing the same undetected modules need to be taken at least twice, with the camera movement distance ≤ 10% of the distance between the camera and the scene, ensuring that the Apriltags label 2 is fully visible; keep the photo size and focal length unchanged to avoid changes in intrinsic parameters. Note that the camera movement distance should not be too large at one time, otherwise it may cause calculation failure. The Apriltags label 2 of each module surface is obtained sequentially, and the position data of the label in different photos is calculated.

[0039] To address the problem that monocular images cannot solve distance measurement, the main idea of ​​this algorithm is to simulate a binocular stereo vision system based on Apriltags and a mobile phone camera, and then extract the real assembly model from the generated point cloud.

[0040] like Figure 3 The diagram shows a simplified binocular stereo vision system containing a single label and two photographs. This method uses three reference frames: camera coordinates (x1, y1, and z1), moving camera (phone lens) coordinates (x2, y2, and z2), and label coordinates (x3, y3, and z3). The origins of these three coordinate systems are the center of the lens and the center of the Apriltag (the optical axis is the z-axis), respectively. Similarly, a calibration process is needed to obtain the camera's intrinsic parameter matrix. K Instead of using the phone's lens parameters, this uses the camera intrinsic matrix. K The coordinate system (u, v) of the image is the same as in the ideal case. Using a calibration target and 15-20 photos, the mobile phone camera can be calibrated to obtain the camera intrinsic parameter matrix K. The calibration process for the camera intrinsic parameter matrix K is as follows: Considering the characteristics of consumer-grade mobile phones and based on accuracy test examples, the calibration process is as follows: (1) Prerequisites and necessity of calibration Source of error: Consumer-grade mobile phones (such as the Samsung Galaxy Note8 used in the test) have time drift errors in their CMOS camera module (CCM). The autofocus / exposure mode changes the focal length, which compromises the stability of the internal parameters.

[0041] Core objective: To obtain an accurate intrinsic parameter matrix K (including focal length, principal point coordinates, and distortion coefficients) through calibration, thereby eliminating the impact of inherent equipment errors on subsequent extrinsic parameter calculations and point cloud generation.

[0042] (2) Calibration preparation Calibration target: Use a standard calibration board (such as a checkerboard or Apriltags array calibration board) as a reference to ensure that the feature points are clear and easy to detect.

[0043] Device settings: Turn off the phone camera's autofocus and auto exposure functions, and fix the focal length (to avoid changes in focal length affecting internal parameters). Environmental requirements: An indoor environment with uniform lighting and no strong reflections, and the calibration board should be placed stably and without obstructions.

[0044] (3) Calibrating the shooting process Strictly follow the simplified operation requirements in the test case: Shooting method: The entire process was completed by holding the phone by hand, without the need for any auxiliary equipment; Shooting posture: Take multiple angle shots around the calibration board, covering different postures such as tilting, rotating, and near and far, and collect a total of 15-20 valid photos; Quality requirements: The calibration board feature points (such as corner points) must be clearly visible in each photo, without blurring or jitter.

[0045] (4) Internal reference calculation and verification Feature point detection: Automatically detect feature points of the calibration board in 15-20 photos; Algorithm solution: Zhang Zhengyou calibration method is adopted to solve the intrinsic parameter matrix K and distortion coefficients through the correspondence of feature points from multiple perspectives; The distortion coefficients include radial distortion and tangential distortion, and are of the following types: ①Radial distortion: Radial distortion is caused by the lens shape, and the formula is:

[0046] In formula (1)-(2): These are the corrected image pixel coordinates; r 2 =x 2 +y 2 It is the square of the distance from the pixel to the center of the image; k1, k2, and k3 are radial distortion coefficients used to describe the radial distortion of the image caused by the shape of the lens.

[0047] ② Tangential distortion: Tangential distortion is caused by the lens not being parallel to the imaging plane, and the formula is:

[0048] In formula (3)-(4): These are the corrected image pixel coordinates; x and y are the pixel coordinates of the original image; r 2 =x 2 +y 2 It is the square of the distance from the pixel to the center of the image; p1 and p2 are tangential distortion coefficients, used to describe the tangential distortion of the image caused by the lens not being parallel to the imaging plane.

[0049] Result verification: Calculate the reprojection error of feature points and ensure that the average error is <1 pixel (the qualified standard verified in the test example). If the error exceeds the standard, recalibrate.

[0050] (5) Output results After calibration, output the camera intrinsic parameter matrix K.

[0051] Step 2: Construct a virtual binocular vision system

[0052] After detecting Apriltags tag 2 in photos 1 and 2, the extrinsic parameters R and T of each photo can be obtained, such as... Figure 4 As shown. Since the same label was detected, the relative position of the camera was calculated using the extrinsic parameters (R1, T1) of photo 1 and (R2, T2) of photo 2. It should be noted that the rotation matrix (R) and translation matrix (T) are derived from the label coordinates relative to the camera coordinates.

[0053] Therefore, through multiple matrix operations and formula derivation, the relative position of the camera (R) is finally obtained. s , T s The calculation formula for ) is as follows: (where the translation matrix is...) This indicates the translation direction of camera 2 relative to camera 1.

[0054]

[0055] In formulas (5)-(7), R1: The rotation matrix of the Apriltags labels in photo 1 relative to the camera 1 coordinate system; T1: The translation vector of the Apriltags label in photo 1 relative to the coordinate system of camera 1; R2: The rotation matrix of the Apriltags labels in photo 2 relative to the camera 2 coordinate system; T2: Translation vector of Apriltags in photo 2 relative to the camera 2 coordinate system; R S : Rotation matrix of camera 2 relative to camera 1; T S : The translation vector of camera 2 relative to camera 1.

[0056] In each image, many structures to be detected are captured, and more than one of them is identical. Since these identical modules are labeled on their surfaces, there will be multiple labels (n) in a single processing. In this case, averaging the results is used to improve detection accuracy. Finally, the camera extrinsic parameters (R) are calculated. once , T once The final calculation is as follows: Formula 8

[0057] In formula (8), R once : The averaged camera extrinsic parameter rotation matrix; T once : The averaged translation vector of the camera's extrinsic parameters; n: The total number of identical Apriltags detected; R S : The camera relative rotation matrix corresponding to a single label (output of formula 5-7); T S : The camera relative translation vector corresponding to a single label (output of Equation 5-7); R S ,T S : The camera extrinsic rotation matrix and translation vector corresponding to the i-th label; : Perform a summation operation on the extrinsic parameters of n labels.

[0058] Through camera calibration, the camera extrinsic parameters (R) are... once , T once By combining this with the camera's intrinsic parameters, we can obtain, for example... Figure 5 The image shows a virtual binocular stereo vision system. It's important to note that the phone should not use autofocus mode when taking the picture; otherwise, the camera's intrinsic parameters may change when the user moves. Other options, such as photo size, should also remain constant. Using binocular stereo vision images, distance information that cannot be obtained from a single photograph can be acquired. First, the epipolar lines of the images are corrected to a common image plane using the parameters of the virtual binocular stereo vision system. This epipolar line correction involves aligning the two images from the virtual binocular system to the same plane, ensuring that corresponding pixel row coordinates are identical. The specific algorithm is as follows: (1) Solving for the fundamental matrix F: The fundamental matrix F describes the geometric constraints between corresponding points in the left and right images. F is calculated using an 8-point method by detecting feature points (such as Apriltags corner points) in the images. The formula is:

[0059] In formula (9): The coordinates of the feature points in the left image are homogeneous. These are the homogeneous coordinates of the corresponding feature points in the right image; F is the basic matrix, describing the geometric constraints between corresponding points in the left and right images.

[0060] (2) Mathematical model for epipolar correction: The rotation matrix R and translation vector T are obtained by decomposing the fundamental matrix F, and the left and right images are transformed into a parallel camera model. The corrected images satisfy:

[0061] In formula (10): H r H l To correct the transformation matrix, ensure that the epipolar lines of the corrected image are parallel to the x-axis and that the row coordinates of corresponding pixels are the same; P L The coordinates of the left image pixel; P R These are the coordinates of the corresponding pixel in the right image.

[0062] (3) Verification of line alignment error: Calculate the row coordinate difference of corresponding feature points in the corrected image to ensure that the average error is less than 1 pixel.

[0063] This process ensures that corresponding pixels have the same row coordinates. Image data alignment makes the image appear as if the two cameras are parallel.

[0064] In this process, the intrinsic parameter matrix K is used to transform the image from the pixel coordinate system to the camera coordinate system, and the distortion coefficients are used to correct the distortion of the image to ensure the accuracy of the correction.

[0065] Then, the disparity of the corrected image pixels is calculated using the semi-global matching (SGM algorithm) method; The parameter settings of the SGM algorithm directly affect the disparity calculation accuracy. The specific parameter determination method is as follows: (1) Estimation of parallax range: parallax range d min ~d max The focal length is determined by the camera movement distance t, the scene depth Z, and the camera focal length f, using the following formula: Formula (11) In formula (11): d is the disparity value; f is the camera focal length; t is the camera movement distance (calculated using Apriltags extrinsic parameters); Z represents the average depth of the structure to be tested (estimated by the dimensions of the design model).

[0066] For example, when f=1000 pixels, t=0.15m, Z=1.5m, d=100 pixels, the parallax range can be set to 0~64 pixels, covering the actual scene requirements; The focal length parameter in the intrinsic parameter matrix K is an important basis for estimating the disparity range, and the image after distortion coefficient correction can make the disparity calculation more accurate.

[0067] Using photo 1 from the camera as a reference image, calculate the disparity map with photo 2. It is important to note that in the disparity calculation, the approximate area of ​​the disparity must be estimated in advance to ensure that the pixel of the structure to be detected 3 is within the minimum and maximum horizontal displacement calculated.

[0068] Finally, a point cloud containing 3D coordinates corresponding to the pixels in the disparity map can be generated. The coordinates of the point cloud are represented as the same camera coordinates (x1, y1, and z1) relative to the optical center of the camera in the virtual binocular stereo vision system. The intrinsic parameter matrix K is used to convert the pixel coordinates in the disparity map into 3D coordinates in the camera coordinate system, thereby generating an accurate 3D point cloud.

[0069] Step 3: Extracting point cloud data of the structure to be detected and generating 3D point cloud data.

[0070] The point clouds generated by virtual binocular stereo vision systems often contain a large number of noisy points. To obtain a realistic model of each target structure 3, it is necessary to extract the relative points belonging to each target structure 3 from the point cloud. Point cloud extraction is completed in the following two sub-steps.

[0071] (1) Crude extraction Because point clouds contain a massive amount of point data, processing them is extremely difficult. Therefore, the first goal of extraction is to capture all points containing the structure to be detected (3) while removing background noise.

[0072] In the image, each module's surface has an Apriltags label2, thus identifying which module was detected. For each label, there is an ideal region (A) encompassing the entire structure to be detected. i It can be estimated based on its camera extrinsic parameters (R, T) and design model dimensions. This part is similar to the calculation of assembly error only in the previous part.

[0073] Next, all ideal regions calculated based on each label are combined. This yields the final overall ideal space. The coarse extraction based on the labels on the structure to be detected (structure 3) is as follows: Figure 6 As shown. It's important to note that the selected spaces (ΔX, ΔY, and ΔZ) will be slightly larger than the ideal space to ensure that the point cloud of the object under inspection can be completely enclosed, even if the structure under inspection 3 itself has some manufacturing errors. The calculation of the tag extrinsic parameters depends on the camera's intrinsic parameter matrix K and distortion coefficients; accurate intrinsic and distortion parameters ensure the accurate determination of the ideal region.

[0074] (2) Fine extraction After coarse extraction, most of the noise, including background noise, was removed. However, to obtain more accurate point clouds for each module, finer extraction is still needed for each module.

[0075] The fine extraction method of the module is as follows Figure 7 As shown.

[0076] In the ideal region (A) of each tag-based structure to be detected 3 i Under the following conditions, determine a value slightly larger than A. i The clipping space is used to extract points within the region. This step further removes a significant amount of noise. However, the target point cloud still contains some noise at this point. Therefore, Principal Component Analysis (PCA) is used to continue clipping points.

[0077] The PCA algorithm is used to determine the best-fit region of the point cloud. The specific steps are as follows: (1) Mathematical operations of PCA: Calculate the covariance matrix C of the point cloud:

[0078] In formula (12): C is the covariance matrix of the point cloud; n is the number of point clouds; P i Let be the coordinates of the i-th point in the point cloud; The mean of the point cloud; To perform a summation operation on n point cloud data.

[0079] Eigenvalue decomposition is performed on the covariance matrix C to obtain eigenvalues ​​λ1≥λ2≥λ3 and corresponding eigenvectors v1,v2,v3. The directions of the eigenvectors are the principal axis directions of the point cloud.

[0080] (2) Determination of the optimal fitting region: The spindle increment is 5% of the design model size. For example, if the design model size is 100mm, the spindle increment is 5mm. Calculate the boundary of the best-fit rectangle based on the principal axis direction and increment:

[0081] In formula (13): The coordinates of the minimum and maximum boundary points of the best-fit rectangle; The mean of the point cloud; k is the main axis increment (5% of the design model size); v iLet be the eigenvector of the i-th principal axis direction.

[0082] Based on the corner points and increments on each principal axis of the principal component analysis, an optimal fitting rectangle smaller than the previous clipping space can be determined. This rectangle is then used to further clip the target point cloud, thereby obtaining the final 3D point cloud data of the structure to be detected.

[0083] To obtain better extraction results, the incremental increment needs to be continuously adjusted based on the characteristics of different structures 3 to be detected and the project situation. It should be noted that it is impractical to crop the point cloud using the edges of traditional point cloud detection. Actual testing has shown that, generally, because these structures 3 to be detected are closely connected to other structures 3 to be detected next to them, there will be cases of missing detection edges.

[0084] After the fine extraction process is completed, a 3-point cloud of the structure to be tested, which can reflect the true assembly quality, is obtained.

[0085] The coordinate transformation of point clouds depends on the intrinsic parameter matrix K. Accurate intrinsic parameters can ensure the accuracy of PCA algorithm calculation, thereby determining the best fitting region.

[0086] Step 4: Multi-view point cloud stitching

[0087] In some cases, due to limitations of the shooting location and the large scale of the project, the facilities and equipment being installed cannot be fully captured in a single photograph. For example, in Figure 2 The image in question only captured one-third of the entire structure to be detected, 3. At this point, the point clouds generated from different images need to be aligned together.

[0088] Since each detectable structure 3 has Apriltags label 2, there is no need to search for features in the point cloud for matching, such as using an iterative nearest-point algorithm. Different detectable structures 3 can be aligned through label transformation, such as... Figure 4 As shown. The relative positions of the point clouds are based on the camera extrinsic parameters (R1, T1) of photo 1 and the camera extrinsic parameters (R... n ,T n The calculation is performed using formulas (6) and (8). Therefore, the point cloud of other structures to be detected (3) can be transformed into... Figure 1 The point cloud in the image is used to obtain the stitched point cloud.

[0089] During point cloud stitching, the intrinsic parameter matrix K and distortion coefficients are used to convert pixel coordinates into camera coordinates, ensuring the consistency of point cloud coordinates from different viewpoints and achieving high-precision point cloud stitching. Example 1

[0090] Example 1 was implemented in Python 3.6, using a Samsung Galaxy Note8 phone. The object to be detected was structure 3, with four Apriltags tags 2 (model Tag36h11) affixed to its surface. The following is in conjunction with the appendix... Figure 1 -8. Detailed explanation of the algorithm flow: Step 1: Single mobile device shooting and camera calibration

[0091] like Figure 1 As shown, multi-view photos are taken using a single moving camera device 1, and camera intrinsic parameter calibration is completed.

[0092] Multi-angle shooting: such as Figure 2 As shown, take at least two photos sequentially along the camera movement direction 4 at the initial phone position 5, the phone position after movement 6, and the phone position after n movements 7 (initial photo 8, photo after movement 9).

[0093] Shooting requirements: (1) Disable autofocus / exposure function; (2) The moving distance is ≤10% of the distance between the camera and the scene (e.g., when the shooting distance is 1.5m, the moving distance is ≤0.15m) to ensure that Apriltags tag 2 is fully visible; Camera intrinsic parameter calibration: such as Figure 8a As shown, a standard chessboard was selected as the calibration board.

[0094] Calibration process: (1) Environmental preparation: an indoor environment with uniform light and no reflection, and the calibration board is placed stably and without obstruction; (2) Take 16 photos from multiple angles (covering tilted, rotated, and near / far poses); (3) Feature point detection and intrinsic parameter solution: Zhang Zhengyou calibration method is used to detect the corner points of the calibration board and solve the intrinsic parameter matrix K; (4) Verification: The reprojection error must be <1 pixel (pass standard). Figure 8b As shown, the overall average error of reprojection is 0.51 pixels, then the output K matrix is: Formula (14) In formula (14): The focal lengths are in the x and y directions; These are the coordinates of the principal point, i.e., the pixel coordinates of the image center.

[0095] Step 2: Construction of Virtual Binocular Vision System like Figure 1 As shown, a virtual binocular system is constructed based on Apriltags extrinsic parameters to generate a 3D point cloud.

[0096] External parameter calculation and optimization: such as Figure 3 As shown, the virtual stereo system comprises three coordinate systems: camera coordinates (x1, y1, z1), moving camera coordinates (x2, y2, z2), and label coordinates (x3, y3, z3). Figure 4 As shown, the extrinsic parameters (R1,T1,R2,T2) of Apriltags label 2 in the photo are detected, and the relative position of the camera (R1,T1,R2,T2) is calculated using formulas (5)-(7). s ,T s If there are n identical labels, the result can be obtained by optimizing using formula (8) (average of multiple label extrinsic parameters) (R). once ,T once ): Formula (8) Virtual binocular system generation: such as Figure 5 As shown, combining the intrinsic parameter matrix K and (R once ,T once Constructing a virtual stereo system includes the following steps: (1) Image correction: The epipolar constraint algorithm is used to correct Photo 1 and Photo 2 to a common plane, so that the row coordinates of corresponding pixels are the same. After correction, the row alignment error is less than 1 pixel, ensuring the effectiveness of the parallel camera model.

[0097] (2) Disparity calculation: The semi-global matching (SGM) algorithm is used, and the disparity range is set to 0-64 pixels to calculate the disparity map between photo 1 and photo 2.

[0098] (3) Point cloud generation: Based on the disparity map and camera parameters, a three-dimensional point cloud is generated, with coordinates corresponding to the camera optical center (x1, y1, z1) of the virtual system.

[0099] Step 3: Point cloud extraction of the structure to be detected like Figure 1 As shown, noise is removed through coarse extraction and fine extraction to obtain a realistic point cloud.

[0100] Coarse extraction: such as Figure 6 As shown, the point cloud contains pixels 10 and background noise 11. Based on the extrinsic parameters of Apriltags label 2 and the design model size, the ideal region 12 is obtained. Combining the ideal regions of all labels, the overall ideal space is obtained. Based on the ideal region 12, the cropping region 15 is obtained by expanding by 10%. The background noise 11 is removed by cropping, resulting in the denoised point cloud 14.

[0101] Fine extraction: such as Figure 7 As shown, the steps are as follows: 1. Using the Principal Component Analysis (PCA) algorithm, with the PCA principal axis increment being 5% of the design model size, determine the best-fit rectangle; 2. The actual point cloud of the region to be detected 16 is obtained by cropping, which reflects the actual shape of the structure to be detected 3.

[0102] Step 4: Multi-view point cloud stitching

[0103] like Figure 1 As shown, point cloud stitching is achieved based on Apriltags coordinate transformation.

[0104] like Figure 4 As shown, without using the ICP algorithm, the relative positions of point clouds from different perspectives are calculated directly through the extrinsic parameters of Apriltags label 2. The point clouds generated by the initial mobile phone position 5, the mobile phone position 6 after movement, ..., the mobile phone position 7 after n movements are aligned to complete the complete point cloud reconstruction of the structure to be detected 3.

[0105] The performance verification of the multi-view vision algorithm in Example 1 is as follows: (1) Accuracy comparison The inverse surface is aligned with the laser-scanned reference mesh, and the surface is fine-tuned using a nearest neighbor algorithm. Finally, the error of the inverse surface relative to the reference mesh can be calculated, representing the error compared to using the proposed method and 3D laser scanning.

[0106] The error distribution on the reference grid is as follows Figure 9 As shown, darker colors indicate greater deviation in the results. Figure 9 It can be seen that, except for the junction of the depressions and some local areas at the edges, the error in most areas is less than 1mm, and the overall consistency is relatively high.

[0107] Quantitative data acquisition: The accuracy comparison in Example 1 is based on an error analysis of a high-precision three-dimensional laser scanning reference model and an inverse model generated by the algorithm of this invention, including the following steps: ① Reference model construction: The actual assembly module was scanned using a handheld laser scanner (parameters are shown in Table 1 for the main parameters of the 3D scanner used) to obtain reference point clouds and perform Mesh processing (generating triangular mesh surfaces) as a benchmark for error comparison.

[0108]

[0109] ② Inverse model generation: Inverse point clouds are generated based on the algorithm of this invention, such as... Figure 10 As shown, the inverse surface is obtained by NURBS surface fitting.

[0110] ③ Error calculation: Align the reverse surface with the reference mesh, calculate the nearest distance error for each point, and calculate the mean, standard deviation and RMS value.

[0111] Quantitative data: The average error is +0.49mm / -0.62mm, the standard deviation is 0.75mm, the RMS estimate is 0.77mm, and the probability of the error value being <1.5mm is 95%, which meets the accuracy requirement of ±2mm (in this case).

[0112] (2) Efficiency verification The efficiency verification of Example 1 is based on the stitching time test of 1 million point clouds, and includes the following steps: ①Test environment: CPU Intel Core i7-8700K, memory 16GB, GPU NVIDIA GeForce GTX1080Ti.

[0113] ② Test data: Select 1 million point cloud data (point cloud density 100 points / mm²).

[0114] ③ Measurement of splicing time: Record the splicing time of the algorithm of this invention and the traditional ICP algorithm. The splicing time of the algorithm of this invention is ≤5s, and the splicing time of the traditional ICP algorithm is ≥7s.

[0115] Point cloud stitching based on Apriltags coordinate transformation does not require the ICP algorithm, and the stitching time for 1 million point clouds is ≤5s, which is 30% more efficient than the ICP algorithm, thus solving the problem of low efficiency in existing point cloud stitching.

[0116] (3) Robustness verification When Apriltags label 2 is partially occluded, the algorithm can still maintain an accuracy of ≥95% through multi-view fusion, indicating that the algorithm has strong adaptability to complex on-site environments.

[0117] The reverse model generated by the algorithm in this embodiment can accurately reflect the actual module deformation, and the error with the laser scanning results is controllable. It also has the advantages of handheld shooting and low cost (only a consumer-grade mobile phone is required), and can be used as an effective means of digital quality monitoring.

[0118] The above description of the embodiments is provided to enable those skilled in the art to understand and use the present invention. It will be apparent to those skilled in the art that various modifications can be easily made to these embodiments, and the general principles described herein can be applied to other embodiments without inventive effort. Therefore, the present invention is not limited to the above embodiments, and any improvements and modifications made by those skilled in the art based on the disclosure of the present invention without departing from the scope of the present invention are within the protection scope of the present invention.

Claims

1. A multi-view vision algorithm based on Apriltags, characterized in that, Includes the following steps: Step S1. Use a single moving camera device to capture at least two photos of the structure to be detected that contain the same Apriltags, where the same Apriltags refer to the same physical tag; Step S2. Calculate the relative position of the camera based on the extrinsic parameters of Apriltags in the photo, and construct a virtual binocular stereo vision system; if there are multiple identical Apriltags, optimize the camera pose accuracy by averaging the extrinsic parameters of multiple tags. Step S3. Use the virtual binocular stereo vision system to generate a three-dimensional point cloud, and perform coarse extraction and fine extraction processing on the point cloud in sequence to obtain the real point cloud of the structure to be detected; Step S4. Based on the coordinate transformation relationship of Apriltags, align the point clouds generated from different photos to complete the point cloud stitching.

2. The multi-view vision algorithm based on Apriltags according to claim 1, characterized in that, Step S1 also includes camera intrinsic parameter calibration, which includes: a. Perform camera intrinsic parameter calibration on a single moving camera device and obtain the camera intrinsic parameter matrix K; b. Take at least two photographs of the target scene containing the same Apriltags, where the same Apriltags refer to the same physical tag.

3. The multi-view vision algorithm based on Apriltags according to claim 2, characterized in that, The camera intrinsic parameter calibration in step S1 includes: a1. Device settings: Turn off the autofocus and auto exposure functions of the single-moving camera device, and fix the focal length; a2. Take calibration photos: Using the standard calibration board as a reference, take photos from multiple angles around the calibration board to collect 15-20 valid photos; a3. Intrinsic parameter calculation: Using the Zhang Zhengyou calibration method, the camera intrinsic parameter matrix K and distortion coefficients are solved by detecting feature points in the calibration photos. The distortion coefficients include radial distortion coefficients k1, k2, k3 and tangential distortion coefficients p1, p2. a4. Verification: Calculate the reprojection error of feature points and ensure that the average reprojection error is less than 1 pixel. If the error exceeds the standard, recalibrate.

4. The multi-view vision algorithm based on Apriltags according to claim 1, characterized in that, The shooting described in step S1 satisfies the following conditions: autofocus is disabled and the image resolution remains unchanged.

5. The multi-view vision algorithm based on Apriltags according to claim 1, characterized in that, The construction of the virtual binocular stereo vision system in step S2 includes: a. Epipolar correction: The rotation matrix R and translation vector T are obtained by decomposing the fundamental matrix F. The image of the photograph is corrected to a parallel camera model so that the row coordinates of corresponding pixels are the same and the average row alignment error is <1 pixel. b. Disparity Calculation: The disparity of the corrected image pixels is calculated using the semi-global matching (SGM) method. The SGM parameters include: Parallax range: Calculated using formula (11): Official (11), In formula (11): d is the disparity value; f is the camera focal length; t is the distance the camera moves; Z represents the average depth of the structure to be tested; c. Point cloud generation: Generate a 3D point cloud based on the disparity map and camera parameters.

6. The multi-view vision algorithm based on Apriltags according to claim 1, characterized in that, The specific method for averaging the multi-label extrinsic parameters in step S2 is as follows: If n identical Apriltags labels are detected, the averaged camera extrinsic parameters are calculated using formula (8): Formula (8) In formula (8), R once : The averaged camera extrinsic parameter rotation matrix; T once : The averaged translation vector of the camera's extrinsic parameters; n: The total number of identical Apriltags detected; R S The camera relative rotation matrix corresponding to a single label; T S The camera's relative translation vector corresponding to a single label; R S ,T S : The camera extrinsic rotation matrix and translation vector corresponding to the i-th label; : Perform a summation operation on the extrinsic parameters of n labels.

7. The multi-view vision algorithm based on Apriltags according to claim 1, characterized in that, The coarse extraction in step S3 includes: combining the ideal regions corresponding to each Apriltags label to determine the overall ideal space covering the structure to be detected, cropping the point cloud to remove background noise, and obtaining the cropped region.

8. The multi-view vision algorithm based on Apriltags according to claim 7, characterized in that, The ideal region refers to the theoretical three-dimensional pixel region calculated based on Apriltags tags, and the cropping region refers to the spatial range expanded by 10% based on the ideal region.

9. The multi-view vision algorithm based on Apriltags according to claim 1, characterized in that, The fine extraction described in step S3 includes: a. Using the principal component analysis (PCA) algorithm, calculate the covariance matrix of the point cloud and perform eigenvalue decomposition to determine the principal axis direction; b. The PCA principal axis increment is 5% of the design model size. The best-fit rectangle is determined based on the principal axis direction and increment. The point cloud is cropped to remove the remaining background noise and obtain the region to be detected.

10. The multi-view vision algorithm based on Apriltags according to claim 1, characterized in that, The point cloud stitching described in step S4 does not require the use of the Iterative Nearest Point (ICP) algorithm; the alignment is achieved directly by calculating the relative positions of the point clouds using the extrinsic parameters of the Apriltags labels.

Images