Intelligent component three-dimensional position acceptance method based on image and laser data fusion

Abstract

The invention discloses an intelligent component three-dimensional position acceptance method based on image and laser data fusion, and the method comprises the steps: (1) providing an oblique photography automatic correction model and employing a step-by-step detection method for solving a problem that the position of a component is difficult to accurately mark and locate by employing a single target detection network, the analysis process of the component image is divided into two independent steps of target detection and positioning. And (2) providing a point cloud segmentation and point cloud and image registration method based on environmental perception for the problem that the pose of the camera and the laser radar is difficult to calibrate in real time. By fusing the above method, the method can be successfully applied to quality detection and evaluation of the embedded plate of a certain nuclear power plant, efficient data acquisition and automatic processing are realized, and the result shows that the method can automatically analyze the three-dimensional position, the rotation angle and the missing of one wall member within 3 min, and visually displays the collision inspection result in combination with the BIM model.

Description

Technical Field

[0001] The invention belongs to the field of structural health monitoring, and in particular relates to an intelligent acceptance method for the three-dimensional position of a component based on image and laser data fusion. Background Art

[0002] With the development of modern prefabricated buildings, the requirements for project quality are getting higher and higher, and the quality acceptance of prefabricated components is particularly important. The installation quality of embedded steel plates in prefabricated components directly affects the quality and safety of the entire building and its use function after production. For curtain wall structures, the columns are mainly connected to the main structure by angle brackets or adapters to achieve force transmission and load-bearing. The connection quality between the angle brackets or adapters and the embedded steel plates of the main structure directly affects the bearing capacity and safety, and is of utmost importance. If the embedded steel plate is misaligned, it may directly lead to insufficient weld length between it and the angle brackets or adapters, and severe misalignment may even cause the two to be unable to connect normally, resulting in invalid installation and serious safety hazards. The quality acceptance of embedded wall plates is labor-intensive and consumes a lot of manpower and material resources, so precise, automated, and intelligent acceptance methods and equipment are required.

[0003] With the development of computer vision technology, target detection and recognition has become an important research field in structural engineering, traffic license plate detection and remote sensing photography, and performance detection based on image data has received widespread attention. In the field of structural engineering health monitoring, computer vision technology occupies a very important part. For example, digital image correlation (DIC), photogrammetry technology and motion capture system (MCS), for the case where the target moves in a plane in the structural system (such as the XY plane), the projection relationship can be simplified to the plane homography between the 2D structural plane and the 2D image plane. The result of using the plane homography reconstruction is usually the two-dimensional structural displacement of the target. The calibration process requires at least four sets of 2D to 2D point correspondences. In the field of traffic license plate detection, computer vision technology has been successfully applied. Many scholars have provided many methods for image tilt correction, which can achieve real-time detection, but they often cannot achieve both high accuracy and large field of view.

[0004] Traditional methods of ensuring the correct placement of prefabricated steel plates require manual inspection, which is time-consuming and expensive. In order to ensure the quality of prefabricated components, various technologies including BIM and 3D laser scanning have been studied. The method of using laser scanning to inspect the quality of prefabricated components can achieve high accuracy, but it is often non-automated and the whole process is very time-consuming. The method of fusing images with laser point clouds provides the possibility of more accurate and rapid acceptance of prefabricated components, so external calibration between the two sensors, LiDAR and camera, becomes crucial. Based on manual targets, these external calibration methods require engineers to manually specify the correspondence between LiDAR point clouds and camera images based on one or more calibration targets (such as chessboard patterns), ArUco labels, custom planar targets, and ordinary boxes. Traditional LiDAR camera calibration methods usually rely on explicit targets or manual intervention, which can be very expensive and cumbersome.

[0005] Most of the research focuses on methods that rely solely on images or laser point clouds to achieve detection and measurement of targets. Most vision-based measurement technologies are limited by the shooting angle and field of view, making general vision-based methods only applicable to a certain angle and distance range. Because general construction sites cannot provide an orthographic perspective, the camera can only be placed on the side for shooting, which makes most methods unsuitable for pre-installed component inspection. Summary of the invention

[0006] Purpose of the invention: The purpose of the present invention is to provide an intelligent acceptance method for the three-dimensional position of components based on image and laser data fusion, so as to solve the problem of low efficiency in detection of missing embedded plates and geometric quality information in cast-in-place concrete structures, and to realize visual collision detection between embedded plates and design models.

[0007] Technical solution: The present invention provides an intelligent acceptance method for the three-dimensional position of a component based on image and laser data fusion, characterized in that it comprises the following steps:

[0008] S1. Acquire the buried plate image through the industrial camera platform and automatically perform tilt correction on the image to obtain the corrected buried plate image;

[0009] S2, perform two-stage detection on the corrected buried plate image based on scale unification;

[0010] S3, integrates the adaptive calibration results of the image and point cloud data, aligns the point cloud depth information corresponding to the corner points in the image, outputs the comparison results between the measured value and the designed value, and completes the automated analysis from data acquisition to embedded plate acceptance.

[0011] Furthermore, in step 1, the industrial camera platform includes a Hikvision MV-CE120-10UC industrial camera, a Mid-70 solid-state area array laser radar, and is equipped with an Intel embedded micro industrial computer NUC to achieve real-time detection, and is configured with a portable VT-88 tripod gimbal.

[0012] Furthermore, step 1 is specifically as follows:

[0013] Design shooting mode. In view of the uneven direction of the wall, divide it into orthogonal walls along the X or Y direction and inclined walls with an angle with the X or Y direction. According to the direction of the wall, the local coordinate system of the detected wall is transferred to the nuclear power plant reference coordinate system, and then the data is compared. The pinhole model is used as the imaging model of the camera;

[0014] Use homography to correct the image to orthophoto, express the initial image pixel coordinates as (x, y), express the image pixel coordinates under the orthographic perspective as (x', y'), and the homography transformation relationship is:

[0015]

[0016] Write equation (1) in the form of matrix AX=0:

[0017]

[0018] According to equation (2), when solving H, we add the constraint h 33 = 1. Although the homography matrix H has 9 unknowns, it has only 8 degrees of freedom, so there are still h 11 ~h 32 A total of 8 unknowns due to a set of matching points (x i ,y i )match(x i ',y i ') to obtain two sets of equations, then n = 4 sets of non-collinear matching points are needed to find the unique solution of H;

[0019] A two-stage image automatic correction method based on known markers is adopted. In the first stage, a deep learning method is used to detect the target and determine the center position of the target. In the second stage, sub-pixel positioning is performed based on the designed target.

[0020] Furthermore, the first stage uses a deep learning method to detect targets and determine the center position of the targets. Specifically, the Cornernet-Lite network is used to detect four targets based on key points, and the upper left corner point (x ltop ,y ltop ), the lower right corner (x rdown ,y rdown ), calculate the center point (xi ,y i ), and obtain the homography transformation matrix H according to equation (2): 1 ;

[0021]

[0022] Furthermore, the second stage performs sub-pixel positioning based on the designed target as follows: according to the color features of the three-channel (R, G, B) image, the G channel is extracted as a grayscale image, the edge features of the grayscale image are extracted, all edge contours are sorted according to the size of the connected domain area, the required contours are obtained, and the centroid coordinates of the target are obtained by fitting, as shown in equation (4):

[0023]

[0024] According to equation (2), the homography transformation matrix H is obtained 2 , substituted into equation (5) to achieve the two-stage automatic view recovery of the tilted view;

[0025]

[0026] Furthermore, step 2 is specifically as follows: first, based on the Cornernet-Lite network of efficient key point detection, the target is segmented into monomers in real time, and then the monomer target image is input to automatically detect the corner points of the target using the Superpoint network, and the buried plate image is scaled so that the scaled buried plate image is close to the input size of the corner point detection model.

[0027] Furthermore, step 3 specifically includes the following steps:

[0028] Step 3.1: segment the point cloud using fitting residuals, color features and reflection intensity; where:

[0029] Fitting residuals: Any object has a non-zero height relative to the plane it is located on, so the fitting residual of a point corresponds to the relative height to the wall;

[0030] Color features: RGB color values ​​are useful information for distinguishing templates from walls. The visible light band template difference index VDVI is introduced to reduce misclassification caused by templates. The VDVI calculation formula is as follows:

[0031]

[0032] Reflection intensity: The reflection intensity recorded by the LiDAR system measures the energy of the laser beam reflected from the surface of an object. To compare the intensity between different walls and buried panels, an empirical model is used to adjust the intensity value:

[0033]

[0034] Among them, I correct and I orig are the corrected intensity value and the original intensity value respectively; D is the distance from the lidar sensor to the wall sampling point; D ref is the reference distance; a is the incident angle, and the steel plate is distinguished from the wall point by adjusting the intensity value;

[0035] Step 3.2, use the local DBSCAN clustering method to cluster the points in the feature space. DBSCAN requires two parameters: Eps and MinPts; Eps is used to find neighbors around the point, that is, the point whose distance to point P is less than Eps belongs to the neighbor of P; MinPts is the minimum number of neighbors within the Eps radius; based on this information, three types of points are defined: core points, boundary points, and noise points; points with more than MinPts neighboring points in Eps are defined as core points; boundary points are defined as points with fewer than MinPts neighboring points in Eps but near the core point; a point that is neither a core point nor a boundary point belongs to noise; MinPts is greater than the data dimension, set MinPts = 5, and at the same time, Eps can be obtained through the kd distance curve. For each LiDAR point, first calculate its distance to the MinPts-th nearest neighbor, then sort the distances of all points from the maximum to the minimum, construct a distance curve, and finally take the Eps value corresponding to the inflection point of the curve as the optimal Eps value;

[0036] Step 3.3: Calibrate the mechanical energy of the LiDAR camera system, adjust the external parameters of the LiDAR-camera system, and calibrate the camera and LiDAR poses in two stages.

[0037] Furthermore, step 3.2 specifically includes the following steps:

[0038] Step 3.2.1, normalize the eigenvalues ​​using min-max scaling;

[0039] Step 3.2.2, calculate the optimal Eps according to the MinPts-th distance curve; here, MinPts = 5, point P i and P j The distance between ij )Use normalized features calculate:

[0040]

[0041] Among them, α, β, and γ are weights; α = 1, β = 1, and γ = 1, indicating that these four features are equally important for estimating distance;

[0042] Step 3.2.3, randomly select a core point P from the unvisited points;

[0043] Step 3.2.4: If the core point P does not belong to any cluster, generate a new cluster;

[0044] Step 3.2.5, find all density-connected points of P and assign them to clusters of P;

[0045] Step 3.2.6: Repeat steps 3.2.3 to 3.2.5 until all points are visited. Points that do not belong to any cluster are treated as noise.

[0046] Furthermore, in step 3.3, the mechanical energy of the LiDAR camera system is calibrated and the external parameters of the LiDAR-camera system are adjusted as follows: a model of the calibration object and the LiDAR camera system is constructed, and the coordinate system of the LiDAR is represented as o l -x l y l z l , the origin represents the position of the LiDAR, and the camera's coordinate system is defined as o c -x c y c z c , the origin represents the position of the camera; A i (i=1,2) refers to the corner points on the auxiliary calibration object, and at least four auxiliary calibration objects are used; the external parameters of the LiDAR-camera system are the rotation matrix R and the transformation vector T; R and T represent the LiDAR coordinate system o d -x d y d z d With the camera coordinate system o c -x c y c z c The rigid body transformation between them, the rotation angles are α, β and γ, and the translations are Δx, Δy and Δz. The calibration method aims to estimate the rotation matrix R and the translation matrix T. The camera and the lidar are integrated together and their relative positions remain unchanged, so α, β and γ are very small, and Δx, Δy and Δz are also small. According to equation (9), from the camera coordinate system o c -x c y c z c To pixel coordinate system o c -uv does not rotate, OP in the camera coordinate system c The angle θ between op and the x-axis in the pixel coordinate system remains unchanged;

[0047]

[0048] Furthermore, in step 3.3, the camera and lidar poses are calibrated in two stages: in the first stage of the camera and lidar pose calibration, the center point O(a, b, x, y) of the embedded plate is defined, where a and b are the length and width of the embedded plate, x and y are the coordinates of the center point of the embedded plate, w and h are the width and height of the image, ab is the shape factor of the embedded plate, x / y is the center point vector, and the point cloud descriptor D is defined according to the quadrant n where the embedded plate is located. point (a / b,(-1) n x / y), the image descriptor is D image (w / h·(a / b),(-1) n x / y); the coordinate set of the center point of the embedded plate in the three-dimensional point cloud is S point , the center point coordinate set of the buried plate is S image , according to equation (10), the 3D point cloud is matched with the pixel interior point, and the initial R is solved based on the PnP method. init With T init ;

[0049]

[0050] The second stage of camera and lidar pose calibration is shown in formula (11), S i ' mage Using R init With T init To indicate that when S i ' mage With S image When the absolute difference is not greater than ε, S image With S point is a pair of matching points. Formula (12) is used as the objective function and the RANSAC method is used to obtain the point With S image Repeated sampling to select interior points I point with I image , calculate R and T;

[0051]

[0052] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:

[0053] (1) The present invention proposes an automatic correction model for oblique photography and adopts a step-by-step detection method, which divides the analysis process of the embedded panel image into two independent steps: target detection and positioning. This solves the problem that a single target detection network is difficult to accurately mark and locate the position of the embedded panel. Indoor tests show that the accuracy of the two-stage correction model is improved by 52.5% to 1.62 mm. The accuracy of the field repeated positioning test is 1.45 mm, which further verifies the effectiveness of the proposed method.

[0054] (2) The developed laser point cloud and image adaptive calibration method based on environmental perception uses the embedded plate as an auxiliary calibration object. Through DBSCAN point cloud segmentation and image target detection, it successfully realizes the adaptive calibration of the on-site equipment. The reprojection error is reduced by 0.13 pixels compared with the laboratory, which is consistent with the laboratory calibration results overall.

[0055] (3) Integrating the above technologies, the present invention further developed a camera and lidar hardware and software integrated detection and automatic acceptance system with edge computing functions. The developed algorithm and system were successfully applied to the quality inspection and evaluation of embedded panels in a nuclear power plant, achieving efficient data collection and automatic processing. The results showed that this method can automatically analyze the three-dimensional position, rotation angle and missing problems of a wall embedded panel within 3 minutes, and combine the design model to display the error information in a visual effect.

[0056] (4) The present invention takes a close-up, all-round photograph of the component based on the marker to overcome the adverse effects caused by angle and distance. Most of the neural networks used in target detection technologies based on vision and deep learning are trained using the same model in both component detection and classification steps, resulting in low accuracy. The present invention uses a key point-based Cornernet-Lite network for component detection, and then uses a Superpoint network for component corner point detection, which is targeted and improves the speed and accuracy of detection. The calibration method between the laser radar and camera sensors heavily relies on manual targets (such as a plane chessboard) and cannot achieve real-time adaptive calibration. The external parameters of the two sensors are calibrated using the DBSCAN-based point cloud segmentation and PnP method to obtain the depth information of the target. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 This is the main research idea in the method of the present invention;

[0058] Figure 2 It is a schematic diagram of a hardware-software integrated platform based on the fusion of image and laser point cloud in the method of the present invention;

[0059] Figure 3 It is a schematic diagram of the automated detection and acceptance system in the method of the present invention;

[0060] Figure 4 It is a schematic diagram of the wall direction and components in the method of the present invention;

[0061] Figure 5 is a schematic diagram of a camera imaging model in the method of the present invention;

[0062] Figure 6 It is a schematic diagram of the two-stage automatic image correction method in the method of the present invention;

[0063] Figure 7 It is a schematic diagram of the method for detecting buried plates in stages in the method of the present invention;

[0064] Figure 8 It is a schematic diagram of the DBSCAN clustering method in the method of the present invention;

[0065] Fig. 9 is a schematic diagram of an adaptive calibration model of a camera and a laser radar in the method of the present invention;

[0066] Fig.10 is a schematic diagram of data collection and processing in an embodiment of the present invention;

[0067] Fig.11 is the result of the two-stage automatic correction of the image in the embodiment of the present invention;

[0068] Fig.12 is a comparison of image measurement results in an embodiment of the present invention;

[0069] Fig.13 is the adaptive fusion calibration result of the image and the laser point cloud in the embodiment of the present invention;

[0070] Fig.14 are test results of different viewing angles in the embodiments of the present invention;

[0071] Fig.15 This is the result analysis of the field test in the embodiment of the present invention. DETAILED DESCRIPTION

[0072] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0073] In order to solve the problem of low efficiency in detecting missing embedded plates and geometric quality information in cast-in-place concrete structures, and realize visual collision detection between embedded plates and design models, the present invention discloses a method for automated quality inspection and acceptance of massive embedded plates based on a camera and lidar fusion platform and a visual inspection algorithm. The framework diagram of the method is shown in the figure. Figure 2 As shown. From the system level, this method can be divided into three parts:

[0074] S1, embedded plate image acquisition and automatic tilt correction based on industrial camera;

[0075] S2, a two-stage buried plate detection method based on scale uniformity consideration;

[0076] S3, automatic step-by-step component detection method based on deep learning.

[0077] like Figure 3As shown, when selecting the hardware platform, the present invention comprehensively considers the four aspects of endurance, positioning accuracy, camera focal length and computing platform. The final assembled detection system consists of: the camera is a Hikvision MV-CE120-10UC industrial camera, a Mid-70 solid-state array laser radar, and an Intel embedded micro industrial computer NUC is equipped to achieve real-time detection, and the battery life is up to 3.5 hours. The fuselage is designed as a high-strength aluminum alloy frame with good heat dissipation performance, and is equipped with a portable VT-88 tripod pan / tilt. The industrial camera is equipped with a 6mm fixed-focus lens to achieve large field of view shooting; the laser radar has a non-repetitive point cloud output capacity of 200,000 points / second. The developed equipment is used for geometric quality inspection of embedded panels, and an automatic acceptance algorithm is integrated at the same time. For the large number of embedded panels in nuclear power plants (several thousand panels in a single plant), on the one hand, the embedded panel design value can be stored locally, and the equipment simultaneously performs data acquisition and processing, and outputs comparison results; on the other hand, the equipment, as a data acquisition terminal, transmits data in real time through a local area network, realizes data cloud processing, and feeds back comparison results.

[0078] The specific contents of the embedded plate image acquisition and automatic tilt correction based on industrial camera in S1 are as follows:

[0079] The developed equipment has the advantage of quickly obtaining the three-dimensional position when detecting buried plates (3 minutes for a single wall). In order to adapt to the actual situation on site (shooting space and wall direction at the construction site), we designed a shooting mode. Figure 4 As shown in the figure, in order to solve the problem of inconsistent wall directions, the walls are often divided into orthogonal walls (along the X or Y direction) and inclined walls (at an angle with the X or Y direction). We transfer the local coordinate system of the detection wall to the nuclear power plant reference coordinate system according to the wall direction, and then compare the data. In order to solve the problem of space limitation due to equipment stacking, the equipment is often tilted to collect data, and targets of known size are designed for automatic correction of tilted images.

[0080] The most commonly used imaging model for cameras is the pinhole model, which is described in a large number of literatures and will not be described in detail here. The present invention uses a fixed-focus camera, and the tilt angle θ can be a variable during the shooting process, such as Figure 5 As shown, the model is modified. Based on the pinhole imaging model, if there is a tilt angle, the photographed object will produce shear deformation compared with the object shape in three-dimensional space. Figure 5 shown.

[0081] Since it is difficult to ensure that the camera is facing the embedded parts when shooting, the originally rectangular embedded parts will be reflected as arbitrary quadrilaterals in the image (such as Figure 5As shown). Use homography transformation to correct the image to orthographic. The homography transformation of an image describes the projection relationship between two images on a plane, and can be used to process geometric transformations such as rotation, scaling, translation, and distortion of an image on a plane. The initial image pixel coordinates are represented as (x, y), and the image pixel coordinates under the orthographic perspective are represented as (x', y'). The homography transformation relationship is:

[0082]

[0083] Write equation (1) in the form of matrix AX=0:

[0084]

[0085] According to equation (2), when solving H, we usually add the constraint h 33 = 1. Although the homography matrix H has 9 unknowns, it has only 8 degrees of freedom, so there are still h 11 ~h 32 A total of 8 unknowns due to a set of matching points Two sets of equations can be obtained, so only n=4 sets of non-collinear matching points are needed to find the unique solution of H.

[0086] In order to improve the detection efficiency and accuracy, the present invention adopts a two-stage image automatic correction method based on known markers (such as Figure 6 In the first stage, deep learning methods are used to detect targets and determine the center position of the targets. Cornernet-Lite is used to detect four targets based on key points, and the upper left corner point (x ltop ,y ltop ), the lower right corner (x rdown ,y rdown ), calculate the center point (x i ,y i ). According to equation (2), the homography transformation matrix H is obtained 1 .

[0087]

[0088] The second stage is to perform sub-pixel positioning based on the designed target. The designed target is Figure 6 As shown, the outer part is a red rectangle and the inner part is a green circle. First, according to the color features of the three-channel (R, G, B) image, the G channel is extracted as the grayscale image (as shown in Figure 6 Extract the edge features of the grayscale image, sort all edge contours according to the size of the connected domain area, the green contour is what is needed, and fit the centroid coordinates of the target, as shown in equation (4).

[0089]

[0090] According to equation (2), the homography transformation matrix H is obtained 2 , substituting into equation (5) to realize the two-stage automatic view restoration of tilted view.

[0091]

[0092] The specific contents of the two-stage buried plate detection method based on scale uniformity consideration in S2 are as follows:

[0093] The purpose of performing step-by-step detection on the acquired embedded panel images is to unify the size of the individual embedded panels segmented after embedded panel recognition as much as possible, and to enhance the details as much as possible to ensure that the final target corner point determination can maintain a stable and high-precision judgment. Figure 7 As shown, the core of the designed step-by-step detection method is to detect and segment the buried plate in the first stage, and scale the buried plate image (generally enlarge it) in the second stage so that the scaled buried plate image is close to the input size of the corner detection model, that is, 240×320.

[0094] The fusion device developed by the present invention is equipped with an Intel embedded micro-industrial computer NUC to realize edge computing functions. Considering the computing resources and endurance of the integrated device, the development algorithm needs to take into account low power consumption and real-time computing capabilities. CornerNet is a neural network architecture for target detection proposed by Hei Law and Jia Deng from the University of California, Berkeley. Unlike traditional target detection methods, CornerNet focuses on detecting the four corners of an object (i.e., the upper left corner, upper right corner, lower left corner, and lower right corner), rather than the center point and width and height of a rectangular bounding box. CornerNet consists of two main parts: CornerNet-Keypoint and CornerNet-Squeeze. CornerNet-Keypoint uses an architecture called the Hourglass network to convert the input image into a feature map and predict the coordinates of the four corner points of each object. CornerNet-Squeeze converts this corner point information into the final detection result of the object, including the object category and the position of the bounding box. The advantage of CornerNet is that it can accurately detect objects of various sizes and shapes, especially in the detection of small objects. In addition, since it does not require pre-defined fixed anchor boxes, it can better adapt to different target shapes and scales, while also reducing the number and complexity of hyperparameters. CornerNet-Lite is a lightweight version of CornerNet that aims to improve speed and efficiency while maintaining relatively high detection accuracy. It is designed to adapt to environments with limited computing resources, such as real-time target detection on mobile devices or edge computing devices. Therefore, the present invention adopts CornerNet-Lite in the first stage of automatic detection of buried panels.

[0095] Superpoint is an image feature point detection and description algorithm based on Convolutional Neural Network (CNN). It was proposed by Daniel DeTone, Tomasz Malisiewicz and Andrew Rabinovich from the University of California, Berkeley. The goal of Superpoint is to detect keypoints in an image and generate a high-dimensional descriptor for each keypoint. These keypoints and descriptors are widely used in computer vision tasks such as image matching, tracking, and 3D reconstruction. The architecture of the Superpoint network mainly consists of two parts: Feature Extraction Network and Descriptor Generation Network. The Feature Extraction Network is a convolutional neural network that extracts feature maps from the input image. The Descriptor Generation Network is a fully connected layer that converts each pixel in the feature map into a corresponding descriptor vector. The advantage of Superpoint is that it can robustly detect a large number of keypoints in the input image and generate high-quality descriptors for each keypoint. It can automatically learn the features and structures in the image by using convolutional neural networks for end-to-end learning, thereby providing more accurate keypoint detection and descriptor matching. In addition, Superpoint also has a faster computing speed and lower memory consumption, and is suitable for real-time applications and resource-constrained environments. Therefore, the present invention uses Superpoint in the second stage of embedded board automatic detection.

[0096] The specific contents of the automatic step-by-step detection method of components based on deep learning in S3 are as follows:

[0097] The device adaptive calibration method based on environmental perception is mainly divided into three steps: data preprocessing, multi-feature based point cloud segmentation and key point selection. The features of each key point include three-dimensional coordinates (XYZ), reflectivity intensity value, and RGB color value. Laser point cloud data filtering is of great significance in applications because it is directly related to the quality of point cloud data and the effect of subsequent applications. Statistical filtering is a commonly used method in point cloud processing. It removes noise by analyzing the neighborhood statistics of points. It is very effective in removing random noise, especially when noise points are clearly distinguished from real data points in space. Segmentation-based methods are expected to be a good solution for filtering point clouds; however, their success depends largely on the premise that the point cloud is accurately segmented. In practice, due to the lack of additional useful information, classical segmentation methods are prone to produce biased fragments on complex landscapes. Therefore, the present invention proposes a multi-feature segmentation method based on DBSCAN for point cloud clustering.

[0098] DBSCAN is a satisfactory clustering method that can find segments of arbitrary shapes in the presence of noise without presetting the number of segments. Using only XYZ information to cluster point clouds will lead to biased results on complex walls. In order to solve the above problems, the present invention uses both geometric and radiation features. Specifically, we use the following features to segment the point cloud: (1) Fitting residual: Any object has a non-zero height relative to its plane position. Therefore, the fitting residual of a point corresponds to the relative height to the wall. Here, the residual of a point is calculated by fitting a plane to the wall; (2) Color feature: RGB color value is useful information to distinguish between templates and walls. The present invention introduces the visible-band template difference index (Visible-band DifferencevegetationIndex, VDVI) to reduce misclassification caused by templates, such as walls surrounded by templates. The VDVI calculation formula is as follows:

[0099]

[0100] (3) Reflection intensity: The reflection intensity recorded by the LiDAR system measures the energy reflected by the laser beam from the surface of an object. The light intensity depends on many parameters such as atmospheric absorption, laser power, angle of incidence, and distance between the sensor and the target. The properties of the target material also have a significant impact on the reflectivity of the intensity; different materials have different intensity values. Therefore, the intensity data provides important information for the classification of walls and buried panels. Here, in order to better compare the intensity between different walls and buried panels, the intensity value is first adjusted using an empirical model:

[0101]

[0102] Among them, I correct and I orig are the corrected intensity value and the original intensity value respectively; D is the distance from the lidar sensor to the wall sampling point; D ref is the reference distance; a is the incident angle. Therefore, by adjusting the intensity value, the steel plate can be distinguished from the wall point.

[0103] The classic DBSCAN is a global clustering method with a computational complexity of O(N 2 ), where N is the total number of study points. Therefore, it is difficult to process a large number of point clouds. For this reason, local DBSCAN is used here to perform clustering in a local area. Assuming there are k grid cells and the number of points in each grid cell is approximately equal, the computational complexity of local DBSCAN is O(N 2 / k). Therefore, local operations greatly improve clustering efficiency. More importantly, this can avoid clustering errors of different objects with similar features in large scenes, thereby enhancing the robustness of the fragment.

[0104] In our study, local DBSCAN is performed in a 4-dimensional clustering space: elevation (Z), fitting residual (R), VDVI (V), and intensity (I). XY coordinates are not used in the feature space, mainly due to the local clustering scheme. In practice, DBSCAN requires two parameters: Eps and MinPts. Eps is used to find neighbors around a point, that is, points whose distance to point P is less than Eps belong to the neighbors of P. MinPts is the minimum number of neighbors within the Eps radius. Based on this information, three types of points are defined ( Figure 8 ): core points, boundary points, and noise points. Specifically, a point with more than MinPts adjacent points in Eps is defined as a core point; a boundary point is defined as a point with fewer than MinPts adjacent points in Eps but near a core point; a point that is neither a core point nor a boundary point belongs to noise. According to experience, MinPts should be greater than the data dimension; therefore, we set MinPts = 5 in our study. At the same time, Eps can be obtained through the kd distance curve. Specifically, for each LiDAR point, its distance to the MinPts-th nearest neighbor is first calculated, and then the distances of all points are sorted from the maximum to the minimum, a distance curve is constructed, and finally the Eps value corresponding to the inflection point of the curve is taken as the optimal Eps value. The detailed steps for local DBSCAN to cluster points in the feature space are as follows:

[0105] (1) Normalize the eigenvalues ​​using min-max scaling.

[0106] (2) Calculate the optimal Eps based on the MinPts-th distance curve; here, MinPts = 5, point P i and P j The distance between ij )Use normalized features calculate:

[0107]

[0108] Among them, α, β, and γ are weights; in this study, we set α = 1, β = 1, and γ = 1. In other words, these four features are equally important for estimating distance.

[0109] (3) Randomly select a core point P from the unvisited points.

[0110] (4) If the core point P does not belong to any cluster, a new cluster is generated.

[0111] (5) Find all density-connected points of P and assign them to clusters of P.

[0112] (6) Repeat (3)-(5) until all points are visited. Obviously, points that do not belong to any cluster are regarded as noise.

[0113] It should be noted that outliers are prevalent in the original LiDAR point cloud, which is mainly caused by LiDAR system errors and multipath reflections. Usually, outliers have obvious deviations from their neighbors and do not belong to any cluster. This is just the definition of noise in DBSCAN. Therefore, DBSCAN has an additional function of outlier detection. Based on this function, the outlier removal process like the elevation histogram based algorithm can be avoided.

[0114] Calibration issues for LiDAR camera systems Fig. 9 The model of the calibration object and lidar camera system is shown in Fig. 9 (a) is shown. In this system, the laser radar and the camera are both mounted on a base, and their relative positions are fixed. The present invention proposes a novel calibration object combination, in which the auxiliary calibration object comes from the actual measurement environment. Fig. 9 The orange object in (a) is shown. The radar coordinate system is represented by o l -x l y l z l , the origin represents the position of the LiDAR, and the camera's coordinate system is defined as o c -x c y c z c , the origin represents the position of the camera. i (i=1,2) refers to the corner points on the auxiliary calibration object. In order to obtain stable calibration results, it is recommended to use at least four auxiliary calibration objects.

[0115] The external parameters of the LiDAR-camera system are the rotation matrix R and the transformation vector T, such as Fig. 9 (b). R, T represent the laser radar coordinate system o d -x d y d z d With the camera coordinate system o c -x c y c z c The proposed calibration method aims to estimate the rotation matrix R and the translation matrix T. Fig. 9As shown in (c), in this study, the camera and the lidar are integrated and their relative positions remain unchanged, so α, β and γ are very small, and Δx, Δy and Δz are also small. According to equation (9), from the camera coordinate system o c -x c y c z c To pixel coordinate system o c -uv does not rotate, that is, OP in the camera coordinate system c The angle θ between op and the x-axis in the pixel coordinate system remains unchanged (such as Fig. 9 (d)).

[0116]

[0117] Therefore, in the first stage of camera and lidar pose calibration, the center point of the embedded panel is defined as O(a, b, x, y), where a and b are the length and width of the embedded panel, x and y are the coordinates of the center point of the embedded panel, and w and h are the width and height of the image. ab is the shape factor of the embedded panel, x / y is the center point vector, and the point cloud descriptor is defined as D according to the quadrant n where the embedded panel is located. point (a / b,(-1) n x / y), the image descriptor is D image (w / h·(a / b),(-1) n x / y). The coordinate set of the center point of the embedded plate in the three-dimensional point cloud is S point , the center point coordinate set of the buried plate is S image , according to equation (10), the 3D point cloud is matched with the pixel interior point, and the initial R is solved based on the PnP method. init With T init .

[0118]

[0119] The second stage of camera and lidar pose calibration is shown in formula (11), S i ' mage You can use R init With T init To indicate that when S i ' mage With S image When the absolute difference is not greater than ε, S image With S point is a pair of matching points. Using formula (12) as the objective function, the RANSAC method is used to obtain point With S image Repeated sampling to select interior points I point with I image , calculate R and T.

[0120]

[0121] Example

[0122] (1) Data collection and processing

[0123] Set up the test scenario as Fig.10 As shown in the figure, there are three sizes of embedded plates, 15cm×15cm, 10cm×15cm, and 10cm×10cm. The embedded plates are attached to the wall and the working conditions are set as follows: (1) attached to the wall; (2) 1.5cm away from the wall; (3) 2.5cm away from the wall; (4) rotated at an angle of 45 degrees. The target is attached to a 900×400mm calibration plate, which is placed close to the wall.

[0124] The image and point cloud data of the wall are collected to measure the three-dimensional position information, side length and rotation angle of the embedded plate. In this experiment, there is no absolute coordinate of the embedded plate, so the measured embedded plate size is compared with the actual size to verify the accuracy of the proposed method. The measured results are shown in Table 1. When the embedded plate is flush with the wall, the measured value in the z direction is the measurement error. It can be seen that the maximum error is 4.175mm, which comes from the image data. The average absolute error is 1.846mm. The maximum angle error of the embedded plate is 2.441°, and the average absolute error is 1.071°. The reasons for the large error in the measurement of the side length of the embedded plate include: (1) the error in the image measurement result caused by the deviation of the target pasting position; (2) the relative position deviation of the target caused by the deformation of the calibration plate. The above factors will cause the image scaling factor calculated by the target to be distorted and the physical size deviation of the embedded plate measured by the image data to be measured.

[0125] In order to eliminate the systematic error caused by the target position and make improvements, repeated positioning measurements of the embedded plate were carried out. Three sets of data were taken from different perspectives (left side, front view, and right side), and a coordinate system was established with the target at the upper left position of the calibration plate. The three-dimensional coordinates of all embedded parts were converted to this coordinate system, and a buried plate position map was drawn to evaluate the error. Fig.11 (a) is the image captured by the camera. The target and embedded board are accurately detected based on Cornernet-Lite. The effect after one stage of correction is as follows Fig.11 (b) Fig.11 (c) is the result of the two-stage correction. It can be seen that the edge of the embedded plate remains straight after the two-stage correction.

[0126] To further quantitatively evaluate the measurement error, the three sets of data for the buried plate plane position are plotted in the same figure, e.g. Fig.12 As shown in (red represents the left view, green represents the front view, yellow represents the right view, and the distance from the center point to the boundary is 5mm). Fig.12As shown in (a), before algorithm optimization, a one-stage emmetropia correction is used, and the measurement error of different viewing angles is less than 1 cm in the horizontal direction (X) and less than 1 cm in the vertical direction (Y). Specifically, the maximum error is 9.97 mm and the average absolute error is 3.41 mm. Fig.12 As shown in (b), a two-stage emmetropia correction is adopted. After the algorithm is optimized, the measurement error of different viewing angles is less than 4mm in the horizontal direction (X) and less than 1mm in the vertical direction (Y). The maximum error is 3.91mm and the average absolute error is 1.62mm, which is an improvement of 52.5%.

[0127] (2) Camera and LiDAR fusion calibration results based on environmental perception

[0128] First, the collected point cloud data is preprocessed, including removing noise and outliers, to improve data quality. In this paper, the point cloud distribution of the wall and buried plate is relatively uniform, and the statistical filtering method is used to remove point cloud noise. Fig.13 (a) is the fifth-order distance curve of the point cloud. Based on this curve, the optimal Eps=0.038 is taken, and the DBSCAN clustering method is used to effectively distinguish the buried plate and the wall; Fig.13 As shown in (b), the embedded parts are detected based on the CornerNet-Lite method. The camera and lidar calibration is successfully realized based on the proposed fusion calibration model, and the point cloud projection to image matching effect is good.

[0129] Fig.13 (a) shows that the proposed laser camera fusion calibration model performs consistently in the laboratory and on-site, and the calibration results are ideal even in the case of more interference outdoors. Specifically, the maximum reprojection error in the laboratory is 2.55 pixels, with an average absolute error of 1.55 pixels; the on-site environment is complex, with a maximum reprojection error of 3.04 pixels, an average absolute error of 1.68 pixels, and a decrease in accuracy of 0.13 pixels. This error result shows that the camera and laser radar fusion calibration method based on environmental perception can be used for on-site calibration.

[0130] (3) Automatic acceptance results of embedded parts

[0131] This field test includes five walls, each scene from a different perspective ( Fig.14 ) to collect data. The multi-view data has two main uses. On the one hand, it is used for repeated positioning of the equipment to verify the accuracy of the equipment itself, and on the other hand, it is compared with the design value to output the acceptance report in real time. Wall 1 is a test comparison of different view angles (left, right, wide view) ( Fig.14 ), this scenario simulates the limited placement of the device and the limited shooting angle, and the accuracy of the proposed method under different viewing angles. Fig.15As shown in (a), the comparison results of the embedded panels from multiple perspectives show that except for the maximum error of the 3D position of embedded part PT1 of 4.09 mm, the repeat positioning errors of other embedded panels are all less than 4 mm. The average absolute error of the 3D position of all embedded panels is 1.89 mm, and the average absolute error of the rotation angle is 1.18°. Fig.15 f), the average absolute error between the three-dimensional position of the embedded plate and the design value is less than 7.50 mm, and the average absolute error of the rotation angle is 0.95°.

[0132] Wall No. 2 is a test comparison of different viewing angles (left and right). Fig.15 (b) shows the positioning error of the embedded plate at different viewing angles. The average absolute error of the three-dimensional position of all embedded plates is 0.94 mm, and the average absolute error of the rotation angle is 0.21°. Fig.15 (f)), the average absolute error between the three-dimensional position of the embedded plate and the design value is less than 7.39 mm, and the average absolute error of the rotation angle is 0.43°. Wall No. 3 is a test comparison of different viewing angles (left, middle). Fig.15 (c) shows the positioning error of the embedded plate at different viewing angles. The average absolute error of the three-dimensional position of all embedded plates is 1.47 mm, and the average absolute error of the rotation angle is 0.39°. Fig.15 (f)), the average absolute error between the three-dimensional position of the embedded plate and the design value is less than 6.68mm, and the average absolute error of the rotation angle is 0.48°. Wall No. 4 is a test comparison of different viewing angles (middle and right). Fig.15 (d) shows the positioning error of the embedded plate at different viewing angles. The average absolute error of the three-dimensional position of all embedded plates is 1.92 mm, and the average absolute error of the rotation angle is 0.56°. Fig.15 (f)), the average absolute error between the three-dimensional position of the embedded plate and the design value is less than 7.48mm, and the average absolute error of the rotation angle is 0.45°. Wall No. 5 is a test comparison of different viewing angles (pitch and elevation). Fig.15 (e) shows the positioning error of the embedded plate at different viewing angles. The average absolute error of the three-dimensional position of all embedded plates is 1.03 mm, and the average absolute error of the rotation angle is 0.27°. Fig.15 (f)), the average absolute error between the three-dimensional position of the embedded plate and the design value is less than 7.71 mm, and the average absolute error of the rotation angle is 0.41°.

[0133] In summary, the embedded panels of the five walls are not missing compared with the design, and the average absolute error of their three-dimensional position is less than 7.35mm, and the average absolute error of the rotation angle is less than 0.54°. The acceptance results meet the project requirements (1cm, 2°). On the other hand, under different viewing angles of the five scenes, the average absolute error of the repeated positioning of the three-dimensional position of the embedded panels is less than 1.45mm, and the average absolute error of the repeated positioning of the rotation angle is less than 0.52°, which fully demonstrates that the accuracy of the developed equipment and the degree of automation of the acceptance can meet engineering applications.

[0134] The examples described in the present invention are merely descriptions of the preferred implementation modes of the present invention, and are not intended to limit the concept and scope of the present invention. Without departing from the design concept of the present invention, various modifications and improvements made to the technical solutions of the present invention by engineers and technicians in this field should all fall within the protection scope of the present invention.

Claims

1. An intelligent acceptance method for the three-dimensional position of a component based on image and laser data fusion, characterized in that: The steps include: S1. Acquire the buried plate image through the industrial camera platform and automatically perform tilt correction on the image to obtain the corrected buried plate image; S2, perform two-stage detection on the corrected buried plate image based on scale unification; S3, integrates the adaptive calibration results of the image and point cloud data, aligns the point cloud depth information corresponding to the corner points in the image, outputs the comparison results between the measured value and the designed value, and completes the automated analysis from data acquisition to embedded plate acceptance.

2. The intelligent acceptance method for component three-dimensional position based on image and laser data fusion according to claim 1, characterized in that: In step 1, the industrial camera platform includes a Hikvision MV-CE120-10UC industrial camera, a Mid-70 solid-state area array laser radar, an Intel embedded micro industrial computer NUC for real-time detection, and a portable VT-88 tripod gimbal.

3. The intelligent acceptance method for component three-dimensional position based on image and laser data fusion according to claim 1, characterized in that: Step 1 is as follows: Design shooting mode. In view of the uneven direction of the wall, divide it into orthogonal walls along the X or Y direction and inclined walls with an angle with the X or Y direction. According to the direction of the wall, the local coordinate system of the detected wall is transferred to the nuclear power plant reference coordinate system, and then the data is compared. The pinhole model is used as the imaging model of the camera; Use homography to correct the image to orthophoto, express the initial image pixel coordinates as (x, y), and express the image pixel coordinates under the orthographic perspective as (x', y'). The homography transformation relationship is: Write equation (1) in the form of matrix AX=0: According to equation (2), when solving H, we add the constraint h 33 = 1. Although the homography matrix H has 9 unknowns, it has only 8 degrees of freedom, so there are still h 11 ~h 32 A total of 8 unknowns due to a set of matching points (x i ,y i )match(x i ',y i ') to obtain two sets of equations, then n = 4 sets of non-collinear matching points are needed to find the unique solution of H; A two-stage image automatic correction method based on known markers is adopted. In the first stage, a deep learning method is used to detect the target and determine the center position of the target. In the second stage, sub-pixel positioning is performed based on the designed target.

4. The intelligent acceptance method for component three-dimensional position based on image and laser data fusion according to claim 3 is characterized in that: The first stage uses deep learning methods to detect targets and determine the center position of the targets. Specifically, the Cornernet-Lite network is used to detect four targets based on key points, and the upper left corner point (x ltop ,y ltop ), the lower right corner (x rdown ,y rdown ), calculate the center point (x i ,y i ), obtain the homography transformation matrix H1 according to equation (2); 。 5. The intelligent acceptance method for component three-dimensional position based on image and laser data fusion according to claim 3 is characterized in that: The second stage is to perform sub-pixel positioning based on the designed target. Specifically, according to the color features of the three-channel (R, G, B) image, the G channel is extracted as a grayscale image, the edge features of the grayscale image are extracted, all edge contours are sorted according to the size of the connected domain area, the required contours are obtained, and the centroid coordinates of the target are fitted, as shown in equation (4): Obtain the homography transformation matrix H2 according to equation (2) and substitute it into equation (5) to realize the two-stage automatic view restoration of the tilted view; 。 6. The intelligent acceptance method for component three-dimensional position based on image and laser data fusion according to claim 1, characterized in that: Step 2 is as follows: first, the Cornernet-Lite network based on efficient key point detection is used to segment the target into monomers in real time, and then the monomer target image is input to automatically detect the corner points of the target using the Superpoint network, and the buried plate image is scaled so that the scaled buried plate image is close to the input size of the corner point detection model.

7. The intelligent acceptance method for component three-dimensional position based on image and laser data fusion according to claim 1, characterized in that: Step 3 specifically includes the following steps: Step 3.1: segment the point cloud using fitting residuals, color features and reflection intensity; where: Fitting residuals: Any object has a non-zero height relative to the plane it is located on, so the fitting residual of a point corresponds to the relative height to the wall; Color features: RGB color values ​​are useful information for distinguishing templates from walls. The visible light band template difference index VDVI is introduced to reduce misclassification caused by templates. The VDVI calculation formula is as follows: Reflection intensity: The reflection intensity recorded by the LiDAR system measures the energy of the laser beam reflected from the surface of an object. To compare the intensity between different walls and buried panels, an empirical model is used to adjust the intensity value: Among them, I correct and I orig are the corrected intensity value and the original intensity value respectively; D is the distance from the lidar sensor to the wall sampling point; D ref is the reference distance; a is the incident angle, and the steel plate is distinguished from the wall point by adjusting the intensity value; Step 3.2, use the local DBSCAN clustering method to cluster the points in the feature space. DBSCAN requires two parameters: Eps and MinPts; Eps is used to find neighbors around the point, that is, the point whose distance to point P is less than Eps belongs to the neighbor of P; MinPts is the minimum number of neighbors within the Eps radius; based on this information, three types of points are defined: core points, boundary points, and noise points; points with more than MinPts neighboring points in Eps are defined as core points; boundary points are defined as points with fewer than MinPts neighboring points in Eps but near the core point; a point that is neither a core point nor a boundary point belongs to noise; MinPts is greater than the data dimension, set MinPts = 5, and at the same time, Eps can be obtained through the kd distance curve. For each LiDAR point, first calculate its distance to the MinPts-th nearest neighbor, then sort the distances of all points from the maximum to the minimum, construct a distance curve, and finally take the Eps value corresponding to the inflection point of the curve as the optimal Eps value; Step 3.3: Calibrate the mechanical energy of the LiDAR camera system, adjust the external parameters of the LiDAR-camera system, and calibrate the camera and LiDAR poses in two stages.

8. The intelligent acceptance method for component three-dimensional position based on image and laser data fusion according to claim 7, characterized in that: Step 3.2 specifically includes the following steps: Step 3.2.1, normalize the eigenvalues ​​using min-max scaling; Step 3.2.2, calculate the optimal Eps according to the MinPts-th distance curve; here, MinPts = 5, point P i and P j The distance between ij )Use normalized features calculate: Among them, α, β, and γ are weights; α = 1, β = 1, and γ = 1, indicating that these four features are equally important for estimating distance; Step 3.2.3, randomly select a core point P from the unvisited points; Step 3.2.4: If the core point P does not belong to any cluster, generate a new cluster; Step 3.2.5, find all density-connected points of P and assign them to clusters of P; Step 3.2.6: Repeat steps 3.2.3 to 3.2.5 until all points are visited. Points that do not belong to any cluster are treated as noise.

9. The intelligent acceptance method for component three-dimensional position based on image and laser data fusion according to claim 7, characterized in that: In step 3.3, the mechanical energy of the LiDAR camera system is calibrated and the external parameters of the LiDAR-camera system are adjusted as follows: the model of the calibration object and the LiDAR camera system is constructed, and the coordinate system of the LiDAR is represented as o l -x l y l z l , the origin represents the position of the LiDAR, and the camera's coordinate system is defined as o c -x c y c z c , the origin represents the position of the camera; A i (i=1,2) refers to the corner points on the auxiliary calibration object, and at least four auxiliary calibration objects are used; the external parameters of the LiDAR-camera system are the rotation matrix R and the transformation vector T; R and T represent the LiDAR coordinate system o d -x d y d z d With the camera coordinate system o c -x c y c z c The rigid body transformation between them, the rotation angles are α, β and γ, and the translations are Δx, Δy and Δz. The calibration method aims to estimate the rotation matrix R and the translation matrix T. The camera and the lidar are integrated together and their relative positions remain unchanged, so α, β and γ are very small, and Δx, Δy and Δz are also small. According to equation (9), from the camera coordinate system o c -x c y c z c To pixel coordinate system o c -uv does not rotate, OP in the camera coordinate system c The angle θ between op and the x-axis in the pixel coordinate system remains unchanged; 。 10. The intelligent acceptance method for component three-dimensional position based on image and laser data fusion according to claim 7, characterized in that: In step 3.3, the camera and lidar poses are calibrated in two stages: In the first stage of the camera and lidar pose calibration, the center point O(a, b, x, y) of the embedded plate is defined, where a and b are the length and width of the embedded plate, x and y are the coordinates of the center point of the embedded plate, w and h are the width and height of the image, a / b is the shape factor of the embedded plate, x / y is the center point vector, and the point cloud descriptor D is defined according to the quadrant n where the embedded plate is located. point (a / b,(-1) n x / y), the image descriptor is D image (w / h·(a / b),(-1) n x / y); the coordinate set of the center point of the embedded plate in the three-dimensional point cloud is S point , the center point coordinate set of the buried plate is S image , according to equation (10), the 3D point cloud is matched with the pixel interior point, and the initial R is solved based on the PnP method. init With T init ; The second stage of camera and lidar pose calibration is shown in formula (11), S i ' mage Using R init With T init To indicate that when S i ' mage With S image When the absolute difference is not greater than ε, S image With S point is a pair of matching points. Formula (12) is used as the objective function and the RANSAC method is used to obtain the point With S image Repeated sampling to select interior points I point with I image , calculate R and T; 。

Images