Multi-view panoramic point cloud splicing method

Abstract

The invention discloses a multi-view panoramic point cloud splicing method, which comprises the steps of performing joint calibration on a binocular camera and a laser radar through a calibration algorithm, and aligning a coordinate system; a binocular camera is used for collecting a two-dimensional image and carrying out distortion correction, a depth map is generated based on parallax calculation, and three-dimensional point cloud reconstruction is carried out in combination with a triangulation principle; performing deep learning target detection on the two-dimensional image, and screening a line rod assembly area through non-maximum suppression; multi-view point cloud data are collected through a laser radar, point cloud segmentation processing is carried out, and a telegraph pole assembly point cloud subset is reserved; matching the two-dimensional pixel area of the telegraph pole assembly with a laser radar point cloud projection result, and screening point cloud data belonging to the telegraph pole assembly; and carrying out alignment and fusion on the multi-view point clouds through initial registration and fine registration by adopting an improved point cloud splicing algorithm to generate a panoramic point cloud image. According to the method, the point cloud splicing processing time is shortened, and the robustness, precision and integrity of point cloud splicing are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-view point cloud stitching and object reconstruction in power scenarios, and particularly to a multi-view panoramic point cloud stitching method. Background Art

[0002] With the continuous increase and enhanced complexity of power facilities, the demand for three-dimensional object measurement in power scenarios is becoming increasingly urgent. In the work of real-time monitoring, maintenance, and fault diagnosis of power equipment, it is necessary to obtain detailed morphological information of objects through high-precision three-dimensional point cloud data. However, existing point cloud measurement technologies are often affected by various factors in practical applications, such as noise of scanning devices, lighting environment, and field of view limitations, resulting in the inability to comprehensively and accurately obtain the three-dimensional point cloud information of the object to be measured. For example, in the same view, the device usually can only capture partial information of the object to be measured and cannot fully present the complex structure of the object. Therefore, it is required to scan the same object from multiple views to obtain more data from different angles in order to reconstruct a complete three-dimensional object model.

[0003] Currently, in the field of point cloud stitching, the most widely used algorithm is the Iterative Closest Point (ICP) algorithm. This algorithm selects corresponding points in different point clouds and repeatedly calculates and optimizes the rigid body transformation to minimize the distance between point clouds and achieve point cloud registration. However, the application of the ICP algorithm in complex scenarios still has certain limitations. For example:

[0004] The calculation time is relatively long. Especially when dealing with a large amount of point cloud data, the time-consuming of point cloud stitching is too long and the efficiency is low.

[0005] The ICP algorithm is relatively sensitive to noise and outliers and is easily interfered, resulting in the stitching effect.

[0006] Due to the limitations of the algorithm, there are certain accuracy errors in the point cloud data after ICP stitching, especially between the plane projection points and the real points, with a large deviation.

[0007] Therefore, there is still much room for improvement in the speed, accuracy, and robustness of point cloud stitching in the prior art.

[0008] CN116310132A discloses a point cloud stitching method based on an improved ICP algorithm, which is mainly used to improve the speed and accuracy in the process of point cloud stitching and reduce the calculation time. This method optimizes the point cloud registration step in the ICP algorithm, introduces some new constraint conditions and acceleration techniques, and attempts to complete the stitching of multi-view point cloud data in a shorter time. Although its stitching speed has been improved, this solution is still affected by noise and outliers, and the stitching result still has low accuracy in complex scenarios. Especially in a large-scale complex power scenario, it is difficult for the point cloud data to meet the requirements of high accuracy and high robustness.

[0009] CN116310250A discloses a point cloud stitching method based on a three-dimensional sensor, which realizes more accurate three-dimensional point cloud stitching through multi-sensor data fusion. This method uses the data fusion of lidar and camera, and through feature matching and multiple point cloud optimization registrations, to improve the stitching accuracy and reduce data redundancy. However, the sparse problem of point cloud data cannot be completely solved during the stitching process, especially in areas with more details, and there will be large errors after stitching, resulting in incomplete or inaccurate point cloud data. Summary of the Invention

[0010] The purpose of this part is to outline some aspects of the embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this part, as well as in the abstract and title of the present application, to avoid obscuring the purpose of this part, the abstract, and the title. However, such simplifications or omissions cannot be used to limit the scope of the present invention.

[0011] In view of the above existing problems, the present invention is proposed.

[0012] Therefore, the technical problems solved by the present invention are: the problems of low point cloud accuracy, large error, and large amount of stitched point cloud data in the prior art.

[0013] To solve the above technical problems, the present invention provides the following technical solutions: Jointly calibrate the binocular camera and the lidar through a calibration algorithm to align the coordinate systems;

[0014] Use the binocular camera to collect two-dimensional images and perform distortion correction, generate a depth map based on disparity calculation, and perform three-dimensional point cloud reconstruction in combination with the principle of triangulation;

[0015] Perform deep learning object detection on the two-dimensional images, and screen out the pole component areas through non-maximum suppression;

[0016] Collect multi-view point cloud data through the lidar and perform point cloud segmentation processing to retain the pole component point cloud subset;

[0017] Match the two-dimensional pixel region of the pole component with the lidar point cloud projection result, and filter the point cloud data belonging to the pole component;

[0018] Adopt an improved point cloud stitching algorithm to align and fuse multi-view point clouds through initial registration and fine registration to generate a panoramic point cloud image.

[0019] As a preferred scheme of the multi-view panoramic point cloud stitching method of the present invention, it includes:

[0020] Construct a target with feature tags, and obtain the two-dimensional image information of the target through the binocular camera and the three-dimensional point cloud data of the target through the lidar;

[0021] Use feature point matching and singular value decomposition to calculate the camera internal parameter matrix and external parameter matrix, and align the coordinate systems;

[0022] Among them, the target with feature tags is a checkerboard target with reflective tags.

[0023] As a preferred scheme of the multi-view panoramic point cloud stitching method of the present invention, in the checkerboard image, extract feature points through Harris corner detection, and use singular value decomposition to calculate the camera internal parameter matrix and external parameter matrix to align the coordinate systems of the two sensors, including:

[0024]

[0025] Among them, s is the scale factor, representing the conversion scale from the world coordinate to the image coordinate, u and v represent the pixel coordinates in the image, that is, the point coordinates on the image plane, K is the internal parameter matrix of the camera, including focal length and principal point coordinate information, used for projecting from the three-dimensional world coordinate to the image coordinate, R is the rotation matrix, describing the rotation relationship between the world coordinate system and the camera coordinate system, T is the translation vector, describing the translation relationship between the world coordinate system and the camera coordinate system, is the three-dimensional point coordinate in the world coordinate system, and 1 is the constant value used to represent a three-dimensional point in homogeneous coordinates.

[0026] As a preferred scheme of the multi-view panoramic point cloud stitching method of the present invention, the parallax calculation adopts a block matching algorithm to generate a parallax map, and converts the pixel coordinates into three-dimensional point coordinates through the reprojection matrix Q. The formula is:

[0027]

[0028] Among them, x and y are the pixel coordinates in the two-dimensional image, and d is the parallax value, representing the pixel difference of the same object point in the left image and the right image, P is the point cloud coordinates after 3D reconstruction, and Q is the reprojection matrix. Based on the internal and external parameters of the camera, pixel coordinates are converted into point coordinates in 3D space.

[0029] As a preferred solution of the multi-view panoramic point cloud stitching method described in the present invention, the deep learning object detection uses a convolutional neural network, including feature extraction, candidate box generation, and non-maximum suppression processing, and outputs the bounding box and confidence of the pole component. Among them:

[0030] Use convolutional layers to extract features, use the region proposal network to generate candidate boxes, remove candidate boxes with high overlap, and retain the optimal detection results.

[0031] As a preferred solution of the multi-view panoramic point cloud stitching method described in the present invention, the point cloud segmentation uses a density clustering algorithm based on DBSCAN. Set the neighborhood radius and the minimum number of points MinPts to classify the point cloud, separate the pole component from the background noise, and combine the separated pole components into a point cloud subset of the pole component.

[0032] As a preferred solution of the multi-view panoramic point cloud stitching method described in the present invention, when matching the two-dimensional pixel region of the pole component with the lidar point cloud projection result, the two-dimensional image region of the pole component needs to be back-projected into 3D space through the camera internal parameter matrix, and then matched with the point cloud data collected by the lidar.

[0033] As a preferred solution of the multi-view panoramic point cloud stitching method described in the present invention, the fine registration uses an improved iterative closest point algorithm. By introducing a bidirectional distance constraint, the registration error is optimized. The formula is:

[0034]

[0035] Among them, E(R,T) is the optimization objective function of point cloud registration, representing the registration error between the rotation matrix R and the translation vector T. The fine alignment of the point cloud is achieved by minimizing this function. w i is the weight factor of point cloud density, representing the importance of each point in the optimization, usually calculated according to the local density of the point cloud. p i is the point coordinate in the target point cloud, q i is the point coordinate in the source point cloud, ∥·∥ 2 represents the square of the Euclidean distance, used to measure the distance between points.

[0036] The beneficial effects of the present invention:

[0037] By adopting the object detection technology based on deep learning, the target area is accurately screened, the amount of point cloud data for stitching is reduced, and thus the stitching speed is improved;

[0038] By performing fine matching and projection optimization on binocular camera images and lidar point cloud data, the problems of low point cloud accuracy and large errors in the prior art are overcome. Especially in complex power scenarios, the accuracy and integrity of the point cloud after stitching can be effectively improved;

[0039] Through filtering and feature extraction techniques, the influence of noise is effectively reduced, making the stitching process more robust to environmental changes and noise. Brief Description of the Drawings

[0040] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings. Among them:

[0041] Figure 1 It is a schematic flowchart of the multi-view panoramic point cloud stitching method shown in the present invention;

[0042] Figure 2 It is a schematic flowchart of the calibration algorithm sub-process shown in the present invention;

[0043] Figure 3 It is a schematic flowchart of the binocular ranging algorithm sub-process shown in the present invention;

[0044] Figure 4 It is a schematic flowchart of the two-dimensional image recognition algorithm sub-process shown in the present invention;

[0045] Figure 5 It is a schematic flowchart of the point cloud processing sub-process shown in the present invention;

[0046] Figure 6 It is a schematic flowchart of the improved point cloud stitching algorithm sub-process shown in the present invention. Detailed Embodiments

[0047] To make the above objects, features, and advantages of the present invention more obvious and understandable, the following will give a detailed description of the specific embodiments of the present invention with reference to the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all embodiments.

[0048] Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the scope of protection of the present invention.

[0049] In the following description, many specific details are set forth to provide a thorough understanding of the present invention. However, the present invention may be practiced in other ways different from those described herein. Those skilled in the art can make similar generalizations without departing from the spirit of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed below.

[0050] According to an embodiment of the present invention, in combination with Figure 1 the flowchart shown, a multi-view panoramic point cloud stitching method specifically includes the following steps:

[0051] S1. Jointly calibrate the binocular camera and the lidar through a calibration algorithm to align the coordinate systems;

[0052] S2. Use the binocular camera to collect two-dimensional images and perform distortion correction, generate a depth map based on disparity calculation, and perform three-dimensional point cloud reconstruction in combination with the principle of triangulation;

[0053] S3. Perform deep learning object detection on the two-dimensional images, and screen out the pole component areas through non-maximum suppression;

[0054] S4. Collect multi-view point cloud data through the lidar and perform point cloud segmentation processing to retain the pole component point cloud subset;

[0055] S5. Match the two-dimensional pixel area of the pole component with the lidar point cloud projection result, and screen out the point cloud data belonging to the pole component;

[0056] S6. Adopt an improved point cloud stitching algorithm to align and fuse the multi-view point clouds through initial registration and fine registration to generate a panoramic point cloud image.

[0057] Preferably, the present invention processes the data collected by the binocular camera to obtain high-precision two-dimensional images, and then processes the point cloud information collected by the lidar to obtain data with good environmental adaptability and high reliability. After the point cloud and the two-dimensional image are matched with each other, a multi-angle panoramic point cloud image is obtained through the point cloud stitching algorithm, which is not restricted in the field of view and obtains a point cloud image with high precision, integrity, and good robustness.

[0058] Next, in combination with Figures 2 to 6 the flowchart shown and some preferred or optional examples of the present invention, the implementation process and / or effects of certain examples of the present invention will be described more specifically.

[0059] It should be noted that a multi-view panoramic point cloud stitching method provided by an embodiment of the present invention processes the data collected by the binocular camera and the lidar, and configures corresponding point cloud image mapping and point cloud stitching vision processing algorithms (including point cloud processing, point cloud image mutual mapping, and point cloud stitching), so as to construct a multi-view panoramic point cloud for the pole components in the power scene; wherein:

[0060] The binocular camera synchronously acquires images through two cameras on the left and right, obtains the pixel difference of the corresponding points imaged on the views of the two cameras, calculates the depth data using the principle of triangulation, and then forms a three-dimensional point cloud to achieve the reconstruction of the object;

[0061] The lidar uses non-contact laser ranging technology. The transmitter emits a laser beam. After the laser beam hits an object, it is diffusely reflected and returns to the receiver. The radar module calculates to form a point cloud.

[0062]

Joint Calibration

[0063] Construct a target, attach labels to the four corners of the target, and obtain the target information collected by the binocular camera and the lidar. Among them, the binocular camera collects the image information of the target, and the lidar collects the geometric information (point cloud data) of the target, and records the accurate coordinates of the four corner points on the target in space in the form of a point cloud;

[0064] Match the features of the image collected by the binocular camera with the target, and calculate the internal parameters of the camera and the coordinates of the labels on the target in the pixel coordinate system;

[0065] The target should at least contain multiple detectable feature points (such as checkerboard corner points, circular points), and image pairs (left / right images) captured by the binocular camera;

[0066] In an alternative embodiment, the image is converted into a grayscale image (if it is a color image) and denoised through image preprocessing, and a corner detection algorithm is used to extract the feature points in the target image to obtain the initial positions (u, v) of the target feature points in the pixel coordinate system;

[0067] For a checkerboard target, since the order and structure of the feature points are known, the matching process can be directly completed through indexing. Using the known physical coordinates (world coordinates) of the target and the corresponding pixel coordinates, calculate the internal parameters of the camera, and construct a mapping relationship using the world coordinates (X, Y, Z) and the pixel coordinates (u, v):

[0068]

[0069] Among them, s is the scale factor, representing the conversion scale from world coordinates to image coordinates, u and v represent the pixel coordinates in the image, that is, the point coordinates on the image plane, K is the internal parameter matrix of the camera, including focal length and principal point coordinate information, used for the projection from three-dimensional world coordinates to image coordinates, R is the rotation matrix, describing the rotation relationship between the world coordinate system and the camera coordinate system, T is the translation vector, describing the translation relationship between the world coordinate system and the camera coordinate system, is the three-dimensional point coordinate in the world coordinate system, and 1 is the constant value used to represent a three-dimensional point in homogeneous coordinates;

[0070] The internal parameters of the camera describe the mapping from the camera coordinate system to the pixel coordinate system. The form of the internal parameter matrix K is:

[0071]

[0072] where f x and f y are the camera focal lengths (in pixels), which are related to the physical focal length of the sensor and the pixel size. c x and c y are the principal point coordinates, representing the position of the optical center in the pixel coordinate system;

[0073] Exemplarily, the point (u, v) in the pixel coordinate system is calculated by the following formula:

[0074]

[0075] where (X c , Y c , Z c ) is the position of the calibration target point in the camera coordinate system, and K is the internal parameter matrix;

[0076] Through the above formula, the pixel coordinates (u, v) of the calibration target point on the image are obtained;

[0077] In an alternative embodiment, the rotation and translation matrix between the pixel coordinate system and the camera coordinate system is calculated by the SVD decomposition method. Taking the lower right corner point of the target as the reference point, the result with the minimum reprojection error after rotation and translation of this point is the external parameter; where:

[0078] The 3D world coordinates Pworld of the calibration target point and the pixel coordinates Ppixel are corresponded. Exemplarily, the relationship between the 3D world coordinates and the pixel coordinates is expressed by the following formula:

[0079]

[0080] Construct a linear equation. For each corresponding point (X i , Y i , Z i ) → (x i , y i ), construct a linear equation:

[0081] x i = r 11 X i + r 12 Y i + r 13 Z i + t x

[0082] y i = r 21 X i + r 22 Y i + r 23 Z i + t y

[0083] z i = r 31 X i + r 32 Y i + r 33 Z i + t z

[0084] Combine the equations of all points to form a matrix form:

[0085] A·[r 11 , r 12 , …, t z T = b

[0086] Use singular value decomposition to decompose the matrix and find the solution that minimizes the error:

[0087] [R|T] = arg min‖A·x - b‖

[0088] Use the point (Xr, Yr, Zr) at the lower right corner of the target to calculate its reprojection error:

[0089]

[0090] where (u r , v r ) are the actual pixel coordinates, and (u' r , v' r ) are the projected pixel coordinates calculated through the external parameters;

[0091] Minimize the reprojection error ∈ by optimizing R and T;

[0092] Furthermore, process the target information collected by the lidar. Obtain the center point coordinates of the label by the circle fitting method. By fitting these points, determine the center coordinates (c x , c y ) and the radius r of the circle; where:

[0093] Use a filtering method to downsample the point cloud, reduce noise and computational complexity. Add conditional constraints to the cropping area, extract potential circular target points, project the point cloud data onto a plane, simplify it to a 2D plane problem, and calculate the center and radius of the circle by least squares fitting; ​

[0094] Calculate the extrinsic parameters of the radar based on the coordinates of the four tags on the target in the world coordinate system;

[0095] The process of calibrating the extrinsic parameters of the lidar is to solve the rigid body transformation between two sets of coordinates of the 3D point cloud, including:

[0096] Select 4 tag points at the four corners of the target, and their positions in the world coordinate system are known, expressed as (X i , Y i , Z i ), where i = 1, 2, 3, 4;

[0097] The positions of the corresponding 4 points collected by the lidar in the lidar coordinate system are (x i , y i , z i ):

[0098]

[0099] Calculate the centroids of the two sets of points:

[0100]

[0101] Decentralize the two sets of points respectively:

[0102] Q world = P world - C world , Q lidar = P lidar - C lidar

[0103] Construct the covariance matrix:

[0104]

[0105] Obtain the rotation and translation matrix from the camera to the radar through the SVD decomposition method, complete the joint calibration algorithm of the camera and the radar, and use the SVD decomposition to solve the rotation matrix R and the translation vector T, where:

[0106] Perform singular value decomposition (SVD) on the covariance matrix H:

[0107] H = UΣV T

[0108] R = VU T

[0109] T = C world - R·C lida

[0110] Among them, U is the left singular matrix, V is the right singular matrix, and ∑V Tis a singular value diagonal matrix.

[0111]

Binocular ranging algorithm

[0112] It should be noted that the binocular ranging algorithm is an algorithm that determines the distance from the device to the utility pole through parallax, where:

[0113] Perform distortion correction on the images collected by the left and right cameras to make the corresponding scan lines parallel;

[0114] Select matching point pairs in the images collected by the left and right cameras, calculate the pixel difference for the densely matched point pairs, and construct a disparity map;

[0115] Based on the disparity map generated from the densely matched point pairs and the sparsely matched point pairs, use the triangle principle to calculate the depth and achieve three-dimensional reconstruction;

[0116] Exemplarily, obtain the internal and external parameters of the camera, correct the lens distortion to make the image regular, collect multiple groups of pictures using a calibration board (such as a checkerboard pattern), use the calibration function in OpenCV to calculate the internal parameters and distortion parameters of the camera, find the matching points for each pixel in the left and right images, and calculate the disparity, that is, the horizontal displacement between pixels. Use the feature point matching method or the block matching method to perform sparse or dense matching on the point pairs in the left and right images. Through the StereoBM or StereoSGBM method in opencv, generate a disparity map. According to the disparity map and the camera parameters, use the triangulation principle to calculate the depth of each pixel point:

[0117]

[0118] Among them, Z is the depth value, that is, the distance from the camera, f is the focal length of the camera, B is the baseline length of the binocular camera, that is, the distance between the optical axis centers of the two cameras, and d is the disparity value, that is, the difference in the abscissa of the matching points in the left and right images;

[0119] According to the depth value, convert the points in the image coordinate system to three-dimensional points in the world coordinate system. Define that the point is in the corrected image, and its normalized pixel coordinates are (x, y). Use the reprojection matrix Q to convert it to a three-dimensional point:

[0120]

[0121] The coordinates of the three-dimensional point are:

[0122]

[0123] Among them, W is the normalization factor of the homogeneous coordinates.

[0124]

Two-dimensional image recognition

[0125] It should be noted that the two-dimensional image recognition algorithm processes the two-dimensional image information collected by the binocular camera to achieve the recognition of the pole component, where:

[0126] Read the two-dimensional images captured by the left and right cameras as input;

[0127] Extract features from the left and right images respectively through the convolutional layer;

[0128] Exemplarily, through the convolutional operation, the information in the input image is converted into a set of high-dimensional feature representations, which are used to capture the key patterns or attributes in the image, such as edges, corners, colors, textures, the contours, shapes, and structures of the objects;

[0129] Stitch the left and right eye features to generate a feature map;

[0130] Input the feature map into the pooling layer and the fully connected layer for processing, and finally output the target recognition result of the pole component;

[0131] As an example, screen all the collected image data, and use the labeling tool to label the detection targets in each image for training;

[0132] The input image is the image to be detected of the pole component. Use a convolutional neural network (such as ResNet, VGG, EfficientNet) to extract features and generate a multi-channel feature map. Different convolutional layers extract different features, including multi-level semantic information such as edges, textures, shapes, and structures. Use a pooling layer (such as max pooling or average pooling) to downsample the feature map, reduce the spatial resolution of the feature map, retain important information, and improve the calculation efficiency, and output the extracted feature map;

[0133] Generate multiple candidate regions by sliding a fixed-size window in the image, map the information in the feature map to each candidate box, extract the features within the candidate box region, and use the region pooling or region alignment method to unify the features of each candidate box to the same size;

[0134] Classify the features of the candidate box region, use a fully connected network or a convolutional classification head to output the probability distribution containing all classes for each candidate box, output shape: [number of candidate boxes] × [number of classes][number of candidate boxes] × [number of classes], perform position regression on the features of the candidate box region, and output the bounding box correction value of each candidate box;

[0135] Use non-maximum suppression to remove redundant candidate boxes and retain the box with the highest confidence, that is, sort the candidate boxes according to the class prediction probability. If the intersection over union of two candidate boxes is greater than a certain threshold (such as 0.5), then only retain the box with the highest confidence;

[0136] Output the final detection bounding boxes for each target, including:

[0137] Target category: The type of target detected;

[0138] Confidence score: The probability that the target belongs to the predicted category;

[0139] Detection bounding box location: The coordinates of the bounding box;

[0140] Exemplarily, input an image of a pole assembly, such as a scene containing components like "crossarm", "insulator", "tower body", etc. The detected pole assembly categories, for example: "crossbar", "insulator", "tower body", "pole". The confidence of the crossbar is 95%, the confidence of the insulator is 89%, the confidence of the tower body is 90%, and the confidence of the pole is 93%.

[0141]

Point Cloud Processing

[0142] It should be noted that for point cloud processing, the Point Cloud Library is adopted to achieve point cloud related acquisition, filtering, segmentation, registration, retrieval, feature extraction, recognition, tracking, surface reconstruction, and visualization, where:

[0143] Read the point cloud data collected from the lidar sensor and accumulate multiple frames of point clouds;

[0144] Perform filtering processing on the point cloud to effectively remove noise points and outliers, making the point cloud smoother;

[0145] Exemplarily, perform outlier (noise point) filtering on the point cloud to remove abnormal or isolated points in the point cloud. For example, if using statistical filtering, calculate the average distance from points within the neighborhood of each point to that point. According to the global distribution statistics, points with an average distance exceeding the preset threshold range are regarded as outliers and removed;

[0146] In an alternative embodiment, perform segmentation processing on the point cloud to exclude invalid point cloud noise within the area range of the pole assembly to be recognized, and segment the pole assembly from the surrounding noise, including:

[0147] Use the point cloud density or spatial distribution characteristics for clustering, divide the point cloud into different clusters. The pole assembly is an independent dense cluster. Calculate the Euclidean distance between points, and group points with a distance less than the set threshold into the same class. Through multiple segmentation operations, continuously delete the useless point cloud. Finally, the output of the clustering is the pole point cloud;

[0148] In an alternative embodiment, extract the features of the target object and complete the point cloud map matching, including:

[0149] By extracting the geometric features of the target object, the spatial transformation relationship (rotation and displacement) between two sets of point clouds is found, thereby completing point cloud registration or map merging;

[0150] Key point detection selects a set of key points in the point cloud for subsequent feature description and matching. For example, the point cloud is divided into three-dimensional grids, and a representative point (such as the center point or average point) is selected in each grid as the key point;

[0151] Feature description is performed on the local neighborhood of each key point to generate a feature vector. For example, the geometric characteristics of the neighborhood of each point are calculated, including the normal vector and the relationship between points;

[0152] When performing global matching on the entire point cloud, global features need to be extracted. For example, the point feature histogram of the entire point cloud is calculated, or the global features of the point cloud are extracted by combining perspective information;

[0153] After feature extraction, the point cloud is registered. By estimating the rigid body transformation (rotation and translation) between two sets of point clouds, point cloud alignment is achieved, where:

[0154] The feature descriptors of the key points are used for feature matching to find the corresponding point pairs between the source point cloud and the target point cloud. The RANSAC algorithm is used to eliminate incorrect matches and estimate the initial transformation matrix. If the initial positions of the two sets of point clouds are significantly different, global features (such as VFH) are used for rough registration;

[0155] Then, the point cloud alignment result is further optimized through fine registration. For example, the closest point pairs are iteratively searched for between the two sets of point clouds to optimize the rigid body transformation matrix. The optimization goal is to minimize the Euclidean distance between the two sets of point clouds, or the point cloud is represented as a Gaussian distribution and registration is performed based on the distribution;

[0156] Exemplarily, after registration, multiple sets of point clouds are merged to generate a complete point cloud map. For example, the source point cloud is transformed to the target point cloud coordinate system using the rigid body transformation matrix and superimposed to generate a complete one, or the point clouds in the overlapping area are fused to avoid duplicate or redundant points;

[0157] The result of point cloud matching is the alignment transformation matrix of the point cloud and the aligned point cloud.

[0158]

Improved Point Cloud Stitching Algorithm

[0159] It should be noted that the point cloud stitching algorithm stitches point clouds from multiple perspectives to obtain a panoramic image containing pole components, where:

[0160] In an alternative embodiment, input point cloud data, and the corresponding pixel values in the image are calculated for the point cloud through the calibration relationship, including:

[0161] The point cloud is calibrated to calculate the corresponding pixel value in the image, and the three-dimensional point cloud data is obtained from the lidar in the format of (x, y, z, intensity), where (x, y, z) is the spatial coordinate and intensity is the reflection intensity. The high-resolution two-dimensional image (left and right) is obtained from the binocular camera in the format of color image or grayscale image.

[0162] Through the calibration of the camera and the lidar, the relationship between the lidar point cloud coordinate system and the camera pixel coordinate system is established to obtain the external parameter matrix and the internal parameter matrix;

[0163] Convert the laser radar point cloud coordinates (x, y, z) to the camera coordinate system:

[0164]

[0165] Then project the point in the camera coordinate system to the image plane to obtain the pixel coordinates:

[0166]

[0167] The projection result is a pixel value (u, v), which indicates the corresponding position of the point cloud in the image;

[0168] In an optional implementation manner, obtaining pixel values ​​according to the pole components identified by the image includes:

[0169] Grayscale the input two-dimensional image to enhance the recognizability of the pole assembly, for example, convert the color image into a grayscale image to reduce the computational complexity, use the Gaussian filter method to remove noise, smooth the image, and use the Sobel operator to extract edge features;

[0170] Locate the pole assembly area using an object detection algorithm;

[0171] Extract all pixel coordinates (u,v) in the identified pole component area;

[0172] In an optional implementation, comparing the pixel values ​​of the two-dimensional image with the pixel values ​​of the point cloud to determine the part of the wire rod component in the point cloud includes:

[0173] The projected pixel values ​​(u, v) of all points in the point cloud are compared with the pixel values ​​of the line-pole component in the image. If the projected pixel value (u, v) of a point in the point cloud falls within the line-pole component area of ​​the image, then the point belongs to the line-pole component.

[0174] According to the comparison results, the point cloud points belonging to the wire rod component are retained, and other point cloud points are eliminated to generate a point cloud subset containing only the wire rod component;

[0175] The multi-view point clouds containing the pole component targets are stitched together to obtain a panoramic point cloud map;

[0176] As an example, the splicing method is as follows:

[0177] Input point cloud data from different perspectives;

[0178] Coarsely align the multi - perspective point clouds through a point cloud splicing algorithm (such as feature matching or initial pose estimation);

[0179] Extract the local features of the point cloud (such as FPFH features), use RANSAC to find the matching point pairs between the point clouds, and estimate the initial rigid body transformation matrix;

[0180] Use external pose information, such as GPS, IMU or odometer data, to provide an initial pose for the point cloud;

[0181] Use a refinement algorithm to optimize the point cloud alignment effect. For example, use a probability distribution method to align the point cloud;

[0182] Fuse multiple frames of point clouds, eliminate the overlapping areas, and generate a complete panoramic point cloud map;

[0183] Filter or weighted - fuse the points in the overlapping areas to remove redundant points.

[0184] It should also be noted in this embodiment that the improved point cloud splicing algorithm, by introducing the fusion of binocular cameras and lidar and combining the precise splicing of multi - perspective point clouds, can not only accurately segment the pole component targets, but also generate a complete panoramic point cloud map, which is applicable to target extraction and map construction tasks in complex environments, such as external parameter calibration, projection calculation, target matching and fusion, ensuring high precision and high reliability of data processing.

[0185] The image pre - processing, feature extraction, and matching calculation methods in the aforementioned joint calibration can be carried out by means and methods in the existing technology, which will not be elaborated in this example.

[0186] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present invention, and they should all be covered within the scope of the claims of the present invention.

Claims

1. A multi-view panoramic cloud stitching method, characterized in that: include: The binocular camera and lidar are jointly calibrated through the calibration algorithm to align the coordinate system; The binocular camera is used to collect two-dimensional images and perform distortion correction, a depth map is generated based on parallax calculation, and three-dimensional point cloud reconstruction is performed in combination with the triangulation principle; Performing deep learning target detection on the two-dimensional image, and screening out the pole component area by non-maximum suppression; The laser radar is used to collect multi-view point cloud data and perform point cloud segmentation processing, and a point cloud subset of the wire rod component is retained; Matching the two-dimensional pixel area of ​​the wire rod assembly with the laser radar point cloud projection result, and filtering the point cloud data belonging to the wire rod assembly; An improved point cloud stitching algorithm is used to align and fuse multi-view point clouds through initial registration and fine registration to generate a panoramic point cloud image.

2. The multi-view panoramic cloud stitching method according to claim 1 is characterized in that: The joint calibration comprises: Constructing a target with a feature label, obtaining two-dimensional image information of the target through the binocular camera, and obtaining three-dimensional point cloud data of the target through the laser radar; Using feature point matching and singular value decomposition, calculate the camera intrinsic parameter matrix and extrinsic parameter matrix to align the coordinate system; Wherein, the target with the characteristic label is a checkerboard target with a reflective label.

3. The multi-view panoramic cloud stitching method according to claim 2 is characterized in that: In the checkerboard image, feature points are extracted through Harris corner detection, and the camera intrinsic parameter matrix and extrinsic parameter matrix are calculated using singular value decomposition to align the coordinate systems of the two sensors, including: Among them, s is the scale factor, which represents the conversion scale from world coordinates to image coordinates, u and v represent the pixel coordinates in the image, that is, the point coordinates on the image plane, K is the camera's intrinsic parameter matrix, which contains the focal length and principal point coordinate information and is used for projection from three-dimensional world coordinates to image coordinates, R is the rotation matrix, which describes the rotation relationship between the world coordinate system and the camera coordinate system, and T is the translation vector, which describes the translation relationship between the world coordinate system and the camera coordinate system. is the coordinate of a three-dimensional point in the world coordinate system, and 1 is a constant value used to represent a three-dimensional point in homogeneous coordinates.

4. The multi-view panoramic cloud stitching method according to claim 1, characterized in that: The disparity calculation uses a block matching algorithm to generate a disparity map, and converts pixel coordinates into three-dimensional point coordinates through a reprojection matrix Q. The formula is: Among them, x and y are the pixel coordinates in the two-dimensional image, and d is the disparity value, which represents the pixel difference of the same object point in the left image and the right image. is the point cloud coordinate after 3D reconstruction, Q is the reprojection matrix, and the pixel coordinates are converted into point coordinates in 3D space based on the intrinsic and extrinsic parameters of the camera.

5. The multi-view panoramic point cloud stitching method according to claim 1, characterized in that: The deep learning target detection adopts a convolutional neural network, including feature extraction, candidate box generation and non-maximum suppression processing, and outputs the bounding box and confidence of the pole component, where: Use the convolutional layer to extract features, use the region proposal network to generate candidate boxes, remove candidate boxes with high overlap, and retain the best detection results.

6. The multi-view panoramic point cloud stitching method according to claim 1, characterized in that: The point cloud segmentation adopts a density clustering algorithm based on DBSCAN, sets a neighborhood radius and a minimum number of points MinPts to classify the point cloud, separates the pole components from the background noise, and combines the separated pole components into a pole component point cloud subset.

7. The multi-view panoramic point cloud stitching method according to claim 1, characterized in that: When matching the two-dimensional pixel area of ​​the wire rod assembly with the laser radar point cloud projection result, the two-dimensional image area of ​​the wire rod assembly needs to be back-projected into three-dimensional space through the camera intrinsic parameter matrix, and then matched with the point cloud data collected by the laser radar.

8. The multi-view panoramic cloud stitching method according to claim 1, characterized in that: The fine registration adopts an improved iterative closest point algorithm and optimizes the registration error by introducing a bidirectional distance constraint. The formula is: Among them, E(R,T) is the optimization objective function of point cloud registration, which represents the registration error between the rotation matrix R and the translation vector T. The fine alignment of the point cloud is achieved by minimizing this function. i is the weight factor of the point cloud density, indicating the importance of each point in the optimization, usually calculated based on the local density of the point cloud, p i is the point coordinate in the target point cloud, q i is the point coordinate in the source point cloud, ∥·∥ 2 Represents the square of the Euclidean distance, which is used to measure the distance between points.

Images