A Multi-View Panoramic Cloud Stitching Method

Abstract

This invention discloses a multi-view panoramic point cloud stitching method, comprising: jointly calibrating a binocular camera and a LiDAR using a calibration algorithm to align the coordinate systems; acquiring two-dimensional images using the binocular camera and performing distortion correction, generating a depth map based on disparity calculation, and reconstructing a three-dimensional point cloud using triangulation principles; performing deep learning object detection on the two-dimensional images, and filtering out the pole component region through non-maximum suppression; acquiring multi-view point cloud data using the LiDAR and performing point cloud segmentation processing, retaining a subset of the pole component point cloud; matching the two-dimensional pixel region of the pole component with the LiDAR point cloud projection results, and filtering point cloud data belonging to the pole component; employing an improved point cloud stitching algorithm to align and fuse the multi-view point clouds through initial registration and fine registration to generate a panoramic point cloud image. This invention reduces point cloud stitching processing time and improves the robustness, accuracy, and completeness of point cloud stitching.

Description

Technical Field

[0001] This invention relates to the technical field of multi-view point cloud stitching and object reconstruction in power scenarios, and particularly to a multi-view panoramic point cloud stitching method. Background Technology

[0002] With the increasing number and complexity of power facilities, the demand for 3D object measurement in power scenarios is becoming increasingly urgent. Real-time monitoring, maintenance, and fault diagnosis of power equipment require high-precision 3D point cloud data to obtain detailed morphological information of objects. However, existing point cloud measurement technologies are often affected by various factors in practical applications, such as noise of the scanning equipment, lighting environment, and field of view limitations, which makes it impossible to obtain comprehensive and accurate 3D point cloud information of the object under test. For example, from the same viewpoint, the equipment can usually only capture part of the information of the object under test and cannot fully present the complex structure of the object. Therefore, it is necessary to scan the same object from multiple viewpoints to obtain more data from different angles in order to reconstruct a complete 3D object model.

[0003] Currently, the most widely used algorithm in point cloud stitching is the Iterative Closest Point (ICP) algorithm. This algorithm selects corresponding points in different point clouds and repeatedly calculates and optimizes rigid body transformations to minimize the distance between point clouds, thus achieving point cloud registration. However, the ICP algorithm still has certain limitations in the application of complex scenes, such as:

[0004] The computation time is long, especially when processing large amounts of point cloud data. Point cloud stitching takes too long and is inefficient.

[0005] The ICP algorithm is sensitive to noise and outliers and is easily interfered with, which can affect the stitching quality.

[0006] Due to the limitations of the algorithm, the point cloud data after ICP stitching has certain accuracy errors, especially between the planar projected points and the real points, where the deviation is relatively large.

[0007] Therefore, there is still much room for improvement in the speed, accuracy, and robustness of existing point cloud stitching technologies.

[0008] CN116310132A discloses a point cloud stitching method based on an improved ICP algorithm, mainly used to improve the speed and accuracy of the point cloud stitching process and reduce computation time. This method optimizes the point cloud registration step in the ICP algorithm, introduces some new constraints and acceleration techniques, and attempts to complete the stitching of multi-view point cloud data in a shorter time. Although it improves the stitching speed, the scheme is still affected by noise and outliers, and the accuracy of the stitching results is still low in complex scenarios, especially in large-scale complex power scenarios, where point cloud data is difficult to achieve the requirements of high accuracy and high robustness.

[0009] CN116310250A discloses a point cloud stitching method based on 3D sensors. This method achieves more accurate 3D point cloud stitching through multi-sensor data fusion. It utilizes data fusion from LiDAR and cameras, and improves stitching accuracy and reduces data redundancy through feature matching and multiple point cloud optimization registrations. However, it cannot completely solve the problem of sparsity in point cloud data during the stitching process, especially in areas with many details. Large errors will occur after stitching, resulting in incomplete or inaccurate point cloud data. Summary of the Invention

[0010] The purpose of this section is to outline some aspects of the embodiments of the present invention and to briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section, as well as in the specification abstract and the title of the invention, to avoid obscuring the purpose of this section, the specification abstract, and the title of the invention. Such simplifications or omissions shall not be used to limit the scope of the invention.

[0011] In view of the aforementioned existing problems, the present invention is proposed.

[0012] Therefore, the technical problem solved by the present invention is the problem of low point cloud accuracy, large error, and large amount of spliced ​​point cloud data in the prior art.

[0013] To solve the above-mentioned technical problems, the present invention provides the following technical solution: a calibration algorithm is used to jointly calibrate the binocular camera and the lidar to align the coordinate system;

[0014] The binocular camera is used to acquire two-dimensional images and perform distortion correction. A depth map is generated based on parallax calculation, and three-dimensional point cloud reconstruction is performed by combining the principle of triangulation.

[0015] Deep learning target detection is performed on the two-dimensional image, and the pole component region is filtered out by non-maximum suppression;

[0016] The laser radar is used to collect multi-view point cloud data and perform point cloud segmentation processing to retain a subset of the point cloud of the pole assembly.

[0017] The two-dimensional pixel region of the pole assembly is matched with the point cloud projection result of the lidar, and the point cloud data belonging to the pole assembly is filtered.

[0018] An improved point cloud stitching algorithm is used to align and fuse multi-view point clouds through initial registration and fine registration to generate a panoramic point cloud image.

[0019] As a preferred embodiment of the multi-view panoramic cloud stitching method described in this invention, it includes:

[0020] A target with feature labels is constructed, and two-dimensional image information of the target is acquired by the binocular camera, and three-dimensional point cloud data of the target is acquired by the lidar.

[0021] Using feature point matching and singular value decomposition, the camera intrinsic and extrinsic parameter matrices are calculated, and the coordinate system is aligned.

[0022] The target with the feature label is a checkerboard target with a reflective label.

[0023] As a preferred embodiment of the multi-view panoramic cloud stitching method described in this invention, feature points are extracted from the checkerboard image using Harris corner detection, and singular value decomposition is used to calculate the camera intrinsic and extrinsic parameter matrices, thereby achieving coordinate system alignment between the two sensors, including:

[0024]

[0025] Where s is the scale factor, representing the transformation scale from world coordinates to image coordinates; u and v represent the pixel coordinates in the image, i.e., the point coordinates on the image plane; K is the camera's intrinsic parameter matrix, containing focal length and principal point coordinate information, used for projection from 3D world coordinates to image coordinates; R is the rotation matrix, describing the rotation relationship between the world coordinate system and the camera coordinate system; and T is the translation vector, describing the translation relationship between the world coordinate system and the camera coordinate system. 1 represents the coordinates of a three-dimensional point in the world coordinate system, and 1 is a constant value used to represent a three-dimensional point in homogeneous coordinates.

[0026] As a preferred embodiment of the multi-view panoramic cloud stitching method described in this invention, the disparity calculation employs a block matching algorithm to generate a disparity map, and converts pixel coordinates into three-dimensional point coordinates using a reprojection matrix Q. The formula is as follows:

[0027]

[0028] Where x and y are the pixel coordinates in the two-dimensional image, and d is the disparity value, representing the pixel difference of the same object point in the left and right images. Let Q be the coordinates of the point cloud after 3D reconstruction, and let Q be the reprojection matrix, which converts pixel coordinates into point coordinates in 3D space based on the camera's intrinsic and extrinsic parameters.

[0029] As a preferred embodiment of the multi-view panoramic cloud stitching method described in this invention, the deep learning object detection employs a convolutional neural network, including feature extraction, candidate box generation, and non-maximum suppression processing, outputting the bounding box and confidence score of the line pole component, wherein:

[0030] Features are extracted using convolutional layers, candidate boxes are generated using a region proposal network, candidate boxes with high overlap are removed, and the best detection results are retained.

[0031] As a preferred embodiment of the multi-view panoramic point cloud stitching method described in this invention, the point cloud segmentation adopts a density clustering algorithm based on DBSCAN, sets the neighborhood radius and minimum number of points MinPts to classify the point cloud, separates the line pole components from the background noise, and combines the separated line pole components into a line pole component point cloud subset.

[0032] As a preferred embodiment of the multi-view panoramic point cloud stitching method described in this invention, when matching the two-dimensional pixel region of the line pole component with the point cloud projection result of the lidar, the two-dimensional image region of the line pole component needs to be back-projected to three-dimensional space through the camera intrinsic parameter matrix, and then matched with the point cloud data collected by the lidar.

[0033] As a preferred embodiment of the multi-view panoramic cloud stitching method described in this invention, the fine registration employs an improved iterative nearest-point algorithm. By introducing bidirectional distance constraints, the registration error is optimized, as shown in the formula:

[0034]

[0035] Where E(R,T) is the objective function for point cloud registration, representing the registration error between the rotation matrix R and the translation vector T. Fine alignment of the point clouds is achieved by minimizing this function. i p is a weighting factor for the point cloud density, representing the importance of each point in the optimization. It is typically calculated based on the local density of the point cloud. i Let q be the coordinates of a point in the target point cloud. i Let be the coordinates of a point in the source point cloud, ∥·∥ 2 This represents the square of the Euclidean distance, used to measure the distance between points.

[0036] The beneficial effects of this invention are:

[0037] By employing deep learning-based target detection technology, target regions are accurately selected, reducing the amount of point cloud data to be stitched and thus improving the stitching speed.

[0038] By performing fine matching and projection optimization on binocular camera images and lidar point cloud data, the problems of low point cloud accuracy and large error in existing technologies are overcome. Especially in complex power scenarios, it can effectively improve the accuracy and integrity of point cloud stitching.

[0039] By using filtering and feature extraction techniques, the impact of noise is effectively reduced, making the splicing process more robust to environmental changes and noise. Attached Figure Description

[0040] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0041] Figure 1 This is a flowchart illustrating the multi-view panoramic cloud stitching method of the present invention;

[0042] Figure 2 This is a schematic diagram of the calibration algorithm sub-process of the present invention;

[0043] Figure 3 This is a schematic diagram of the sub-process of the binocular ranging algorithm shown in this invention;

[0044] Figure 4 This is a schematic diagram of a sub-process of the two-dimensional image recognition algorithm shown in this invention;

[0045] Figure 5 This is a schematic diagram of the point cloud processing sub-process of the present invention;

[0046] Figure 6 This is a schematic diagram of the sub-process of the improved point cloud stitching algorithm shown in this invention. Detailed Implementation

[0047] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0048] Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without inventive effort should fall within the scope of protection of this invention.

[0049] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0050] According to an embodiment of the present invention, in combination Figure 1 The flowchart shown illustrates a multi-view panoramic cloud stitching method, which specifically includes the following steps:

[0051] S1. The binocular camera and lidar are jointly calibrated using a calibration algorithm to align the coordinate system;

[0052] S2. Use a binocular camera to acquire two-dimensional images and perform distortion correction. Generate a depth map based on parallax calculation and reconstruct a three-dimensional point cloud using the principle of triangulation.

[0053] S3. Perform deep learning target detection on the two-dimensional image and filter out the pole component region through non-maximum suppression;

[0054] S4. Collect multi-view point cloud data using lidar and perform point cloud segmentation processing, retaining a subset of the point cloud for the pole assembly;

[0055] S5. Match the two-dimensional pixel area of ​​the pole assembly with the LiDAR point cloud projection result, and filter the point cloud data belonging to the pole assembly.

[0056] S6. An improved point cloud stitching algorithm is adopted to align and fuse multi-view point clouds through initial registration and fine registration to generate a panoramic point cloud image.

[0057] Preferably, the present invention processes the data collected by the binocular camera to obtain a high-precision two-dimensional image, and then processes the point cloud information collected by the lidar to obtain data with good environmental adaptability and high reliability. After the point cloud and the two-dimensional image are matched, a multi-angle panoramic point cloud image is obtained through the point cloud stitching algorithm, which is not limited in the field of view and obtains a high-precision, complete, and robust point cloud image.

[0058] The following is combined Figures 2-6 The flowchart shown, along with some preferred or optional examples of the present invention, more specifically describes the implementation process and / or effects of certain embodiments of the present invention.

[0059] It should be noted that the multi-view panoramic point cloud stitching method provided in this embodiment of the invention processes the data collected by binocular cameras and LiDAR, configures corresponding point cloud image mapping and point cloud stitching visual processing algorithms (including point cloud processing, point cloud image mutual mapping, and point cloud stitching), thereby constructing a multi-view panoramic point cloud for pole components in power scenarios; wherein:

[0060] A binocular camera simultaneously acquires images using two cameras, obtaining the pixel difference between corresponding points in the views of the two cameras. It then uses the principle of triangulation to calculate depth data, thereby forming a 3D point cloud and reconstructing the object.

[0061] LiDAR uses non-contact laser ranging technology. It emits a laser beam through a transmitter. After the laser beam encounters an object, it is diffusely reflected back to the receiver. The radar module then calculates and forms a point cloud.

[0062] [Joint Calibration]

[0063] Construct a target, attach labels to the four corners of the target, and obtain target information collected by a binocular camera and a lidar. The binocular camera collects the image information of the target, and the lidar collects the geometric information (point cloud data) of the target. Record the precise coordinates of the four corner points of the target in space in the form of point cloud.

[0064] Feature point matching is performed between the images acquired by the binocular camera and the target, and the intrinsic parameters of the camera are calculated to obtain the coordinates of the label on the target in the pixel coordinate system.

[0065] The target contains at least multiple detectable feature points (such as checkerboard corner points, circular points) and image pairs (left / right images) captured by a binocular camera;

[0066] In an optional implementation, the image is converted to a grayscale image (or a color image) and denoised by image preprocessing, and feature points in the target image are extracted using a corner detection algorithm to obtain the initial position (u,v) of the target feature points in the pixel coordinate system.

[0067] For a checkerboard target, since the order and structure of the feature points are known, the matching process can be completed directly using indexing. The camera's intrinsic parameters are calculated using the target's known physical coordinates (world coordinates) and corresponding pixel coordinates. A mapping relationship is then constructed using world coordinates (X,Y,Z) and pixel coordinates (u,v).

[0068]

[0069] Where s is the scale factor, representing the transformation scale from world coordinates to image coordinates; u and v represent the pixel coordinates in the image, i.e., the point coordinates on the image plane; K is the camera's intrinsic parameter matrix, containing focal length and principal point coordinate information, used for projection from 3D world coordinates to image coordinates; R is the rotation matrix, describing the rotation relationship between the world coordinate system and the camera coordinate system; and T is the translation vector, describing the translation relationship between the world coordinate system and the camera coordinate system. 1 represents the coordinates of a three-dimensional point in the world coordinate system, and 1 is a constant value used to represent a three-dimensional point in homogeneous coordinates.

[0070] The camera's intrinsic parameters describe the mapping from the camera coordinate system to the pixel coordinate system, and the intrinsic parameter matrix K has the following form:

[0071]

[0072] Among them, f x f y The camera focal length (in pixels) is related to the sensor's physical focal length and pixel size. x c y Principal point coordinates indicate the position of the optical center in the pixel coordinate system;

[0073] For example, a point (u,v) in the pixel coordinate system is calculated by the following formula:

[0074]

[0075] Among them, (X) c ,Y c Z c ) represents the position of the target point in the camera coordinate system, and K is the intrinsic parameter matrix;

[0076] The above formula is used to obtain the pixel coordinates (u,v) of the target point on the image;

[0077] In an optional implementation, the rotation and translation matrix between the pixel coordinate system and the camera coordinate system is calculated using the SVD decomposition method. Taking the lower right corner of the target as the reference point, the result that minimizes the reprojection error after rotation and translation at that point is taken as the extrinsic parameter; where:

[0078] The target point's 3D world coordinates Pworld and pixel coordinates Ppixel are mapped together. For example, the relationship between 3D world coordinates and pixel coordinates is expressed by the following formula:

[0079]

[0080] Construct a linear equation for each corresponding point (X) i ,Y i Z i )→(x i ,y i Construct a linear equation:

[0081] x i =r 11 X i +r 12 Y i +r 13 Z i +t x

[0082] y i =r 21 X i +r 22 Y i +r 23 Z i +t y

[0083] z i =r 31 X i +r 32 Y i +r 33 Z i +t z

[0084] Combine the equations of all points to form a matrix:

[0085] A·[r 11 r 12 , ..., t z ] T =b

[0086] We use singular value decomposition to decompose the matrix and find the solution that minimizes the error.

[0087] [R|T] = arg min‖A·xb‖

[0088] Calculate the reprojection error using the point (Xr, Yr, Zr) at the lower right corner of the target:

[0089]

[0090] Among them, (u r ,v r ) represents the actual pixel coordinates, (u' r ,v' r () represents the projected pixel coordinates calculated using extrinsic parameters;

[0091] Minimize the reprojection error by optimizing R and T;

[0092] Furthermore, the target information acquired by the lidar is processed, and the center point coordinates of the tag are obtained by fitting a circle. By fitting these points, the center coordinates of the circle (c) are determined. x c y and radius r; where:

[0093] The point cloud is downsampled using a filtering method to reduce noise and computation. Conditional constraints are added to the cropped region to extract potential circular target points. The point cloud data is then projected onto a plane, simplifying it into a 2D plane problem. The center and radius of the circle are calculated by fitting using the least squares method.

[0094] The radar's external parameters are calculated based on the coordinates of the four tags on the target in the world coordinate system.

[0095] The lidar extrinsic parameter calibration process involves solving the rigid body transformation between two sets of coordinates in a 3D point cloud, including:

[0096] Four labels are selected at the four corners of the target, and their positions in the world coordinate system are known, denoted as (X...). i ,Y i Z i ), where i = 1, 2, 3, 4;

[0097] The positions (x, y) of the four corresponding points acquired by the lidar in the lidar coordinate system. i ,y i ,z i ):

[0098]

[0099] Calculate the centroids of the two point sets:

[0100]

[0101] Decentralize the two sets of points respectively:

[0102] Q world =P world -C world Q lidar =P lidar -C lidar

[0103] Construct the covariance matrix:

[0104]

[0105] The rotation and translation matrices from the camera to the radar are obtained using the SVD decomposition method, thus completing the joint calibration algorithm for the camera and radar. The rotation matrix R and translation vector T are solved using SVD decomposition, where:

[0106] Singular value decomposition (SVD) is performed on the covariance matrix H:

[0107] H=UΣV T

[0108] R = VU T

[0109] T = C world -R·C lida

[0110] Where U is the left singular matrix, V is the right singular matrix, and ∑V TIt is a singular value diagonal matrix.

[0111] Binocular ranging algorithm

[0112] It should be noted that the binocular ranging algorithm determines the distance from the device to the pole based on parallax, where:

[0113] Distortion correction is performed on the images captured by the left and right cameras to make the corresponding scan lines parallel;

[0114] In the images captured by the left and right cameras, select matching point pairs, calculate the pixel difference for densely matched point pairs, and construct a disparity map;

[0115] Based on the disparity map generated by dense matching point pairs and sparse matching point pairs, the depth is calculated using the triangle principle to achieve 3D reconstruction;

[0116] For example, the camera's intrinsic and extrinsic parameters are obtained, and lens distortion is corrected to make the image more regular. Multiple sets of images are acquired using a calibration board (such as a checkerboard pattern). OpenCV's calibration functions are used to calculate the camera's intrinsic and distortion parameters, find matching points for each pixel in the left and right images, and calculate the disparity, i.e., the horizontal displacement between pixels. Feature point matching or block matching methods are used to sparsely or densely match the point pairs in the left and right images. A disparity map is generated using OpenCV's StereoBM or StereoSGBM methods. Based on the disparity map and camera parameters, the depth of each pixel is calculated using the principle of triangulation.

[0117]

[0118] Where Z is the depth value, i.e. the distance from the camera, f is the focal length of the camera, B is the baseline length of the binocular camera, i.e. the distance between the optical axis centers of the two cameras, and d is the parallax value, i.e. the difference in the horizontal coordinates of the matching points in the left and right images.

[0119] Based on the depth value, points in the image coordinate system are transformed into 3D points in the world coordinate system. A point in the corrected image is defined with normalized pixel coordinates (x, y), and its transformation into a 3D point is achieved using the reprojection matrix Q.

[0120]

[0121] The coordinates of the three-dimensional point are:

[0122]

[0123] Where W is the normalization factor for homogeneous coordinates.

[0124] 2D Image Recognition

[0125] It should be noted that the 2D image recognition algorithm processes the 2D image information acquired by the binocular camera to achieve the identification of the pole component, wherein:

[0126] Read the two-dimensional images captured by the left and right cameras as input;

[0127] Feature extraction is performed on the left and right images using convolutional layers respectively;

[0128] For example, through convolution operations, information in the input image is transformed into a set of high-dimensional feature representations. These feature representations are used to capture key patterns or attributes in the image, such as edges, corners, colors, textures, and the contours, shapes, and structures of objects.

[0129] The features of the left and right eyes are concatenated to generate a feature map;

[0130] The feature map is input into the pooling layer and the fully connected layer for processing, and finally the target recognition result of the pole assembly is output.

[0131] As an example, all the collected image data is filtered, and the detection targets in each image are labeled using a labeling tool for training purposes;

[0132] The input image is the image to be detected of the pole component. Convolutional neural networks (such as ResNet, VGG, EfficientNet) are used to extract features and generate multi-channel feature maps. Different convolutional layers extract different features, including multi-level semantic information such as edges, textures, shapes, and structures. Pooling layers (such as max pooling or average pooling) are used to downsample the feature maps to reduce the spatial resolution of the feature maps, retain important information, improve computational efficiency, and output the extracted feature maps.

[0133] Multiple candidate regions are generated by sliding a fixed-size window across the image. Information from the feature map is mapped to each candidate box, and features within the candidate box are extracted. Region pooling or region alignment methods are used to unify the features of each candidate box to the same size.

[0134] The features of the candidate box regions are classified using a fully connected network or a convolutional classifier head. For each candidate box, a probability distribution containing all categories is output, with the output shape being: [number of candidate boxes] × [number of categories] × [number of candidate boxes] × [number of categories]. The features of the candidate box regions are regressed to their positions, and the bounding box correction value of each candidate box is output.

[0135] Non-maximum suppression is used to remove redundant candidate boxes and retain the box with the highest confidence. That is, the candidate boxes are sorted according to the predicted probability of the category. If the intersection-union ratio of two candidate boxes is greater than a certain threshold (such as 0.5), only the box with the highest confidence is retained.

[0136] Output the final detection bounding box for each target, including:

[0137] Target category: The type of target detected;

[0138] Confidence score: The probability that the target belongs to the predicted category;

[0139] Detection box position: coordinates of the bounding box;

[0140] For example, an image of a pole assembly is input, such as a scene containing components like "crossarm", "insulator", and "tower". The detected pole assembly categories are, for example, "crossarm", "insulator", "tower", and "pole". The confidence level for the crossarm is 95%, the confidence level for the insulator is 89%, the confidence level for the tower is 90%, and the confidence level for the pole is 93%.

[0141] Point cloud processing

[0142] It should be noted that the point cloud processing utilizes the Point Cloud Library to implement point cloud-related functions such as acquisition, filtering, segmentation, registration, retrieval, feature extraction, recognition, tracking, surface reconstruction, and visualization.

[0143] Read point cloud data collected from lidar sensors and accumulate multiple frames of point cloud data;

[0144] The point cloud is filtered to effectively remove noise points and outliers, making the point cloud smoother.

[0145] For example, outlier (noise) filtering is performed on the point cloud to remove abnormal or isolated points. If statistical filtering is used, the average distance from each point's neighborhood to that point is calculated. Based on global distribution statistics, points whose average distance exceeds a preset threshold range are considered outliers and removed.

[0146] In an optional implementation, the point cloud is segmented to exclude invalid point cloud noise within the area of ​​the pole assembly to be identified, thus separating the pole assembly from the surrounding noise, including:

[0147] Clustering is performed using point cloud density or spatial distribution characteristics to divide the point cloud into different clusters. The pole component is an independent dense cluster. The Euclidean distance between points is calculated, and points with a distance less than a set threshold are grouped into the same class. Through multiple segmentation operations, useless point clouds are continuously deleted. Finally, the output of the clustering is the pole point cloud.

[0148] In an optional implementation, feature extraction of the target object is performed to complete point cloud map matching, including:

[0149] By extracting the geometric features of the target object, the spatial transformation relationship (rotation and displacement) between two sets of point clouds is found, thereby completing point cloud registration or map merging;

[0150] Key point detection selects a set of key points in a point cloud for subsequent feature description and matching. For example, the point cloud is divided into a three-dimensional grid, and a representative point (such as the center point or average point) is selected in each grid as a key point.

[0151] For each key point, the local neighborhood is described to generate a feature vector. For example, the neighborhood geometric characteristics of each point are calculated, including the normal vector and the relationship between points.

[0152] When matching the entire point cloud, it is necessary to extract global features, such as calculating the point feature histogram of the entire point cloud, or extracting global features of the point cloud by combining viewpoint information.

[0153] After feature extraction, point clouds are registered. Point cloud alignment is achieved by estimating the rigid transformations (rotation and translation) between the two sets of point clouds, where:

[0154] Feature matching is performed using the feature descriptors of key points to find corresponding point pairs between the source point cloud and the target point cloud. The RANSAC algorithm is used to remove incorrect matches and estimate the initial transformation matrix. If the initial positions of the two sets of point clouds differ significantly, coarse registration is performed using global features (such as VFH).

[0155] Further optimization of the point cloud alignment results can be achieved through fine registration. For example, the nearest point pair can be found iteratively between the two sets of point clouds, and the rigid body transformation matrix can be optimized. The optimization objective is to minimize the Euclidean distance between the two sets of point clouds. Alternatively, the point cloud can be represented as a Gaussian distribution, and registration can be performed based on the distribution.

[0156] For example, after registration, multiple point clouds can be merged to generate a complete point cloud map. For instance, a rigid body transformation matrix can be used to transform the source point cloud to the target point cloud coordinate system and overlay it to generate a complete map, or the point clouds in overlapping areas can be fused to avoid duplicate or redundant points.

[0157] The result of point cloud matching is the alignment transformation matrix of the point cloud and the aligned point cloud.

[0158] Improved point cloud stitching algorithm

[0159] It should be noted that the point cloud stitching algorithm stitches together point clouds from multiple perspectives to obtain a panoramic image containing the pole components, wherein:

[0160] In an optional implementation, point cloud data is input, and the pixel values ​​corresponding to the point cloud in the image are calculated using calibration relationships, including:

[0161] The pixel values ​​corresponding to the point cloud in the image are calculated through the calibration relationship. The three-dimensional point cloud data is obtained from the LiDAR in the format of (x,y,z,intensity), where (x,y,z) are spatial coordinates and intensity is the reflection intensity. High-resolution two-dimensional images (left and right images) are obtained from the binocular camera in the format of color or grayscale images.

[0162] By calibrating the camera and LiDAR, the relationship between the LiDAR point cloud coordinate system and the camera pixel coordinate system is established, and the extrinsic and intrinsic parameter matrices are obtained.

[0163] Transform the lidar point cloud coordinates (x, y, z) to the camera coordinate system:

[0164]

[0165] Then, project the points in the camera coordinate system onto the image plane to obtain the pixel coordinates:

[0166]

[0167] The projection result is a pixel value (u,v), which represents the corresponding position of the point cloud in the image;

[0168] In an optional implementation, pixel values ​​are obtained based on the pole component identified from the image, including:

[0169] The input 2D image is converted to grayscale to enhance the recognizability of the pole components. For example, color images are converted to grayscale to reduce computational complexity, Gaussian filtering is used to remove noise and smooth the image, and the Sobel operator is used to extract edge features.

[0170] The target detection algorithm is used to locate the pole assembly area;

[0171] Extract all pixel coordinates (u,v) in the identified pole component area;

[0172] In an optional implementation, comparing the pixel values ​​of the two-dimensional image with the pixel values ​​of the point cloud to determine the rod component portion in the point cloud includes:

[0173] The projected pixel values ​​(u,v) of all points in the point cloud are compared with the pixel values ​​of the line pole components in the image. If the projected pixel value (u,v) of a point in the point cloud falls within the line pole component region of the image, then the point belongs to the line pole component.

[0174] Based on the comparison results, retain the point cloud points belonging to the pole assembly, remove other point cloud points, and generate a point cloud subset containing only the pole assembly;

[0175] By stitching together the multi-view point clouds containing pole components, a panoramic point cloud map is obtained.

[0176] As an example, the splicing method is as follows:

[0177] Input point cloud data from different perspectives;

[0178] The multi-view point clouds are roughly aligned using point cloud stitching algorithms (such as feature matching or initial pose estimation).

[0179] Extract local features of the point cloud (such as FPFH features), use RANSAC to find matching point pairs between point clouds, and estimate the initial rigid body transformation matrix;

[0180] Use external pose information, such as GPS, IMU or odometry data, to provide an initial pose for the point cloud;

[0181] Use refinement algorithms to optimize point cloud alignment, for example, use probability distribution methods to align point clouds;

[0182] Multi-frame point cloud data is fused to eliminate overlapping areas and generate a complete panoramic point cloud map.

[0183] Points in overlapping areas are filtered or weighted and merged to remove redundant points.

[0184] It should also be noted in this embodiment that the improved point cloud stitching algorithm, by introducing the fusion of binocular camera and LiDAR, combined with the accurate stitching of multi-view point clouds, can not only accurately segment the target of the pole component, but also generate a complete panoramic point cloud map. It is suitable for target extraction and map building tasks in complex environments, such as extrinsic parameter calibration, projection calculation, target matching and fusion, ensuring high precision and high reliability of data processing.

[0185] The image preprocessing, feature extraction, and matching calculation methods in the aforementioned joint calibration can be carried out using existing technologies and methods, and will not be elaborated further in this example.

[0186] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A multi-view panoramic cloud stitching method, characterized in that, include: The binocular camera and lidar are jointly calibrated using a calibration algorithm to align the coordinate system. The binocular camera is used to acquire two-dimensional images and perform distortion correction. A depth map is generated based on parallax calculation, and three-dimensional point cloud reconstruction is performed by combining the principle of triangulation. The disparity calculation uses a block matching algorithm to generate a disparity map, and converts pixel coordinates into three-dimensional point coordinates through the reprojection matrix Q, as shown in the formula: Where x and y are the pixel coordinates in the two-dimensional image, and d is the disparity value, representing the pixel difference of the same object point in the left and right images. Let Q be the coordinates of the point cloud after 3D reconstruction, and let Q be the reprojection matrix. The pixel coordinates are converted into point coordinates in 3D space based on the camera's intrinsic and extrinsic parameters. Deep learning target detection is performed on the two-dimensional image, and the pole component region is filtered out by non-maximum suppression; The laser radar is used to collect multi-view point cloud data and perform point cloud segmentation processing to retain a subset of the point cloud of the pole assembly. The point cloud segmentation uses a density clustering algorithm based on DBSCAN, with a set neighborhood radius. and minimum points To classify the point cloud, separate the line pole components from the background noise, and combine the separated line pole components into a line pole component point cloud subset; The two-dimensional pixel region of the pole assembly is matched with the point cloud projection result of the lidar, and the point cloud data belonging to the pole assembly is filtered. When matching the two-dimensional pixel region of the pole assembly with the point cloud projection result of the lidar, the two-dimensional image region of the pole assembly needs to be back-projected to three-dimensional space through the camera intrinsic parameter matrix, and then matched with the point cloud data collected by the lidar. An improved point cloud stitching algorithm is used to align and fuse multi-view point clouds through initial registration and fine registration to generate a panoramic point cloud image. The fine registration employs an improved iterative nearest-point algorithm, which optimizes the registration error by introducing bidirectional distance constraints. The formula is as follows: in, The objective function for point cloud registration represents the registration error between the rotation matrix R and the translation vector T. Fine alignment of the point cloud is achieved by minimizing this function. This is a weighting factor for the point cloud density, representing the importance of each point in the optimization process. It is typically calculated based on the local density of the point cloud. Given the coordinates of a point in the target point cloud, The coordinates of the points in the source point cloud. This represents the square of the Euclidean distance, used to measure the distance between points.

2. The multi-view panoramic cloud stitching method according to claim 1, characterized in that, The joint calibration includes: A target with feature labels is constructed, and two-dimensional image information of the target is acquired by the binocular camera, and three-dimensional point cloud data of the target is acquired by the lidar. Using feature point matching and singular value decomposition, the camera intrinsic and extrinsic parameter matrices are calculated, and the coordinate system is aligned. The target with the feature label is a checkerboard target with a reflective label.

3. The multi-view panoramic cloud stitching method according to claim 2, characterized in that, In the checkerboard image, feature points are extracted using Harris corner detection, and singular value decomposition is used to calculate the camera intrinsic and extrinsic parameter matrices, thereby achieving coordinate system alignment between the two sensors, including: Where s is the scale factor, representing the transformation scale from world coordinates to image coordinates; u and v represent the pixel coordinates in the image, i.e., the point coordinates on the image plane; K is the camera's intrinsic parameter matrix, containing focal length and principal point coordinate information, used for projection from 3D world coordinates to image coordinates; R is the rotation matrix, describing the rotation relationship between the world coordinate system and the camera coordinate system; and T is the translation vector, describing the translation relationship between the world coordinate system and the camera coordinate system. 1 represents the coordinates of a three-dimensional point in the world coordinate system, and 1 is a constant value used to represent a three-dimensional point in homogeneous coordinates.

4. The multi-view panoramic cloud stitching method according to claim 1, characterized in that, The deep learning object detection employs a convolutional neural network, including feature extraction, candidate box generation, and non-maximum suppression processing, outputting the bounding box and confidence score of the wire rod component, wherein: Features are extracted using convolutional layers, candidate boxes are generated using a region proposal network, candidate boxes with high overlap are removed, and the best detection results are retained.

Images