Multi-view heterogeneous point cloud registration method and device based on semantic segmentation

Abstract

The invention discloses a multi-view-angle different-source point cloud registration method and device based on semantic segmentation, and the method comprises the steps: collecting overlook and ground head-up view angle different-source data of a real scene through different sensors, constructing a different-source three-dimensional scene point cloud through a three-dimensional reconstruction algorithm, and carrying out the downsampling preprocessing, thereby obtaining a training data set; the data set is imported into a RandLA-Net algorithm for training; a semantic segmentation algorithm is utilized to respectively obtain point cloud semantic tags of different visual angles, and specific tag point cloud is extracted and point cloud filtering processing is carried out; and making a three-dimensional bounding box for the filtered label point cloud, and indirectly obtaining two heterogenous point cloud transformation matrixes by registering the bounding boxes in the two world coordinate systems, thereby realizing multi-view heterogenous point cloud registration fusion. According to the method, the multi-view different-source point cloud registration problem can be solved by utilizing the technologies of point cloud downsampling, semantic segmentation, point cloud filtering, a three-dimensional bounding box and the like, the registration robustness can be enhanced, and the registration accuracy can be improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of point cloud registration, and in particular relates to a multi-view heterogeneous point cloud registration method and device based on semantic segmentation, which can be used for scene registration fusion reconstructed by different sensors in multi-view situations. Background Art

[0002] Point cloud registration aims to correctly align multiple point clouds to the same coordinate system to form a more complete point cloud. It is widely used in fields such as three-dimensional reconstruction, parameter evaluation, positioning and posture estimation. Point cloud registration technology is also involved in emerging applications such as autonomous driving, robotics and augmented reality.

[0003] According to the different acquisition principles of the point clouds to be registered, the point cloud registration problem can be divided into two categories: homologous point cloud registration and heterologous point cloud registration. At present, homologous point cloud registration is widely studied, and usually adopts registration methods based on traditional optimization, feature learning methods, and end-to-end learning-based registration methods. The registration method based on traditional optimization (ICP algorithm and its optimized variant algorithm) can accurately register two completely corresponding point cloud data without external points through two stages of correspondence search and transformation estimation. The feature learning method (3DMatch, PPFNet) uses a deep neural network to extract the features of the point cloud and estimates the transformation matrix based on the correspondence relationship of the features or direct regression from the features; the end-to-end learning method (FMR, PRNet) inputs two point clouds into the end-to-end neural network, and the output is the transformation matrix that aligns the two point clouds; these two learning-based methods can well solve the registration of homologous point clouds with low overlap rate. The top-down and head-up perspective heterologous point cloud registration requires the fusion of heterologous point cloud registration under two different perspectives and different world coordinate systems, the sky and the ground. Compared with homologous point clouds, multi-view heterogeneous point clouds have the characteristics of large outliers, density differences, and large rotation poses. In particular, the density differences of heterogeneous point clouds and the increase in the number of point clouds lead to poor robustness of homologous point cloud registration methods, and the performance of registration methods is significantly reduced, making it difficult to apply to multi-view heterogeneous 3D point cloud registration problems. Hybrid Fusion is based on the traditional optimization registration method. It uses traditional methods for fine registration based on coarse matching, which solves the problem of multi-view heterogeneous point cloud registration accuracy. However, it is not applicable to multi-view heterogeneous point clouds with large initial pose differences. In addition, there is currently no large-scale multi-view heterogeneous data set available, which hinders the development of neural networks in this field. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the present invention provides a multi-view heterogeneous point cloud registration method and device based on semantic segmentation, aiming to solve the problem of multi-view heterogeneous point cloud registration with large initial poses.

[0005] The present invention retrains the RandLA-Net model to obtain the semantic CLASS of specific objects in large-scene multi-view heterogeneous point clouds, and extracts specific label point clouds based on the semantic CLASS. The specific label point clouds are filtered by a vertical ground plane segmentation method and a density clustering filtering algorithm. A minimum external matrix frame is made for the filtered label point cloud, and the minimum external matrix frame is converted into a point cloud. By aligning the minimum external frames of the point clouds in two world coordinate systems, the transformation matrix between the two heterogeneous point clouds can be indirectly obtained, and finally the multi-view heterogeneous point cloud registration and fusion is realized.

[0006] The first aspect of the present invention relates to a multi-view heterogeneous point cloud registration method based on semantic segmentation, comprising the following steps:

[0007] Step 1: Collect outdoor multi-view heterogeneous point cloud datasets to be registered and preprocess the datasets:

[0008] Step 1-1: Crop, align the coordinate system, and downsample each point cloud in the dataset;

[0009] Step 1-2: semantically annotate the preprocessed point cloud data to obtain the Heshan-3D dataset;

[0010] Step 1-3: Build the RandLA-Net model;

[0011] Step 1-4: Use the Heshan-3D dataset to train the RandLA-Net model to obtain parameters and weights of the point cloud samples, and use the parameters and weights as the parameters and weights of the trained RandLA-Net model;

[0012] Step 1-5: Use the trained RandLA-Net model to perform semantic segmentation on the standardized scene point cloud to be registered, predict the semantic category Class of each point in the scene point cloud, and define Class car For car category, Class building For room type, Class tree for the tree category;

[0013] Step 2: Segment the class according to semantics car 、Class building 、Class tree , extract the respective category point cloud data clusters respectively;

[0014] Step 3: Filter the point cloud data clusters to remove points with semantic classification errors and obtain cleaner point cloud category clusters:

[0015] Step 3-1: Vertical ground segmentation filtering: RANSAC fits the ground plane and sets a suitable ground plane distance threshold, filtering out all point clouds within the threshold.

[0016] Step 3-2: Use DBSCAN density clustering filter to filter out clean point cloud category clusters from misclassified point cloud clusters;

[0017] Step 4: Make a minimum bounding box for the point cloud category cluster after filtering, and convert the minimum bounding box into point cloud;

[0018] Step 5: Use the pose estimation algorithm to obtain the rotation matrix R and translation matrix t between the minimum bounding boxes of the two world coordinate systems;

[0019] Step 6: R and t obtained in step 5 constitute the transformation matrix T between the two heterogeneous point clouds to be registered. T obtained through the minimum bounding box indirectly realizes the registration fusion of the top-down and head-on point clouds.

[0020] Preferably, the multi-view heterogeneous point cloud to be registered in step 1 is a point cloud collected by two different sensors from two viewpoints; multi-view refers to the data collection viewpoint, which is the ground level collection viewpoint and the aerial top-down collection viewpoint; heterogeneous refers to the sensor used for data collection, which is the ground laser radar and the RGB camera carried by the aerial drone; the multi-view heterogeneous point cloud to be registered P = {p1, p2, ..., p n}, Q={q1,q2,...,q m The data that needs to be obtained by the sensor is subsequently obtained using the reconstruction algorithm.

[0021] Preferably, the point cloud filtering method described in step 3 is to filter out points with semantic classification errors by combining vertical plane segmentation and density clustering filtering to obtain a relatively clean point cloud category cluster; specifically, according to the plane formula: Ax+By+Cz+D=0, at least three points are required for plane fitting, with P building For example, from P building Three points are randomly selected to calculate the plane parameters A, B, C, and D. The remaining point cloud coordinates are substituted into the plane formula to calculate the error, and then compared with the threshold error. If the point is less than the threshold error, it is regarded as a point on the plane, thus fitting the ground plane; according to the distance formula: The distance from each point outside the plane to the fitting plane is calculated and compared with the set height threshold. The points within the threshold are regarded as noise points at a certain height from the ground plane. According to this principle, most of the noise points with incorrect semantic classification can be effectively removed. Then, DBSCAN density clustering is used to remove the remaining noise points with incorrect semantic classification again, so as to obtain Class building The target point cloud P' of the class building , the same method can be used to obtain P'car , P' tree , Q' building , Q' car , Q' tree .

[0022] Preferably, since the collected data is obtained by two different sensors, the large density difference brings serious challenges to the attitude estimation algorithm. In order to solve this challenge, step 4 makes a minimum external matrix frame for the filtered point cloud, and then converts the external frame into point cloud, which are represented as P frame ={p1,p2,...,p k}, Q frame ={q1,q2,...,q l}, where k, l represent the number of point clouds in the box after point clouding; P frame and Q frame Select as a new point cloud cluster to be matched, according to the formula: R * , The rotation matrix R between the point clouded frames can be calculated * and the translation matrix t * , where N is P frame With Q frame Match points between.

[0023] Preferably, the pose estimation algorithm described in step 5 is a 4PCS algorithm.

[0024] Preferably, the transformation matrix described in step 6 is is the rotation matrix R obtained in step 5 * and the translation matrix t * The structure is obtained, T can make Minimum, so that the initial point cloud to be registered P = {p1, p2, ..., p n}, Q={q1,q2,...,q m}Registration and fusion.

[0025] In the present invention, two heterogeneous point clouds with different viewing angles are assumed to be P = {p1, p2, ..., p n}, Q={q1,q2,...,q m}, let the category point cloud cluster extracted from the semantic classification result be P tree ={p1,p2,...,p n}, P building ={p1,p2,...,p n}, P car ={p1,p2,...,p n}, Q tree ={q1,q2,...,q m}, Qbuilding ={q1,q2,...,q m}, Q car ={q1,q2,...,q m}, where n and m above only represent the difference in the total number of point clouds in the two point cloud clusters;

[0026] According to the plane formula: Ax+By+Cz+D=0, plane fitting requires at least three points. building For example, from P building Three points are randomly selected to calculate the plane parameters A, B, C, and D. The coordinates of the remaining point clouds are substituted into the plane formula to calculate the error, which is then compared with the threshold error. If the point is smaller than the threshold error, it is regarded as a point on the plane, thus fitting the ground plane.

[0027] According to the distance formula: The distance from each point outside the plane to the fitting plane is calculated and compared with the set height threshold. The points within the threshold are regarded as noise points at a certain height from the ground plane. According to this principle, most of the noise points with incorrect semantic classification can be effectively removed. Then, DBSCAN density clustering is used to remove the remaining noise points with incorrect semantic classification again, so as to obtain Class building The target point cloud P' of the class building , the same method can be used to obtain P' car , P' tree , Q' building , Q' car , Q' tree ;

[0028] In order to effectively solve the challenge brought by the density difference of heterogeneous point clouds, the minimum bounding box of the point cloud obtained in step 4 is converted into point cloud, which are P frame ={p1,p2,...,p k}, Q frame ={q1,q2,...,q l}, where k and l represent the number of point clouds in the box after point clouding;

[0029] P frame and Q frame Select as a new point cloud cluster to be matched, according to the formula: R * , The rotation matrix R between the point clouded frames can be calculated * and the translation matrix t * , where N is P frame With Q frame Matching points between;

[0030] According to the rotation matrix R obtained in step 5 * and the translation matrix t *, which can form the final pose transformation matrix T can make Minimum, so that the initial point cloud to be registered P = {p1, p2, ..., p n}, Q={q1,q2,...,q m}Registration and fusion.

[0031] The second aspect of the present invention relates to a multi-view heterogeneous point cloud registration device based on semantic segmentation, comprising a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement a multi-view heterogeneous point cloud registration method based on semantic segmentation of the present invention.

[0032] The third aspect of the present invention relates to a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements a multi-view heterogeneous point cloud registration method based on semantic segmentation of the present invention.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] 1. The present invention proposes a multi-view heterogeneous point cloud registration method and device based on semantic segmentation. For the first time, the semantic segmentation technology is applied to the heterogeneous point cloud registration from the top-down and top-down perspectives. Compared with the prior art that uses the entire point cloud as the registration object, the present invention uses the semantic segmentation technology to extract specific objects in a targeted manner, thereby effectively filtering out the background and outlier point clouds, and reducing the outlier point cloud noise and the interference of other irrelevant point clouds;

[0035] 2. Compared with the precise registration of heterogeneous point clouds from top-down and head-up perspectives based only on coarse matching, the present invention can effectively register heterogeneous point clouds with large initial poses;

[0036] 3. The present invention adopts ground plane segmentation, and density clustering filtering can effectively filter out semantic segmentation misclassified point clouds, reduce the error of estimated rigid transformation matrix, and enhance the robustness of registration;

[0037] 4. The present invention uses the rotation matrix and translation matrix between the minimum bounding box of the specific object after estimating the filtering to indirectly realize the top-down and head-up perspective heterogeneous point cloud registration. This method can effectively solve the challenges brought by the density difference of heterogeneous point clouds. At the same time, compared with the prior art that uses the entire point cloud as the processing object, the indirect point cloud minimum bounding box of the specific object after filtering can realize the processing of very few point clouds and greatly reduce the calculation speed of rigid transformation estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 is a flow chart of the method of the present invention;

[0039] Figure 2It is the overall framework diagram of the present invention;

[0040] Figure 3 Schematic diagram of the viewing angle and registration scene used in the present invention;

[0041] Figure 4 The original point cloud P image of one of the viewing angles (top view) of the present invention;

[0042] Figure 5 It is the original point cloud Q map of one of the viewing angles (horizontal viewing angle) of the present invention;

[0043] Figure 6 It is the original relative posture relationship diagram of the original point cloud P and the original point cloud Q of the present invention;

[0044] Figure 7 It is a registration and fusion completion diagram of the original point cloud P and the original point cloud Q of the present invention;

[0045] Figure 8 It is a schematic diagram of the effect of completing the registration and fusion of the original point cloud P and the original point cloud Q of the present invention.

[0046] Fig. 9 This is a local schematic diagram of the registration and fusion of the control group of the present invention.

[0047] Fig.10 This is a global schematic diagram of the registration and fusion of the control group of the present invention. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are clearly and completely described below in combination with the drawings and specific embodiments, but the implementation methods of the present invention are not limited to this.

[0049] Example 1

[0050] This embodiment relates to a method for 3D reconstruction of ground landscape using a multi-view heterogeneous point cloud registration method based on semantic segmentation of the present invention, as shown in the flowchart of the present invention ( Figure 1 ), including the input of the heterogeneous point cloud to be registered from the head-up and top-down perspectives, data set preparation and preprocessing, obtaining semantic segmentation results and extracting point cloud clusters of each category according to semantic category labels, performing vertical ground plane segmentation filtering and density clustering filtering on the extracted point cloud clusters to obtain clean point cloud clusters, making a minimum external frame for the clean point cloud and estimating the rotation matrix and translation matrix between the frames, and indirectly obtaining the transformation matrix of the point cloud to be registered using the rotation matrix and translation matrix between the frames, and finally realizing the registration and fusion of heterogeneous point clouds from the head-up and top-down perspectives. To further facilitate the understanding of the multi-view heterogeneous point cloud registration method, Figure 2 The overall framework of the registration process is presented.

[0051] The specific steps are as follows:

[0052] Step 1: Collect outdoor multi-view heterogeneous point cloud datasets to be registered and preprocess the datasets (such as Figure 3 As shown in the figure, the point clouds to be registered are obtained from the ground-based laser radar in the horizontal view and the aerial drone with an RBG camera in the top view. ):

[0053] Step 1-1: Crop, align the coordinate system, and downsample each point cloud in the dataset;

[0054] Step 1-2: semantically annotate the preprocessed point cloud data to obtain the Heshan-3D dataset;

[0055] Step 1-3: Build the RandLA-Net model;

[0056] Step 1-4: Use the Heshan-3D dataset to train the RandLA-Net model to obtain parameters and weights of the point cloud samples, and use the parameters and weights as the parameters and weights of the trained RandLA-Net model;

[0057] Step 1-6: Use the trained RandLA-Net model to perform semantic segmentation on the standardized scene point cloud to be registered, predict the semantic category Class of each point in the scene point cloud, and define Class car For car category, Class building For room type, Class tree for the tree category;

[0058] Step 2: Segment the class according to semantics car 、Class building 、Class tree , extract the respective category point cloud data clusters respectively;

[0059] Step 3: Filter the point cloud data clusters to remove points with semantic classification errors and obtain cleaner point cloud category clusters:

[0060] Step 3-1: Vertical ground segmentation filtering: RANSAC fits the ground plane and sets a suitable ground plane distance threshold, filtering out all point clouds within the threshold.

[0061] Step 3-2: Use DBSCAN density clustering filter to filter out clean point cloud category clusters from misclassified point cloud clusters;

[0062] Step 4: Make a minimum bounding box for the point cloud category cluster after filtering, and convert the minimum bounding box into point cloud;

[0063] Step 5: Use the pose estimation algorithm to obtain the rotation matrix R and translation matrix t between the minimum bounding boxes of the two world coordinate systems;

[0064] Step 6: R and t obtained in step 5 constitute the transformation matrix T between the two heterogeneous point clouds to be registered. T obtained through the minimum bounding box indirectly realizes the registration fusion of the top-down and head-on point clouds.

[0065] Step 7: Use the registration and fusion results of the top-down and head-on point clouds in step 6 to perform three-dimensional reconstruction of the ground landscape.

[0066] In one embodiment of the present invention, steps 1 to 6 include:

[0067] Assume that two heterogeneous point clouds from different perspectives are P = {p1, p2, ..., p n}( Figure 4 ), Q={q1,q2,...,q m}( Figure 5 ), let the category point cloud cluster extracted from the semantic classification result be P tree ={p1,p2,...,p n}, P building ={p1,p2,...,p n}, P car ={p1,p2,...,p n}, Q tree ={q1,q2,...,q m}, Q building ={q1,q2,...,q m}, Q car ={q1,q2,...,q m}, where n and m above only represent the difference in the total number of point clouds in the two point cloud clusters;

[0068] According to the plane formula: Ax+By+Cz+D=0, plane fitting requires at least three points. building For example, from P building Three points are randomly selected to calculate the plane parameters A, B, C, and D. The coordinates of the remaining point clouds are substituted into the plane formula to calculate the error, which is then compared with the threshold error. If the point is smaller than the threshold error, it is regarded as a point on the plane, thus fitting the ground plane.

[0069] According to the distance formula: The distance from each point outside the plane to the fitting plane is calculated and compared with the set height threshold. The points within the threshold are regarded as noise points at a certain height from the ground plane. According to this principle, most of the noise points with incorrect semantic classification can be effectively removed. Then, DBSCAN density clustering is used to remove the remaining noise points with incorrect semantic classification again, so as to obtain Classbuilding The target point cloud P' of the class building , the same method can be used to obtain P' car , P' tree , Q' building , Q' car , Q' tree ;

[0070] In order to effectively solve the challenge brought by the density difference of heterogeneous point clouds, the minimum bounding box of the point cloud obtained in step 4 is converted into point cloud, which are P frame ={p1,p2,...,p k}, Q frame ={q1,q2,...,q l}, where k and l represent the number of point clouds in the box after point clouding;

[0071] P frame and Q frame Select as a new point cloud cluster to be matched, according to the formula: R * , The rotation matrix R between the point clouded frames can be calculated * and the translation matrix t * , where N is P frame With Q frame Matching points between;

[0072] According to the rotation matrix R obtained in step 5 * and the translation matrix t * , which can form the final pose transformation matrix

[0073] Figure 6 is the original relative pose relationship between the original point cloud P and the original point cloud Q, and the pose transformation matrix T can make

[0074] Minimum, realize the registration fusion of original point cloud P and original point cloud Q ( Figure 7 ). Figure 8 It is a schematic diagram of the effect of completing the registration and fusion of the original point cloud P and the original point cloud Q of the present invention.

[0075] The above is only a specific implementation of the present invention, but the protection scope of the present invention is not limited thereto. Those skilled in the art should understand that the present invention includes but is not limited to the contents described in the drawings and the above specific implementation. Any modification that does not deviate from the functional and structural principles of the present invention will be included in the scope of the claims.

[0076] In order to verify the effectiveness of the method of the present invention, the present invention is compared with the method based on traditional optimization (ICP) from the two aspects of registration efficiency and registration accuracy. Due to the large number of point clouds of outdoor heterogeneous point cloud data to be registered (original point cloud P and original point cloud Q), the calculation amount of the traditional optimization method is too large, which greatly affects the efficiency of the verification work. For this reason, the present invention downsamples the original point clouds P and Q. It has been experimentally demonstrated that the present invention only takes 7.53516 seconds to complete the registration, while the method based on traditional optimization takes 62.7094 seconds to register the point clouds to be registered, and there are large rotation errors and translation errors.

[0077]

[0078] Table 1 Registration rotation error and registration time

[0079]

[0080]

[0081] Table 2 Registration translation error

[0082] Note: The indicator calculation formula is as follows:

[0083] Assume the transformation matrix is The translation matrix of the transformation matrix is ​​t, the three-axis rotation errors of the Ground Truth (GT) of the transformation matrix are ax', ay', az', and the translation matrix of GT is t'

[0084] The calculation formula for the rotation angle of the three axes is:

[0085] ay=-arcsin(R 31 );

[0086]

[0087] Rotation error calculation formula:

[0088] x-axis rotation error=|ax-ax'|,

[0089] y-axis rotation error=|ay-ay'|,

[0090] z-axis rotation error=|az-az'|;

[0091] The translation error calculation formula is: |t-t'|.

[0092] Example 2

[0093] The present embodiment relates to a multi-view heterogeneous point cloud registration device based on semantic segmentation, comprising a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement a multi-view heterogeneous point cloud registration method based on semantic segmentation of embodiment 1.

[0094] Example 3

[0095] This embodiment relates to a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, a multi-view heterogeneous point cloud registration method based on semantic segmentation of Embodiment 1 is implemented.

[0096] The contents described in the embodiments of this specification are merely an enumeration of the implementation forms of the inventive concept. The protection scope of the present invention should not be regarded as limited to the specific forms described in the embodiments. The protection scope of the present invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. A multi-view heterogeneous point cloud registration method based on semantic segmentation, characterized in that: The following steps are involved: Step 1: Collect outdoor multi-view heterogeneous point cloud datasets to be registered and preprocess the datasets: Step 1-1: Crop, align the coordinate system, and downsample each point cloud in the dataset; Step 1-2: semantically annotate the preprocessed point cloud data to obtain the Heshan-3D dataset; Step 1-3: Build the RandLA-Net model; Step 1-4: Use the Heshan-3D dataset to train the RandLA-Net model to obtain parameters and weights of the point cloud samples, and use the parameters and weights as the parameters and weights of the trained RandLA-Net model; Step 1-5: Use the trained RandLA-Net model to perform semantic segmentation on the standardized scene point cloud to be registered, predict the semantic category Class of each point in the scene point cloud, and define Class car For car category, Class building For room type, Class tree for the tree category; Step 2: Segment the class according to semantics car 、Class building 、Class tree , extract the respective category point cloud data clusters respectively; Step 3: Filter the point cloud data clusters to remove points with semantic classification errors and obtain clean point cloud category clusters: Step 3-1: Vertical ground segmentation filtering: RANSAC fits the ground plane and sets a suitable ground plane distance threshold, filtering out all point clouds within the threshold. Step 3-2: Use DBSCAN density clustering filter to filter out clean point cloud category clusters from misclassified point cloud clusters; Step 4: Make a minimum bounding box for the point cloud category cluster after filtering, and convert the minimum bounding box into point cloud; Step 5: Use the pose estimation algorithm to obtain the rotation matrix R and translation matrix t between the minimum bounding boxes of the two world coordinate systems; Step 6: R and t obtained in step 5 constitute the transformation matrix T between the two heterogeneous point clouds to be registered. T obtained through the minimum bounding box indirectly realizes the registration fusion of the top-down and head-on point clouds.

2. According to the method of multi-view heterogeneous point cloud registration based on semantic segmentation in claim 1, it is characterized by: The multi-view heterogeneous point cloud to be registered in step 1 is a point cloud acquired by two different sensors from two viewpoints; multi-view refers to the data acquisition viewpoint, which is a ground level acquisition viewpoint and an aerial bird's-eye view acquisition viewpoint; Heterogeneous sources refer to the sensors used for data collection, which are ground-based laser radar and RGB cameras carried by aerial drones; the multi-view heterogeneous point cloud to be registered P = {p1, p2, ..., p n }, Q={q1,q2,...,q m The data that needs to be obtained by the sensor is subsequently obtained using the reconstruction algorithm.

3. According to the method of multi-view heterogeneous point cloud registration based on semantic segmentation in claim 1, it is characterized by: The point cloud filtering method described in step 3 is to combine vertical ground plane segmentation and density clustering filtering to filter out points with semantic classification errors to obtain a relatively clean point cloud category cluster; specifically, according to the plane formula: Ax+By+Cz+D=0, it is known that plane fitting requires at least three points, with P building For example, from P building Three points are randomly selected to calculate the plane parameters A, B, C, and D. The remaining point cloud coordinates are substituted into the plane formula to calculate the error, and then compared with the threshold error. If the point is less than the threshold error, it is regarded as a point on the plane, thus fitting the ground plane; according to the distance formula: The distance from each point outside the plane to the fitting plane is calculated and compared with the set height threshold. The points within the threshold are regarded as noise points at a certain height from the ground plane. According to this principle, most of the noise points with incorrect semantic classification can be effectively removed. Then, DBSCAN density clustering is used to remove the remaining noise points with incorrect semantic classification again, so as to obtain Class building The target point cloud P' of the class building , the same method can be used to obtain P' car , P' tree , Q' building , Q' car , Q' tree .

4. The multi-view heterogeneous point cloud registration method based on semantic segmentation according to claim 1, characterized in that: The collected data are obtained by two different sensors. The large difference in density poses a serious challenge to the attitude estimation algorithm. In order to solve this challenge, step 4 makes a minimum external matrix frame for the filtered point cloud, and then converts the external frame into point cloud, which are represented as P frame ={p1,p2,...,p k }, Q frame ={q1,q2,...,q l }, where k, l represent the number of point clouds in the box after point clouding; P frame and Q frame Select as a new point cloud cluster to be matched, according to the formula: The rotation matrix R between the point clouded frames can be calculated * and the translation matrix t * , where N is P frame With Q frame Match points between.

5. The multi-view heterogeneous point cloud registration method based on semantic segmentation according to claim 1, characterized in that: The pose estimation algorithm described in step 5 is the 4PCS algorithm.

6. The multi-view heterogeneous point cloud registration method based on semantic segmentation according to claim 1, characterized in that: The transformation matrix described in step 6 is the rotation matrix R obtained in step 5 * and the translation matrix t * The structure is obtained, T can make Minimum, so that the initial point cloud to be registered P = {p1, p2, ..., p n }, Q={q1,q2,...,q m }Registration and fusion.

7. A multi-view heterogeneous point cloud registration device based on semantic segmentation, characterized in that: It includes a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, it is used to implement a multi-view heterogeneous point cloud registration method based on semantic segmentation as described in any one of claims 1-6.

8. A computer-readable storage medium, characterized in that: A program is stored thereon, and when the program is executed by a processor, a multi-view heterogeneous point cloud registration method based on semantic segmentation as described in any one of claims 1 to 6 is implemented.

Images