Multi-source point cloud fusion reconstruction method driven by scene semantics

Abstract

The invention discloses a scene semantic driven multi-source point cloud fusion reconstruction method, which comprises the following steps of: firstly, carrying out semantic segmentation on an indoor scene by utilizing a point cloud semantic segmentation network so as to shrink a reconstruction domain of the whole indoor scene into a meticulously designed semantic object; a robust feature matching relation is quickly constructed by fusing scene semantic advanced plane / line descriptors, so that the solution of seven-degree-of-freedom conversion in multi-source point cloud reconstruction is realized, and meanwhile, a ground prior registration module is proposed to solve an initial rotation matrix, so that the problem of large rotation transformation solution caused by significant change of a visual angle in multi-source point cloud scanning is solved, and the multi-source point cloud reconstruction efficiency is improved. And efficient and accurate initial iterative conversion is realized by using an RANSAC algorithm based on scene semantic extension, so that the scene reconstruction reasoning time delay is greatly reduced. According to the method, mixed structure differences of multi-source point cloud data are overcome through plane-based high-dimensional feature matching, robust fusion of a three-dimensional map is realized, and meanwhile, priori knowledge is introduced into fusion scene semantics in a multi-source point cloud registration process, so that the reconstruction rate is further accelerated.

Description

Technical Field

[0001] The present invention belongs to the technical field of three-dimensional reconstruction, and in particular relates to a scene semantics-driven multi-source point cloud fusion reconstruction method. Background Art

[0002] In recent years, with the construction of real-life 3D China and digital cities, urban digitization has gradually shifted from 2D to 3D, and has put forward higher requirements for the integrity and semantic richness of 3D map information. Intelligent agents need more semantic information of indoor environments to complete advanced tasks, such as finding a cup in the kitchen and pouring water into it. Therefore, complete and rich scene information is a necessary capability for future intelligent agents to perform 3D perception applications. Although intelligent agents can currently carry a single sensor to collect data sequentially to reconstruct a global point cloud map of the scene, the limited collection angle cannot cover the surface of all targets in the scene, resulting in the inevitable loss of some scene information representation. For example, traditional LiDAR scanning and reconstruction technology can quickly restore the contour size information of the target object to generate a large map of the scene, and achieve millimeter or even micron level accuracy, but it lacks the surface texture representation and attribute information of the object. Although the 3D reconstruction technology based on depth image can record detailed target object structure and semantic attribute information, it actually lacks real size ratio information and is greatly affected by lighting, resulting in poor 3D map reconstruction effect. It is more suitable for detail collection of close-range scenes.

[0003] Traditional multi-source point cloud registration technology uses independent feature encoding branches when processing point cloud data from different sensor sources, and fuses the generated data according to a specific fusion strategy. It not only focuses on the input data type (such as raw data, features, results) and the stage of feature encoding during the fusion process, but also strives to improve the method of multi-source point cloud fusion in stages to adapt to diverse task requirements and scenarios. Point correspondence-based methods find corresponding points by directly designing optimization strategies or manual features on the original point cloud data, and use this to estimate the seven-degree-of-freedom transformation matrix. This type of method relies on the setting of initial parameters. Therefore, in order to make the method universal and flexible, the model correspondence-based method has become a current research hotspot. Conventional model correspondence-based methods convert the multi-source point cloud data fusion problem into a model-based matching and alignment problem by encoding or characterizing the hybrid structure differences.

[0004] This type of method requires a preprocessing step to reduce the impact of structural differences in multi-source point cloud data, and applies optimization strategies or manual features for reconstructing the same-source point cloud to solve the structural difference problem in multi-source fusion. Its advantages are good versatility and strong predictability. However, the robustness of solving the multi-source point cloud fusion problem through geometric feature-level matching is weak, and its essence does not solve the topological structure differences of the original multi-source point cloud data, resulting in point cloud distortion, deformation and void problems still existing during the fusion process. Summary of the invention

[0005] In order to overcome the shortcomings of the prior art, the present invention provides a scene semantics driven multi-source point cloud fusion reconstruction method. First, the indoor scene is semantically segmented using a point cloud semantic segmentation network, thereby reducing the reconstruction domain of the entire indoor scene to carefully designed semantic objects, including the ground, tables, chairs, walls, etc. The robust feature matching relationship is quickly constructed using the fusion scene semantics advanced plane / line descriptors, so as to achieve the solution of the seven-degree-of-freedom transformation in multi-source point cloud reconstruction. At the same time, a ground prior registration module is proposed to solve the initial rotation matrix, overcome the large rotation transformation solution caused by the significant change in the perspective of multi-source point cloud scanning, and use the scene semantics based extended RANSAC algorithm to achieve efficient and accurate initial iterative conversion, greatly reducing the scene reconstruction reasoning delay. The present invention uses the following principles: through plane-based high-dimensional feature matching to overcome the mixed structure differences of multi-source point cloud data, to achieve robust fusion of three-dimensional maps, and at the same time, the fusion scene semantics introduces prior knowledge in the multi-source point cloud registration process, thereby skipping unnecessary iterations and further accelerating the reconstruction rate.

[0006] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0007] Step 1: Point cloud semantic segmentation;

[0008] The convolutional neural network-based point cloud semantic segmentation algorithm RandLA-Net is used to predict point-by-point semantic labels for multi-source point cloud scans;

[0009] Two rounds of filtering preprocessing operations are performed. In the first round, each semantic instance S is fitted into a plane model P(S) using the least squares method, and points that are more than a centimeters away from the plane are filtered out. In the second round, the outlier removal algorithm in the PCL library is used to further remove sparse outliers, thereby obtaining the filtered instance point cloud S.

[0010] Step 2: plane semantic feature extraction;

[0011] Step 2-1: Obtain all possible intersection lines by calculating the intersection lines of two non-parallel planes;

[0012] Step 2-2: Randomly select two intersection lines from the intersection line set and construct a descriptor using the geometric relationship between the plane and the intersection line. The constructed descriptor is a seven-dimensional feature vector d 7 ;

[0013] Step 2-3: Define a plane-based feature descriptor, that is, a pair of non-parallel planes and a line segment constitute a five-dimensional feature vector d 5 :

[0014]

[0015] Among them, L * represents the intersection line between two non-parallel planes, P * represents two non-parallel planes, where the subscript * represents an integer from 1 to 4; Represents the angle between plane or line geometric primitives, min(.) and max(.) represent the minimum and maximum values ​​between two angles respectively;

[0016] Step 2-4: Enumerate all plane / line combinations to collect a set of plane feature descriptors for the source point cloud X and the target point cloud Y:

[0017]

[0018] in, and They represent the sets of seven-dimensional and five-dimensional feature descriptors, respectively, where the subscript * represents X and Y;

[0019] Step 2-5: Fusion semantics to label each plane feature descriptor (d * ,s), where s∈S is the semantic instance label output from step 1; d * is the plane feature descriptor outputted from step 2-3; the plane semantic feature description set of the source point cloud X and the target point cloud Y is expressed as (D X ∪S X ) and (D Y ∪S Y );

[0020] Step 3: Ground prior registration module;

[0021] Step 3-1: Estimate the plane model P(G) for the point cloud instance G on the ground;

[0022] Step 3-2: Rotate the ground part of the lidar point cloud and the depth camera point cloud so that the corresponding ground is parallel to the xy plane of the geographic reference coordinate system, and calculate the initial rotation matrix according to the normal vector of the fitted ground plane model;

[0023] The cloth filter simulation algorithm CSF is used to filter out the plane model P(G) from the ground point cloud instance G; the normal vector information of the plane model P(G) is fitted in the source point cloud and the target point cloud, and the rotation matrix R is calculated using the Rodrigues function:

[0024]

[0025] Where I is the identity matrix, v is the cross product of the fitted ground point cloud plane model and the normal vector of the xy plane, s is the magnitude of vector v, c is the dot product of the fitted ground point cloud plane model and the normal vector of the xy plane, and [v] is defined as the antisymmetric cross product matrix of vector v:

[0026]

[0027] Among them, v x ,v y ,v z are the three components of vector v;

[0028] Ground registration prior rotation matrix R G By R L and R D Joint decision.

[0029] R G =R L *R D

[0030] Step 4: Point cloud reconstruction module;

[0031] It includes the following parts:

[0032] Scale estimation: for two pairs of corresponding points in the source point cloud X and the target point cloud Y and When there is a scale difference between two point clouds, the length distance of the line segment of the same geometric structure is different in the two point clouds; specifically, if there is a scale scaling factor s, the relationship between the length of the line segment between two points in the source point cloud X and the length of the corresponding line segment in the target point cloud Y is expressed as:

[0033]

[0034] Rotation and translation transformation estimation: The rotation matrix R is obtained through step 3 G At the same time, step 2 extracts the fusion semantic plane feature description set (D X ∪S X ) and (D Y ∪S Y) Using the idea similar to the RANSAC algorithm to find the corresponding relationship between the plane feature description sets;

[0035] Double label verification: A double-layer label verification mechanism is used to narrow the search space when guessing the correspondence; specifically, for the plane feature description set D of the target point cloud Y Y Each plane feature descriptor in Only consider the plane feature description set D of the source point cloud for the target point cloud X X With Planar feature descriptors with the same semantic label s;

[0036] Early inspection mechanism: using the ground prior matrix R output from step 3 G When the RANSAC algorithm has an incorrect correspondence, the iteration process is terminated early; after randomly guessing the correspondence of three pairs of plane feature descriptors, the rotation matrix of the current iteration is compared. and the rotation matrix R obtained in step 3 G The similarity check of is used to check the validity of the random hypothesis in this iteration. The time complexity of this operation is O(1):

[0037]

[0038] Preferably, a=5.

[0039] A computer program, wherein the computer program enables a computer to execute the above-mentioned multi-source point cloud fusion reconstruction method.

[0040] An electronic device comprises: a processor and a memory; the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the electronic device executes the above-mentioned multi-source point cloud fusion reconstruction method.

[0041] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the multi-source point cloud fusion reconstruction method.

[0042] A chip includes: a processor, which is used to call and run a computer program from a memory, so that a device equipped with the chip executes the above-mentioned multi-source point cloud fusion reconstruction method.

[0043] A computer program product, comprising a computer storage medium storing a computer program, wherein the computer program comprises instructions executable by at least one processor, and when the instructions are executed by the at least one processor, the multi-source point cloud fusion reconstruction method is implemented.

[0044] The beneficial effects of the present invention are as follows:

[0045] The present invention is a scene semantics-driven multi-source point cloud fusion reconstruction method, which uses a scene semantics-based multi-source point cloud fusion method to achieve robust matching correspondence and posture transformation solutions by combining high-level plane geometry primitives. This method solves the problem that traditional point correspondence-based methods require a large number of initial parameter settings and are prone to falling into local optimality. At the same time, in order to speed up the reconstruction of three-dimensional maps, an extension strategy of fusion semantics is introduced in the feature matching process, which can effectively avoid unnecessary iterative solutions, thereby speeding up the fusion speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is a framework diagram of the method of the present invention.

[0047] Figure 2 It is the element graph of the plane feature descriptor of the present invention.

[0048] Figure 3 Comparison chart of reconstruction accuracy and reconstruction delay based on point-level and feature-level reconstruction methods and the present invention in the RESSO multi-source point cloud dataset. DETAILED DESCRIPTION

[0049] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0050] The present invention utilizes the following principle: the multi-source point cloud fusion method based on scene semantics can combine semantics to reduce the huge reconstruction matching space into a fine semantic reconstruction domain, thereby achieving robust matching correspondence and posture transformation solution. It can effectively solve the local optimal problem existing in the point correspondence method. At the same time, the expansion strategy of fusion semantics is introduced in the search process, which can effectively avoid unnecessary iterative solutions, thereby accelerating the fusion speed.

[0051] In order to achieve the above tasks, the present invention adopts the following technical solutions:

[0052] A scene semantics driven multi-source point cloud fusion reconstruction method comprises the following steps:

[0053] Step 1: Point cloud semantic segmentation. Point cloud semantic segmentation refers to the semantic segmentation of multi-source point cloud data scanned by different sensor devices, removing semantic object point sets that are not important for reconstruction, such as dynamic objects such as pedestrians. Only large-area static semantic objects, namely ten common semantic objects such as tables, chairs, walls and floors, are segmented into instances with semantic labels in multi-source point clouds, and retained for further reconstruction processing in subsequent modules to ensure that the reconstruction performance of the dynamic indoor environment model can still run robustly. The present invention adopts the point cloud semantic segmentation algorithm RandLA-Net based on Convolutional Neural Networks (CNN) to predict point-by-point semantic labels for multi-source point cloud scans, in order to effectively integrate semantic labels into the reconstruction pipeline based on feature correspondence.

[0054] A large number of outliers (i.e., points that belong to an instance object but are outside the instance contour due to movement during scanning) and non-instance points that may be introduced by semantic segmentation will be generated at the edge of the semantic instance point cloud. The present invention performs two rounds of filtering preprocessing operations. In the first round of processing, a plane model P(S) is fitted using the least squares method for each semantic instance S, and points that are more than 5 cm away from the plane are filtered out. In the second round, the outlier removal algorithm in the PCL library is used to further remove sparse outliers, thereby obtaining a filtered instance point cloud S. Each semantic instance will be processed independently in subsequent steps.

[0055] Step 2: Plane semantic feature extraction: Plane semantic feature extraction defines the structural hierarchical descriptors of plane and line primitives in each semantic object point cloud, and integrates the semantic instance labels to establish the structural and semantic plane descriptors as the feature correspondence set for the subsequent point cloud reconstruction module.

[0056] This method first obtains all possible intersection lines by calculating the intersection lines of two non-parallel planes. Next, two intersection lines are randomly selected from the intersection line set, and the geometric relationship between the plane and the intersection line is used to construct a descriptor. The constructed descriptor is a seven-dimensional feature vector d 7 In addition, we introduce line segments as supplementary features to help better establish the correspondence between two point clouds and provide more information about the geometric structure of the point clouds, thereby enhancing the descriptor's ability to express low-overlapping areas. In addition to the descriptor based on seven-dimensional planes, the present invention defines another plane feature descriptor based on fewer geometric primitives, that is, a pair of non-parallel planes and a line segment constitute a five-dimensional feature vector d 5 .

[0057]

[0058] Among them, L * represents the intersection line between two non-parallel planes, P *represents two non-parallel planes, Represents the angle between plane or line geometric primitives. min(*,*) and max(*,*) represent the minimum and maximum values ​​between two angles, respectively.

[0059] A set of plane feature descriptors for the source point cloud X and the target point cloud Y are collected by enumerating all plane / line combinations.

[0060]

[0061] in, and Represents the set of feature descriptors of seven and five dimensions respectively.

[0062] Since the structure-level plane descriptor mainly encodes the geometric information of the plane, simply enumerating all combinations of planes will lead to repeated transformation redundancy. That is, different plane combinations will produce the same or very similar transformation results. This leads to repeated calculation of a large number of redundant transformations in the subsequent matching process. Therefore, it is an important strategy to integrate semantic tags into the plane feature descriptor to enhance the matching efficiency and accuracy. The present invention integrates semantics to mark each plane feature descriptor (d * ,s), where s∈S is the semantic instance label output from step 1. The planar semantic feature description set of the source point cloud X and the target point cloud Y can be expressed as (D X ∪S X ) and (D Y ∪S Y ).

[0063] Step 3: Ground prior registration module. The ground prior registration module uses semantic objects related to the ground or walls to stably predict the initial rotation matrix between multi-source point cloud scan pairs, providing strong prior knowledge for more accurate alignment and speeding up the reasoning of subsequent reconstruction modules.

[0064] This step only focuses on the semantic instance object point cloud of the ground output in step 1. First, the plane model P(G) is estimated for the point cloud instance G on the ground. Then, the ground part of the lidar point cloud and the depth camera point cloud is rotated so that the corresponding ground is parallel to the xy plane of the geographic reference coordinate system, and the initial rotation matrix is ​​calculated based on the normal vector of the fitted ground plane model. We first use the Cloth Simulation Filtering (CSF) algorithm to filter out the plane model P(G) from the ground point cloud instance G. The rotation matrix R is calculated using the Rodrigues function through the normal vector information of the fitted plane model P(G) in the source point cloud and the target point cloud.

[0065]

[0066] Where I is the identity matrix and v is the cross product of the fitted ground point cloud plane model and the normal vector of the xy plane s is the magnitude of vector v, c is the dot product of the fitted ground point cloud plane model and the normal vector of the xy plane, and [v] is defined as the antisymmetric cross product matrix of vector v.

[0067]

[0068] Among them, v x ,v y ,v z Are the three components of the vector v. The ground registration prior rotation matrix R G Can be R L and R D Joint decision.

[0069] R G =R L *R D

[0070] Step 4: Point cloud reconstruction module. The point cloud reconstruction module is to quickly find the three-dimensional matrix of the correspondence between multi-source point clouds and plane feature descriptor point sets based on the RANSAC idea.

[0071] Scale estimation. For two pairs of corresponding points in the source point cloud X and the target point cloud Y and When there is a scale difference between two point clouds, the length distance of the line segments of the same geometric structure is different in the two point clouds. Specifically, if there is a scale scaling factor s, the relationship between the length of the line segment between two points in the source point cloud X and the length of the corresponding line segment in the target point cloud Y can be expressed as:

[0072]

[0073] Rotation and translation transformation estimation. We obtain the rotation matrix R through step 3 G At the same time, step 2 extracts the fusion semantic plane feature description set (D X ∪S X ) and (D Y ∪S Y ) can use the idea similar to the RANSAC algorithm to quickly find the corresponding relationship between the plane feature description sets.

[0074] Double label verification. This method uses a double-layer label verification mechanism to narrow the search space when guessing the corresponding relationship. Specifically, for the plane feature description set D of the target point cloud Y Y Each plane feature descriptor in We only consider the planar feature description set D of the source point cloud for the target point cloud X X With This strategy greatly improves the probability of correct prediction by using flat feature descriptors with the same semantic label s.

[0075] Early inspection mechanism. The early inspection mechanism refers to the use of the ground prior matrix R output in step 3 G When the RANSAC algorithm has an incorrect correspondence, the iteration process is terminated early, thereby reducing unnecessary calculations. After randomly guessing the correspondence of three pairs of plane feature descriptors, this method compares the rotation matrix of the current iteration. and the rotation matrix R obtained in step 3 G The similarity of is used to quickly check the validity of the random hypothesis in this iteration, and the time complexity of this operation is O(1).

[0076]

[0077] Example:

[0078] There are 3 drawings in this invention, please refer to Figure 1 As shown, the specific steps of the present invention are as follows:

[0079] Step 1: Point cloud semantic segmentation. Point cloud semantic segmentation refers to the semantic segmentation of multi-source point cloud data scanned by different sensor devices, removing semantic object point sets that are not important for reconstruction, such as dynamic objects such as pedestrians. Only large-area static semantic objects, namely ten common semantic objects such as tables, chairs, walls and floors, are segmented into instances with semantic labels in multi-source point clouds, and retained for further reconstruction processing in subsequent modules to ensure that the reconstruction performance of the dynamic indoor environment model can still run robustly. The present invention adopts the point cloud semantic segmentation algorithm RandLA-Net based on Convolutional Neural Networks (CNN) to predict point-by-point semantic labels for multi-source point cloud scans, in order to effectively integrate semantic labels into the reconstruction pipeline based on feature correspondence.

[0080] Step 2: Plane semantic feature extraction: Plane semantic feature extraction defines the structural hierarchical descriptors of plane and line primitives in each semantic object point cloud, and integrates the semantic instance labels to establish the structural and semantic plane descriptors as the feature correspondence set for the subsequent point cloud reconstruction module.

[0081] Step 3: Ground prior registration module. The ground prior registration module uses semantic objects related to the ground or walls to stably predict the initial rotation matrix between multi-source point cloud scan pairs, providing strong prior knowledge for more accurate alignment and speeding up the reasoning of subsequent reconstruction modules.

[0082] Step 4: Point cloud reconstruction module. The point cloud reconstruction module is to quickly find the three-dimensional matrix of the correspondence between multi-source point clouds and plane feature descriptor point sets based on the RANSAC idea.

[0083] Scale estimation. For two pairs of corresponding points in the source point cloud X and the target point cloud Y and When there is a scale difference between two point clouds, the length distance of the line segments of the same geometric structure is different in the two point clouds. Specifically, if there is a scale scaling factor s, the relationship between the length of the line segment between two points in the source point cloud X and the length of the corresponding line segment in the target point cloud Y can be expressed as:

[0084]

[0085] Rotation and translation transformation estimation. The rotation matrix R is obtained through step 3 G At the same time, step 2 extracts the fusion semantic plane feature description set (D X ∪S X ) and (D Y ∪S Y ) can use the idea similar to the RANSAC algorithm to quickly find the corresponding relationship between the plane feature description sets.

[0086] Double label verification. This method uses a double-layer label verification mechanism to narrow the search space when guessing the correspondence. Specifically, for the plane feature description set D of the target point cloud Y Y Each plane feature descriptor in Only consider the plane feature description set D of the source point cloud for the target point cloud X X With This strategy greatly improves the probability of correct prediction by using flat feature descriptors with the same semantic label s.

[0087] Early inspection mechanism. The early inspection mechanism refers to the use of the ground prior matrix R output in step 3 G When the RANSAC algorithm has an incorrect correspondence, the iteration process is terminated early, thereby reducing unnecessary calculations. After randomly guessing the correspondence of three pairs of plane feature descriptors, this method compares the rotation matrix of the current iteration. and the rotation matrix R obtained in step 3 G The similarity of is used to quickly check the validity of the random hypothesis in this iteration, and the time complexity of this operation is O(1).

[0088]

[0089] Figure 2The descriptor is a feature map of the plane feature descriptor. Its descriptor is constructed by combining or encoding simple plane intersection distances and plane normals, sizes, positions, and other elements. We found that a key challenge in processing multi-source point clouds of different scales is that the shortest distances between plane intersections are no longer equal, which makes it infeasible to directly use this metric for plane feature matching for descriptor construction. This is particularly true in the construction of multi-source point cloud plane descriptors because different data sources (such as lidar and depth cameras) capture the same physical space at different scales.

[0090] Figure 3 The figure is a comparison of the reconstruction accuracy and reconstruction delay of the reconstruction methods based on point level and feature level and the present invention in the RESSO multi-source point cloud data set. It can be observed from the figure that the method based on feature level correspondence has an average improvement of 123.7% in the reconstruction recall rate on the reconstruction accuracy evaluation indicators (RR, RRE, RTE) compared with the method based on point level correspondence, and the relative rotation error and translation error have decreased by 54.7% and 51.5% on average, respectively. The present invention and the best-performing feature-level reconstruction method have an improvement of 10.8% in the reconstruction recall rate on the reconstruction indicators (RR, RRE, RTE), and the relative rotation error and translation error have decreased by 10.0% and 46.1%, respectively. The reconstruction delay has been improved by about 1.4 times.

Claims

1. A scene semantics driven multi-source point cloud fusion reconstruction method, characterized in that: The steps include: Step 1: Point cloud semantic segmentation; The convolutional neural network-based point cloud semantic segmentation algorithm RandLA-Net is used to predict point-by-point semantic labels for multi-source point cloud scans; Two rounds of filtering preprocessing operations are performed. In the first round, each semantic instance S is fitted into a plane model P(S) using the least squares method, and points that are more than a centimeters away from the plane are filtered out. In the second round, the outlier removal algorithm in the PCL library is used to further remove sparse outliers, thereby obtaining the filtered instance point cloud S. Step 2: plane semantic feature extraction; Step 2-1: Obtain all possible intersection lines by calculating the intersection lines of two non-parallel planes; Step 2-2: Randomly select two intersection lines from the intersection line set and construct a descriptor using the geometric relationship between the plane and the intersection line. The constructed descriptor is a seven-dimensional feature vector d 7 ; Step 2-3: Define a plane-based feature descriptor, that is, a pair of non-parallel planes and a line segment constitute a five-dimensional feature vector d 5 : Among them, L * represents the intersection line between two non-parallel planes, P * represents two non-parallel planes, where the subscript * represents an integer from 1 to 4; Represents the angle between plane or line geometric primitives, min(.) and max(.) represent the minimum and maximum values ​​between two angles respectively; Step 2-4: Enumerate all plane / line combinations to collect a set of plane feature descriptors for the source point cloud X and the target point cloud Y: in, and They represent the sets of seven-dimensional and five-dimensional feature descriptors, respectively, where the subscript * represents X and Y; Step 2-5: Fusion semantics to label each plane feature descriptor (d * ,s), where s∈S is the semantic instance label output from step 1; d * is the plane feature descriptor outputted from step 2-3; the plane semantic feature description set of the source point cloud X and the target point cloud Y is expressed as (D X ∪S X ) and (D Y ∪S Y ); Step 3: Ground prior registration module; Step 3-1: Estimate the plane model P(G) for the point cloud instance G on the ground; Step 3-2: Rotate the ground part of the lidar point cloud and the depth camera point cloud so that the corresponding ground is parallel to the xy plane of the geographic reference coordinate system, and calculate the initial rotation matrix according to the normal vector of the fitted ground plane model; The cloth filter simulation algorithm CSF is used to filter out the plane model P(G) from the ground point cloud instance G; the normal vector information of the plane model P(G) is fitted in the source point cloud and the target point cloud, and the rotation matrix R is calculated using the Rodrigues function: Where I is the identity matrix, v is the cross product of the fitted ground point cloud plane model and the normal vector of the xy plane, s is the magnitude of vector v, c is the dot product of the fitted ground point cloud plane model and the normal vector of the xy plane, and [v] is defined as the antisymmetric cross product matrix of vector v: Among them, v x ,v y ,v z are the three components of vector v; Ground registration prior rotation matrix R G By R L and R D Joint decision making; R G =R L *R D Step 4: Point cloud reconstruction module; It includes the following parts: Scale estimation: for two pairs of corresponding points in the source point cloud X and the target point cloud Y and When there is a scale difference between two point clouds, the length distance of the line segment of the same geometric structure is different in the two point clouds; specifically, if there is a scale scaling factor s, the relationship between the length of the line segment between two points in the source point cloud X and the length of the corresponding line segment in the target point cloud Y is expressed as: Rotation and translation transformation estimation: The rotation matrix R is obtained through step 3 G At the same time, step 2 extracts the fusion semantic plane feature description set (D X ∪S X ) and (D Y ∪S Y ) Using the idea similar to the RANSAC algorithm to find the corresponding relationship between the plane feature description sets; Double label verification: A double-layer label verification mechanism is used to narrow the search space when guessing the correspondence; specifically, for the plane feature description set D of the target point cloud Y Y Each plane feature descriptor in Only consider the plane feature description set D of the source point cloud for the target point cloud X X With Planar feature descriptors with the same semantic label s; Early inspection mechanism: using the ground prior matrix R output from step 3 G When the RANSAC algorithm has an incorrect correspondence, the iteration process is terminated early; after randomly guessing the correspondence of three pairs of plane feature descriptors, the rotation matrix of the current iteration is compared. and the rotation matrix R obtained in step 3 G The similarity check of is used to check the validity of the random hypothesis in this iteration. The time complexity of this operation is O(1):

2. The scene semantics-driven multi-source point cloud fusion reconstruction method according to claim 1, characterized in that: Said a=5.

3. A computer program, characterized in that The computer program enables a computer to execute the method according to any one of claims 1 to 2.

4. An electronic device, characterized in that: include: Processor and memory; The memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the electronic device executes the method as claimed in any one of claims 1 to 2.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 2 is implemented.

6. A chip, characterized in that: include: A processor, configured to call and run a computer program from a memory, so that a device equipped with the chip executes a method as claimed in any one of claims 1 to 2.

7. A computer program product, characterized in that The computer program product comprises a computer storage medium storing a computer program, wherein the computer program comprises instructions executable by at least one processor, and when the instructions are executed by the at least one processor, the method according to any one of claims 1 to 2 is implemented.

Images