Point cloud semantic information assisted mixed reality virtual-real registration method

Abstract

The present invention provides a point cloud semantic information assisted mixed reality virtual-reality registration method. The present invention uses a mixed reality device to collect, classify and identify three-dimensional data of an indoor environment, adds advanced semantic information to the mixed reality, and realizes a simple understanding of the surrounding environment by the mixed reality device; the point cloud registration method is used to perform accurate registration between the real world and the digital model, without the need to segment and extract features of the point cloud, thereby realizing the automation of the virtual-reality registration and ensuring the registration accuracy and robustness; the present invention is suitable for structural point cloud data of indoor scenes to participate in the registration calculation, saving a lot of computing overhead and greatly improving the virtual-reality model registration speed.

Description

Technical Field

[0001] The invention relates to a point cloud semantic information assisted mixed reality virtual-real registration method. Background Art

[0002] Mixed Reality (MR) technology combines virtual digital information with the real world, allowing users to not only feel the actual objects in the real environment, but also obtain digital information in the virtual environment. At a time when mixed reality is developing rapidly, maintaining accurate registration between the real world and the digital model is one of the most critical technologies for mixed reality. Common mixed reality devices support manual registration, surface-based registration, and QR code-based registration. Among them, manual registration is to manually scale, translate, and rotate the model to align with the real world, which requires high operator quality, large registration errors, and is uncontrollable; surface-based registration is limited by the detection range of the mixed reality device to the surrounding environment, and can only guarantee local registration effects; QR code-based registration requires pasting QR codes in digital models and the real world. The preliminary preparation work is relatively complicated, and the paper QR codes attached to the real world are difficult to retain for a long time in a dynamic and complex environment.

[0003] With the expansion of emerging application scenarios of 3D vision, point clouds have received widespread attention. Conventional feature point methods are used for matching and tracking, because it is necessary to calculate the descriptors corresponding to each feature point and then match the descriptors. A large number of descriptors will cause the system processing process to be relatively slow and the matching process is also very time-consuming. The subsequent registration process requires mixed reality devices to perform a large number of calculations such as translation, rotation, and rendering, which will also cause a large amount of system memory usage, easily causing image display lags, and even navigation system freezes and paralysis. Therefore, searching for the point cloud of the target object in the huge amount of data in the environmental point cloud is prone to problems such as large amount of calculation and high time complexity, which is difficult to achieve in practical applications and does not meet the real-time requirements of highly dynamic environments. Summary of the invention

[0004] The purpose of the present invention is to provide a point cloud semantic information assisted mixed reality virtual-real registration method.

[0005] In order to solve the above problems, the present invention provides a point cloud semantic information assisted mixed reality virtual-real registration method, comprising:

[0006] Step 1: Use mixed reality devices to obtain sampling space point cloud data;

[0007] Step 2: Preprocess the sampled space point cloud data to obtain indoor real-time point cloud data;

[0008] Step 3: Automatically identify the semantics of the indoor real-life point cloud to obtain a point cloud set with semantic information;

[0009] Step 4: Process the BIM model to obtain a virtual model point cloud set with label information;

[0010] Step 5: Match the point cloud set with semantic information with the virtual model point cloud set for virtual and real scene matching.

[0011] Furthermore, in the above method, step 1: obtaining point cloud data of the sampling space using a mixed reality device includes:

[0012] Step 1.1: Use the mixed reality device to scan the indoor environment and obtain the sampling space mapping grid data;

[0013] Step 1.2: Use the Poisson disk sampling method to convert the acquired spatial mapping grid data into point cloud data.

[0014] Furthermore, in the above method, step 1.2: using the Poisson disk sampling method to convert the acquired spatial mapping grid data into point cloud data includes:

[0015] Step 1.2.1: According to the indoor area S, the number of point clouds after sampling is set to N,

[0016] N = K1*S;

[0017] Among them, K1 is the proportionality coefficient, which is a constant obtained by observing multiple sets of indoor data collected;

[0018] Step 1.2.2: Randomly select a sample point p in the grid data sample i , with sample point p i As the center, set r as the sample point p according to the side length of the mesh triangulation network i The enclosing radius generates a sampling point q within the circle range of (r,2r) i ;

[0019] Step 1.2.3: Set the sample point p i The number of sampling times is k1;

[0020] Step 1.2.4: Determine the sampling point q i The distance to other points, when the distance is less than r, the sampling point q is discarded i , and regenerate a sampling point q within the circle range of (r,2r) i , until the sampling point q i The distances to other points are all greater than r;

[0021] Step 1.2.5: If no new available point is found within the circle range of r, 2r after k1 samplings, the sample point p is abandoned. i Sampling of

[0022] Step 1.2.6: Continuously update the sampling points and repeat the above steps so that the sampling ring fills the entire sampling space until the number of sampling points reaches N, and the sampling space cloud data PointCloud_1 is obtained.

[0023] Furthermore, in the above method, step 2: preprocessing the sampled space point cloud data to obtain indoor real-world point cloud data includes:

[0024] Step 2.1: Use point cloud direct filtering to filter the non-indoor area in the sampled space point cloud data PointCloud_1, and retain the point cloud data PointCloud_2 of the original indoor area;

[0025] Step 2.2: Use point cloud statistical filtering to remove the sparse outlier noise points in the original indoor area point cloud PointCloud_2 caused by the influence of ambient light, and obtain the indoor real-world point cloud PointCloud_3.

[0026] Furthermore, in the above method, step 2.1: filtering the non-indoor area in the sampled space point cloud data PointCloud_1 by using point cloud through filtering, and retaining the point cloud data PointCloud_2 of the original indoor area, includes:

[0027] Step 2.1.1: Input the sampling space point cloud data PointCloud_1;

[0028] Step 2.1.2: Set the value ranges of x, y, and z to (min_x, max_x), (min_y, max_y), and (min_z, max_z) respectively.

[0029] Step 2.1.3: Eliminate the point cloud outside the range of x, y, and z values ​​to obtain the original indoor area point cloud data PointCloud_2;

[0030] Furthermore, in the above method, step 2.2: using point cloud statistical filtering to remove sparse outlier noise points in the original indoor area point cloud PointCloud_2 caused by the influence of ambient light, includes:

[0031] Step 2.2.1: Calculate the average distance between each point in the original indoor area point cloud PointCloud_2 and its K adjacent points:

[0032]

[0033] Step 2.2.2: Find the mean μ and standard deviation σ of all average distances;

[0034] Step 2.2.3: Calculate the distance threshold dmax,

[0035] dmax=μ+S'*σ,

[0036] Among them, S' is the proportional coefficient, which is a constant determined by the number of adjacent points K; the larger the S' value, the fewer points are lost;

[0037] Step 2.2.4: Traverse all points in the point set of the point cloud data PointCloud_2 of the original indoor area and calculate the average distance to its K adjacent points. When the distance is greater than dmax, the point is regarded as an outlier and deleted from the point cloud data, and finally the indoor real-world point cloud data PointCloud_3 is obtained.

[0038] Furthermore, in the above method, step 3: automatically recognize the semantics of the point cloud data PointCloud_3 of the indoor real-life point cloud to obtain a point cloud set PointCloud_4 with semantic information, including:

[0039] Step 3.1: Preset point cloud semantic elements: Ceiling, Ground, Wall, Column, Beam and Background;

[0040] Step 3.2: According to the preset point cloud deep learning model, automatically perform semantic analysis on the indoor real-life point cloud data PointCloud_3;

[0041] The preset point cloud deep learning model is used to automatically identify the preset point cloud semantic elements. The input is the indoor real-world point cloud data PointCloud_3; the output is the point cloud set PointCloud_4 with semantic information. The point cloud set PointCloud_4 with semantic information contains: the semantic elements corresponding to the single point and the point cloud coordinate information x i ,y i , z i .

[0042] Furthermore, in the above method, step 4: performing BIM model preprocessing to obtain a virtual model point cloud set PointCloud_5 with label information includes:

[0043] Step 4.1: According to the category information of the components in the BIM model, obtain the floor components, wall components, column components and beam components;

[0044] Step 4.2: For the floor components obtained, according to the floor elevation, define the upper surface of the floor with a lower elevation as the ground, and assign the ground a "Ground" label; define the lower surface of the floor with a higher elevation as the ceiling, and assign the ceiling a "Ceiling" label;

[0045] Step 4.3: For the obtained wall components, obtain the inward-facing surface of the wall through the Interior type attribute in the type attribute of Function, and assign the "Wall" label to the inward-facing surface;

[0046] Step 4.4: For the obtained column components, assign a "Column" label to each column surface;

[0047] Step 4.5: For the obtained beam components, assign a "Beam" label to each beam surface;

[0048] Step 4.6: For all faces with label information, triangulate to form a corresponding mesh, traverse all triangles of the mesh and obtain all vertices to obtain a virtual model point cloud set PointCloud_5 with label information.

[0049] Furthermore, in the above method, step 5: matching the point cloud set with semantic information with the virtual model point cloud set in a virtual and real scene includes:

[0050] Step 5.1: Extract the planar point clouds with point cloud semantic elements of "Ceiling" and "Ground" from the point cloud set PointCloud_4 with semantic information, and extract the planar point clouds with corresponding label information of "Ceiling" and "Ground" from the point cloud set PointCloud_5;

[0051] Step 5.2: Taking the surface point cloud with semantic elements of “Ceiling” and “Ground” in the point cloud set PointCloud_4 with semantic information as the reference, align the “Ceiling” and “Ground” point clouds in the virtual model point cloud set PointCloud_5 with label information to the “Ceiling” and “Ground” point cloud planes in the point cloud set PointCloud_4 with semantic information, respectively, so that the scale of the virtual model is consistent with that of the display scene, and obtain the scale scaling parameter Scale;

[0052] Step 5.3: Extract the planar point cloud with the point cloud semantic element "Wall" from the point cloud set PointCloud_4 with semantic information, and extract the planar point cloud with the label information "Wall" from the virtual model point cloud set PointCloud_5 with label information;

[0053] Step 5.4: Based on the centroid point set of the surface point cloud extracted from the two point cloud sets, the RANSAC method is used to achieve rapid coarse registration of the virtual and real scenes;

[0054] Step 5.5: Use the point cloud set PointCloud_4 with semantic information as the reference point cloud, and the virtual model point cloud set PointCloud_5 with label information as the point cloud to be registered, and use the ICP algorithm to register the two point cloud sets;

[0055] Step 5.6: Use the Scale parameter obtained in step 5.2 to scale the BIM model, use the rigid body transformation matrix H1 obtained in step 5.4 to achieve rapid coarse alignment between the BIM model and the real-world scenic spot cloud, and finally use the rigid body transformation matrix H2 obtained in step 5.5 to achieve precise alignment between the BIM model and the real-world scenic spot cloud, thereby completing the virtual-real scene alignment.

[0056] Furthermore, in the above method, step 5.4: based on the centroid point set of the planar point cloud extracted from the two point cloud sets, using the RANSAC method to achieve rapid coarse registration of the virtual and real scenes, includes:

[0057] Step 5.4.1: Calculate the centroid of each planar point cloud with the semantic element "Wall" in the point cloud set PointCloud_4 with semantic information, and construct the point set Points1 = {point1, point2, ..., point n}; Calculate the centroid of the surface point cloud with the label information "Wall" in the virtual model point cloud set PointCloud_5 with label information, and form a point set Points2 = {point1, point2, ..., point n};

[0058] Step 5.4.2: Randomly select 3 points in the point set Points1, randomly select 3 corresponding point pairs in the point set Points2, and solve the rigid body transformation matrix;

[0059] Step 5.4.3: Calculate the distance error of the remaining point pairs under the rigid body transformation matrix obtained in step 5.4.2. If the distance error of one of the point pairs is less than the set error threshold C1, then the point is an in-sample point, otherwise it is an out-sample point, and the number of in-sample points is counted;

[0060] Step 5.4.4: Set the upper limit of the number of generations to k1, repeat the above steps until the upper limit of the number of generations is k1, count the number of points inside the samples under different rigid body transformation models, and obtain the rigid body transformation matrix H1 when the number of points inside the sample is the largest;

[0061] Step 5.5: Using the point cloud set PointCloud_4 with semantic information as the reference point cloud and the virtual model point cloud set PointCloud_5 with label information as the point cloud to be registered, the point cloud set PointCloud_4 and the virtual model point cloud set PointCloud_5 are registered using the ICP algorithm, including:

[0062] Step 5.5.1: Use the KD tree nearest neighbor search algorithm to calculate the corresponding close points of each point in the virtual model point cloud set PointCloud_5 in the point cloud set PointCloud_4 to obtain the corresponding close point pairs;

[0063] Step 5.5.2: Calculate the rigid body transformation that minimizes the average distance between the corresponding close point pairs and obtain the rigid body transformation matrix;

[0064] Step 5.5.3: Use the rigid body transformation matrix obtained in step 5.5.2 to perform spatial transformation on the point cloud set PointCloud_4 to obtain a new transformed point cloud set PointCloud_6;

[0065] Step 5.5.4: If the new transformed point cloud PointCloud_6 and the point cloud set PointCloud_4 satisfy that the average distance between the corresponding close point pairs of the two point clouds is less than the given threshold C2, or the number of iterations reaches the set maximum value k2, then stop the iterative calculation;

[0066] Among them, if the average distance between the two point clouds after registration is greater than the given threshold C2, the point clouds with the point cloud semantic element "Column" are deleted from the virtual model point cloud set PointCloud_5 and the point cloud set PointCloud_4, and steps 5.5.1 to 5.5.4 are repeated; if at this time the average distance between the corresponding near point pairs after the new transformed point cloud PointCloud_6 and the point cloud set PointCloud_4 are still greater than the given threshold C2, the point clouds with the point cloud semantic element "Beam" are deleted from the virtual model point cloud set PointCloud_5 and the point cloud set PointCloud_4, and steps 5.5.1 to 5.5.4 are repeated; if at this time the average distance between the corresponding near point pairs after the new transformed point cloud PointCloud_6 and the point cloud set PointCloud_4 are still greater than the given threshold C2, then the ratio ratio of the average distance between the corresponding near point pairs of the new transformed point cloud PointCloud_6 and the point cloud set PointCloud_4 that is less than the given threshold C2 is counted, and the result is returned to the user.

[0067] Compared with the prior art, the present invention uses mixed reality equipment to collect, classify and identify three-dimensional data of indoor environments, adds high-level semantic information in mixed reality, and realizes a simple understanding of the surrounding environment by mixed reality devices; uses point cloud registration methods to accurately align the real world and digital models, without the need to segment and extract features from point clouds, thereby realizing the automation of virtual-reality registration and ensuring registration accuracy and robustness; the present invention is suitable for structural point cloud data of indoor scenes to participate in registration calculations, saving a lot of computing overhead and greatly improving the speed of virtual-reality model registration.

[0068] The present invention aims to enrich the semantic information of real scenes collected by mixed reality devices with the help of deep learning methods, and then screen the corresponding surface structure data in the real scenes and models to participate in the registration calculation, thereby ensuring accuracy and robustness while saving computing overhead, greatly improving the registration speed, and providing a solution for fast and accurate registration between the real world and digital models. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 is a flow chart of a method for point cloud semantic information-assisted mixed reality virtual-real registration according to an embodiment of the present invention;

[0070] Figure 2 is a schematic diagram of point cloud data of a sampling space before preprocessing according to an embodiment of the present invention;

[0071] Figure 3 is a schematic diagram of preprocessing point cloud data of a sampling space according to an embodiment of the present invention;

[0072] Figure 4 It is a schematic diagram of virtual-real scene registration according to an embodiment of the present invention. DETAILED DESCRIPTION

[0073] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0074] like Figures 1 to 4 As shown, Figure 2 is a schematic diagram of point cloud data of a sampling space before preprocessing according to an embodiment of the present invention; Figure 3 is a schematic diagram of preprocessing point cloud data of a sampling space according to an embodiment of the present invention; Figure 4 3 is a schematic diagram of virtual-real scene registration according to an embodiment of the present invention, in which the white part represents the BIM model and the black part represents the real-world point cloud.

[0075] The present invention provides a point cloud semantic information assisted mixed reality virtual-real registration method, comprising:

[0076] Step 1: Use mixed reality devices to obtain sampling space point cloud data;

[0077] Step 1.1: Use the HoloLens head-mounted device to scan the indoor environment and obtain the sampling space mapping grid data;

[0078] Step 1.2: Use the Poisson disk sampling method to convert the acquired spatial mapping grid data into point cloud data;

[0079] Step 1.2.1: According to the indoor area S, the number of point clouds after sampling is set to N,

[0080] N = K1*S;

[0081] Among them, K1 is the proportionality coefficient, which is a constant obtained by observing multiple sets of indoor data collected;

[0082] Here, the number of point clouds after sampling can be set to N according to the indoor area S. After observing multiple sets of indoor data collected, K1=25000 is set, and we have:

[0083] N = K1*S = 25000*S;

[0084] For example, the laboratory area is 50m 2 For room 0, the number of point clouds after sampling is N = 25000*50 = 1250000;

[0085] Step 1.2.2: Randomly select a sample point p in the grid data sample i , with p i As the center, set r as point p according to the side length of the mesh triangulation network i The enclosing radius generates a sampling point q within the circle range of (r,2r) i ;

[0086] Here, a sample point p is randomly selected in the grid data sample i , with p i As the center, set r = 0.015m as point p i Maximum enclosing radius, generating a sampling point q within the circle range of (0.015m, 0.03m) i ;

[0087] Step 1.2.3: Set the sample point p i The number of sampling times is k1;

[0088] Here, the sample point p can be set i The number of samplings is 30;

[0089] Step 1.2.4: Determine the sampling point q i The distance to other points, when the distance is less than r, the sampling point q is discardedi , and regenerate a sampling point q within the circle range of (r,2r) i , until the sampling point q i The distances to other points are all greater than r;

[0090] Here, the sampling point q can be determined i The distance to other points is less than 0.015m, and the sampling point is discarded. i , and regenerate a sampling point q within the circle range of (0.015m, 0.03m) i , until the sampling point q i The distance to other points must be greater than 0.015m;

[0091] Step 1.2.5: If no new available point is found within the circle of (r, 2r after k1 samplings, the sample point p is abandoned. i Sampling of

[0092] Here, if no new available point is found within the circle range of (0.015m, 0.03m) after 30 samplings, the sample point p is abandoned. i Sampling of

[0093] Step 1.2.6: Continuously update the sampling points and repeat the above steps to make the sampling ring fill the entire sampling space until the number of sampling points reaches N, and the indoor real-time point cloud data PointCloud_1 is obtained.

[0094] Here, the sampling points are continuously updated and the above steps are repeated so that the sampling ring fills the entire sampling space until the number of sampling points reaches 1,250,000, and the sampling space point cloud data PointCloud_1 is obtained.

[0095] Step 2: Preprocess the sampled space point cloud data to obtain indoor real-time point cloud data;

[0096] Step 2.1: Use point cloud direct filtering to filter the non-indoor area in the sampled space point cloud data PointCloud_1, and retain the point cloud data PointCloud_2 of the original indoor area;

[0097] Step 2.1.1: Input the sampling space point cloud data PointCloud_1;

[0098] Step 2.1.2: Set the value ranges of x, y, and z to (min_x, max_x), (min_y, max_y), and (min_z, max_z) respectively;

[0099] Here, the Cartesian coordinate system with the HoloLens depth sensor as the origin is used to set the x, y, and z value ranges of this sampling space to (-4m, 6m), (-2.5m, 2.5m), and (-1.7m, 1.5m);

[0100] Step 2.1.3: Eliminate the point cloud outside the range of x, y, and z values ​​to obtain the original indoor area point cloud data PointCloud_2;

[0101] For example, the number of points in the point cloud PointCloud_2 after straight-through filtering is 1210641;

[0102] Step 2.2: Use point cloud statistical filtering to remove sparse outlier noise points in the original indoor area point cloud PointCloud_2 caused by ambient light;

[0103] Step 2.2.1: Calculate the average distance between each point in the original indoor area point cloud PointCloud_2 and its K adjacent points

[0104]

[0105] Here, the average distance between each point and the eight adjacent points in the point cloud data PointCloud_2 of the original indoor area can be calculated.

[0106] Step 2.2.2: Find the mean μ and standard deviation σ of all average distances;

[0107] For example, find the mean μ = 0.016 and standard deviation σ = 0.0019 of all average distances;

[0108] Step 2.2.3: Calculate the distance threshold dmax,

[0109] dmax=μ+S'*σ,

[0110] Among them, S' is the proportional coefficient, which is a constant determined by the number of adjacent points K; the larger the S' value, the fewer points are lost.

[0111] Here, the distance threshold dmax can be calculated, where the proportional coefficient S'=2 is set, that is, if the distance of a point exceeds the average distance by more than two standard deviations, it is marked as an outlier, then dmax=0.016+2*0.0019=0.0198m;

[0112] Step 2.2.4: Traverse all points in the point set of the original indoor area point cloud data PointCloud_2 and calculate the average distance to its K adjacent points. When the distance is greater than dmax, the point is regarded as an outlier and deleted from the point cloud data, and finally the processed indoor real-world point cloud data PointCloud_3 is obtained;

[0113] Here, the processed indoor real-world point cloud data PointCloud_3 can be finally obtained, and the number of points in the point cloud PointCloud_3 after filtering is statistically 1,164,953.

[0114] Step 3: Automatically recognize the semantics of point clouds of indoor real-life point cloud data to obtain a point cloud set with semantic information;

[0115] Step 3.1: Preset point cloud semantic elements: Ceiling, Ground, Wall, Column, Beam and BackGround;

[0116] Among them, the ceiling, floor, wall, column, and beam are building structure data. The relative position of this type of data is stable in space and can be used as a fixed reference in the virtual-real registration process. For movable objects such as indoor facilities, they are uniformly classified as background and no longer classified in detail.

[0117] Step 3.2: According to the preset point cloud deep learning model, automatically perform semantic analysis on the indoor real-life point cloud data PointCloud_3;

[0118] The preset point cloud deep learning model is used to automatically identify the preset point cloud semantic elements. The input is the indoor real-world point cloud data PointCloud_3; the output is the point cloud set PointCloud_4 with semantic information. The point cloud set PointCloud_4 with semantic information contains: the semantic elements corresponding to the single point and the point cloud coordinate information (x i ,y i , z i ).

[0119] Step 4: Process the BIM model to obtain a virtual model point cloud set with label information;

[0120] Step 4.1: According to the category information of the components in the BIM model, obtain the floor components, wall components, column components and beam components;

[0121] Here, in the Revit software, the components of the categories of floor, wall, column and beam in the BIM model are filtered through the class filter (classFilter);

[0122] Step 4.2: For the obtained floor components, according to the floor elevation, define the upper surface of the floor with a lower elevation as the ground, and assign the ground a "Ground" label; define the lower surface of the floor with a higher elevation as the ceiling, and assign the ceiling a "Ceiling" label;

[0123] Step 4.3: For the obtained wall components, obtain the inward-facing surface of the wall through the Interior type attribute in the type attribute of Function, and assign the "Wall" label to the inward-facing surface;

[0124] Step 4.4: For the obtained column components, assign a "Column" label to each column surface;

[0125] Step 4.5: For the obtained beam components, assign a "Beam" label to each beam surface;

[0126] Step 4.6: For all faces with label information, triangulate to form a corresponding mesh, traverse all triangles of the mesh and obtain all vertices to obtain a virtual model point cloud set PointCloud_5 with label information.

[0127] Step 5: Match the point cloud set with semantic information and the virtual model point cloud set with virtual and real scenes;

[0128] Step 5.1: Extract the planar point clouds with point cloud semantic elements of "Ceiling" and "Ground" from the point cloud set PointCloud_4 with semantic information, and extract the planar point clouds with corresponding label information of "Ceiling" and "Ground" from the point cloud set PointCloud_5;

[0129] Step 5.2: Taking the surface point cloud with semantic elements of “Ceiling” and “Ground” in the point cloud set PointCloud_4 with semantic information as the reference, align the “Ceiling” and “Ground” point clouds in the virtual model point cloud set PointCloud_5 with label information to the “Ceiling” and “Ground” point cloud planes in the point cloud set PointCloud_4 with semantic information, respectively, so that the scale of the virtual model is consistent with that of the display scene, and obtain the scale scaling parameter Scale=3.5;

[0130] Step 5.3: Extract the planar point cloud with the point cloud semantic element "Wall" from the point cloud set PointCloud_4 with semantic information, and extract the planar point cloud with the label information "Wall" from the virtual model point cloud set PointCloud_5 with label information;

[0131] Step 5.4: Based on the centroid point set of the surface point cloud extracted from the two point cloud sets, the RANSAC method is used to achieve rapid coarse registration of the virtual and real scenes;

[0132] Step 5.4.1: Calculate the centroid of each planar point cloud with the semantic element "Wall" in the point cloud set PointCloud_4 with semantic information, and construct the point set Points1 = {point1, point2, ..., point n}; Calculate the centroid of the surface point cloud with the label information "Wall" in the virtual model point cloud set PointCloud_5 with label information, and form a point set Points2 = {point1, point2, ..., point n};

[0133] For example, for the experimental room, the centroids of the surface point clouds with the point cloud semantic element "Wall" in PointCloud4 are calculated to form the point set Points1 = {point1, point2, point3, point4, point5, point6}, and the centroids of the surface point clouds with the "Wall" label information in PointCloud5 are calculated to form the point set Points2 = {point1, point2, point3, point4, point5, poibt6};

[0134] Step 5.4.2: Randomly select 3 points in the point set Points1, randomly select 3 corresponding point pairs in the point set Points2, and solve the rigid body transformation matrix;

[0135] Step 5.4.3: Calculate the distance error of the remaining point pairs under the rigid body transformation matrix obtained in step 5.4.2. If the distance error of one of the point pairs is less than the set error threshold C1, then the point is an in-sample point, otherwise it is an out-sample point, and the number of in-sample points is counted;

[0136] Here, the distance error of the remaining point pairs under the rigid body transformation matrix obtained in step 5.4.2 can be calculated. If the distance error of one of the point pairs is less than the set error threshold of 0.02, then the point is an in-sample point, otherwise it is an out-sample point, and the number of the former is counted;

[0137] Step 5.4.4: Set the upper limit of the number of generations to k1, repeat the above steps until the upper limit of the number of generations is k1, count the number of points inside the samples under different rigid body transformation models, and obtain the rigid body transformation matrix H1 when the number of points inside the sample is the largest;

[0138] Here, the upper limit of the number of iterations is set to 100, and the above steps are repeated until the upper limit of the number of iterations is reached. The number of sample internal points under different rigid body transformation models is counted, and the transformation matrix with the largest number of sample internal points is obtained.

[0139]

[0140] Step 5.5: Take the point cloud set PointCloud_4 with semantic information as the reference point cloud, and the virtual model point cloud set PointCloud_5 with label information as the point cloud to be registered, and use the ICP algorithm to register the point cloud set PointCloud_4 and the virtual model point cloud set PointCloud_5;

[0141] Step 5.5.1: Use the KD tree nearest neighbor search algorithm to calculate the corresponding close points of each point in the virtual model point cloud set PointCloud_5 in the point cloud set PointCloud_4 to obtain the corresponding close point pairs;

[0142] Step 5.5.2: Calculate the rigid body transformation that minimizes the average distance between the corresponding close point pairs and obtain the rigid body transformation matrix;

[0143] Step 5.5.3: Use the rigid body transformation matrix obtained in step 5.5.2 to perform spatial transformation on the point cloud set PointCloud_4 to obtain a new transformed point cloud PointCloud_6;

[0144] Step 5.5.4: If the new transformed point cloud PointCloud_6 and the point cloud set PointCloud_4 satisfy that the average distance between the corresponding close point pairs of the two point clouds is less than the given threshold C2, or the number of iterations reaches the set maximum value k2, then stop the iterative calculation;

[0145] Among them, if the average distance after the two point clouds are registered is greater than the given threshold C2, then the point clouds with the point cloud semantic element "Column" are deleted from the virtual model point cloud set PointCloud_5 and the point cloud set PointCloud_4 at the same time, and steps 5.5.1 to 5.5.4 are repeated; if at this time the average distance of the corresponding near point pairs after the new transformed point cloud PointCloud_6 and the point cloud set PointCloud_4 are still greater than the given threshold C2, then the point clouds with the point cloud semantic element "Beam" are deleted from the virtual model point cloud set PointCloud_5 and the point cloud set PointCloud_4 at the same time, and steps 5.5.1 to 5.5.4 are repeated; if at this time the average distance of the corresponding near point pairs after the new transformed point cloud PointCloud_6 and the point cloud set PointCloud_4 are still greater than the given threshold C2, then the ratio ratio of the average distance of the corresponding near point pairs of the new transformed point cloud PointCloud_6 and the point cloud set PointCloud_4 that is less than the given threshold C2 is counted, and the result is returned to the user, who decides whether to use the obtained rigid body transformation matrix for spatial transformation;

[0146] Among them, if the user agrees to use the obtained rigid body transformation matrix for spatial transformation, the virtual-real registration operation of this stage ends, and the user modifies the input BIM model or cleans up the on-site environment to ensure the consistency between the model and the real scene; if the user agrees to use the obtained rigid body transformation matrix for spatial transformation, proceed to the next step;

[0147] Here, if the average distance between the new transformed point cloud PointCloud_6 and the point cloud set PointCloud_4 is less than the given threshold 0.005, or the number of iterations reaches the set maximum value of 5000, the iterative calculation is stopped; otherwise, the new transformed point cloud is used as the new point cloud to be registered and continues to iterate until the objective function is met;

[0148] Step 5.5.5: Get the rigid body transformation matrix H2 after stopping the iterative calculation;

[0149] Here, after calculation, the average distance between the two point clouds after registration is 0.003, which is less than the given threshold 0.005, so the rigid body change matrix after stopping iteration is obtained.

[0150]

[0151] Step 5.6: Use the Scale parameter obtained in step 5.2 to scale the BIM model, use the rigid body transformation matrix H1 obtained in step 5.4 to achieve rapid coarse alignment between the BIM model and the real-world scenic spot cloud, and finally use the rigid body transformation matrix H2 obtained in step 5.5 to achieve precise alignment between the BIM model and the real-world scenic spot cloud, thereby completing the virtual-real scene alignment.

[0152] In summary, the present invention uses mixed reality devices to collect, classify and identify three-dimensional data of indoor environments, adds high-level semantic information to mixed reality, and enables mixed reality devices to have a simple understanding of the surrounding environment; uses point cloud registration methods to accurately register the real world and digital models, without the need to segment and extract features from point clouds, thereby achieving automation of virtual-reality registration and ensuring registration accuracy and robustness; the present invention is suitable for structural point cloud data of indoor scenes to participate in registration calculations, saving a lot of computing overhead and greatly improving the speed of virtual-reality model registration.

[0153] The present invention aims to enrich the semantic information of real scenes collected by mixed reality devices with the help of deep learning methods, and then screen the corresponding surface structure data in the real scenes and models to participate in the registration calculation, thereby ensuring accuracy and robustness while saving computing overhead, greatly improving the registration speed, and providing a solution for fast and accurate registration between the real world and digital models.

[0154] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0155] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0156] Obviously, those skilled in the art can make various changes and modifications to the invention without departing from the spirit and scope of the invention. Thus, if these modifications and variations of the invention fall within the scope of the claims of the invention and their equivalents, the invention is also intended to include these modifications and variations.

Claims

1. A point cloud semantic information assisted mixed reality virtual-real registration method, characterized in that: include: Step 1: Use mixed reality devices to obtain sampling space point cloud data; Step 2: Preprocess the sampling space point cloud to obtain the indoor real-world point cloud; Step 3: Automatically recognize the semantics of the indoor real-life point cloud to obtain a point cloud set with semantic information; Step 4: Process the BIM model to obtain a virtual model point cloud set with label information; Step 5: Match the point cloud set with semantic information with the virtual model point cloud set; Step 1: Use the mixed reality device to obtain point cloud data of the sampling space, including: Step 1.1: Use the mixed reality device to scan the indoor environment and obtain the sampling space mapping grid data; Step 1.2: Use the Poisson disk sampling method to convert the acquired spatial mapping grid data into point cloud data; Step 1.2: Use the Poisson disk sampling method to convert the acquired spatial mapping grid data into point cloud data, including: Step 1.2.1: According to the indoor area S, the number of point clouds after sampling is set to N, N = K1*S; Among them, K1 is the proportionality coefficient, which is a constant obtained by observing multiple sets of indoor data collected; Step 1.2.2: Randomly select a sample point p in the grid data sample i , with sample point p i As the center, set r as the enclosing radius of the sample point according to the side length of the mesh triangulation network, and generate a sampling point q within the circle range of (r, 2r) i ; Step 1.2.3: Set the sample point p i The number of sampling times is k1; Step 1.2.4: Determine the sampling point q i The distance to other points, when the distance is less than r, the sampling point q is discarded i , and regenerate a sampling point q within the circle range of (r, 2r) i , until the sampling point q i The distances to other points are all greater than r; Step 1.2.5: If no new available point is found within the circle of (r, 2r) after k1 samplings, the sample point p is abandoned. i Sampling of Step 1.2.6: Continuously update the sampling points and repeat the above steps so that the sampling ring fills the entire sampling space until the number of sampling points reaches N, and the sampling space point cloud data PointCloud1 is obtained; Step 2: Preprocess the sampled space point cloud to obtain indoor real-time point cloud data, including: Step 2.1: Use point cloud direct filtering to filter the non-indoor area in the sampling space point cloud PointCloud_1, and retain the point cloud data PointCloud_2 of the original indoor area; Step 2.2: Use point cloud statistical filtering to remove sparse outlier noise points caused by ambient light in the original indoor area point cloud PointCloud_2, and obtain the indoor real-world point cloud PointCloud_3; Step 2.1: Use point cloud pass-through filtering to filter the non-indoor area in the sampled space point cloud data PointCloud_1, and retain the point cloud data PointCloud_2 of the original indoor area, including: Step 2.1.1: Input the sampling space point cloud data PointCloud_1; Step 2.1.2: Set the value ranges of x, y, and z to (min_x, max_x), (min_y, max_y), and (min_z, max_z) respectively; Step 2.1.3: Eliminate the point cloud outside the range of x, y, and z values ​​to obtain the original indoor area point cloud data PointCloud_2; Step 2.2: Use point cloud statistical filtering to remove sparse outlier noise points in the original indoor area point cloud PointCloud_2 caused by ambient light, including: Step 2.2.1: Calculate the average distance between each point in the original indoor area point cloud PointCloud_2 and its K adjacent points: Step 2.2.2: Find the mean μ and standard deviation σ of all average distances; Step 2.2.3: Calculate the distance threshold dmax, dmax=μ+S'*σ, Among them, S' is the proportional coefficient, which is a constant determined by the number of adjacent points K; the larger the S' value, the fewer points are lost; Step 2.2.4: Traverse all points in the original indoor area point cloud PointCloud_2 and calculate the average distance to its K adjacent points. When the distance is greater than dmax, the point is regarded as an outlier and deleted from the point cloud data, and finally the indoor real-world point cloud data PointCloud_3 is obtained; Step 3: Automatically recognize the semantics of the indoor real-world point cloud data PointCloud_3 to obtain a point cloud set PointCloud_4 with semantic information, including: Step 3.1: Preset point cloud semantic elements: Ceiling, Ground, Wall, Column, Beam and BackGround; Step 3.2: According to the preset point cloud deep learning model, automatically perform semantic analysis on the indoor real-life point cloud data PointCloud_3; The preset point cloud deep learning model is used to automatically identify the preset point cloud semantic elements. The input is the indoor real-world point cloud data PointCloud_3; the output is the point cloud set PointCloud_4 with semantic information. The point cloud set PointCloud_4 with semantic information contains: the semantic elements corresponding to the single point and the point cloud coordinate information (x i ,y i , z i ); Step 4: Process the BIM model to obtain a virtual model point cloud set PointCloud_5 with label information, including: Step 4.1: According to the category information of the components in the BIM model, obtain the floor components, wall components, column components and beam components; Step 4.2: For the acquired floor components, according to the floor elevation, define the upper surface of the floor with a lower elevation as the ground, and assign the ground a "Ground" label; define the lower surface of the floor with a higher elevation as the ceiling, and assign the ceiling a "Ceiling" label; Step 4.3: For the obtained wall components, obtain the inward-facing surface of the wall through the Interior type attribute in the type attribute of Function, and assign the "Wall" label to the inward-facing surface; Step 4.4: For the obtained column components, assign a "Column" label to each column surface; Step 4.5: For the obtained beam components, assign a "Beam" label to each beam surface; Step 4.6: For all faces with label information, triangulate to form corresponding meshes, traverse all triangles of the meshes and obtain all vertices to obtain a virtual model point cloud set PointCloud_5 with label information; Step 5: Match the point cloud set with semantic information with the virtual model point cloud set to the virtual and real scene, including: Step 5.1: Extract the planar point clouds with point cloud semantic elements "Ceiling" and "Ground" from the point cloud set PointCloud_4 with semantic information, and extract the planar point clouds with corresponding label information of "Ceiling" and "Ground" from the point cloud set PointCloud_5; Step 5.2: Taking the surface point cloud with semantic elements "Ceiling" and "Ground" in the point cloud set PointCloud_4 with semantic information as the reference, align the "Ceiling" and "Ground" point clouds in the virtual model point cloud set PointCloud_5 with label information to the "Ceiling" and "Ground" point cloud planes in the point cloud set PointCloud_4 with semantic information, so that the scale of the virtual model is consistent with that of the display scene, and obtain the scale scaling parameter Scale; Step 5.3: Extract the planar point cloud with the point cloud semantic element "Wall" from the point cloud set PointCloud_4 with semantic information, and extract the planar point cloud with the label information "Wall" from the virtual model point cloud set PointCloud_5 with label information; Step 5.4: Based on the centroid point set of the surface point cloud extracted from the two point cloud sets, the RANSAC method is used to achieve rapid coarse registration of the virtual and real scenes; Step 5.5: Use the point cloud set PointCloud_4 with semantic information as the reference point cloud, and the virtual model point cloud set PointCloud_5 with label information as the point cloud to be registered, and use the ICP algorithm to register the two point cloud sets; Step 5.6: Use the Scale parameter obtained in step 5.2 to scale the BIM model, use the rigid body transformation matrix H1 obtained in step 5.4 to achieve rapid coarse alignment between the BIM model and the real-world scenic spot cloud, and finally use the rigid body transformation matrix H2 obtained in step 5.5 to achieve precise alignment between the BIM model and the real-world scenic spot cloud, thereby completing the virtual-real scene alignment.

2. The point cloud semantic information assisted mixed reality virtual-real registration method according to claim 1, characterized in that: Step 5.4: Based on the centroid point set of the surface point cloud extracted from the two point cloud sets, the RANSAC method is used to achieve rapid coarse registration of the virtual and real scenes, including: Step 5.4.1: Calculate the centroid of each planar point cloud with the semantic element "Wall" in the point cloud set (PointCloud_4) with semantic information, and construct the point set Points1 = {point1, point2, ..., point n }; Calculate the centroid of the surface point cloud with "Wall" label information in the virtual model point cloud set (PointCloud_5) with label information, and form a point set Points2 = {point1, point2, ..., point n }; Step 5.4.2: Randomly select 3 points in the point set Points1, randomly select 3 corresponding point pairs in the point set Points2, and solve the rigid body transformation matrix; Step 5.4.3: Calculate the distance error of the remaining point pairs under the rigid body transformation matrix obtained in step 5.4.

2. If the distance error of one of the point pairs is less than the set error threshold C1, then the point is an in-sample point, otherwise it is an out-sample point, and the number of in-sample points is counted; Step 5.4.4: Set the upper limit of the number of generations to k1, repeat the above steps until the upper limit of the number of generations is k1, count the number of points inside the samples under different rigid body transformation models, and obtain the rigid body transformation matrix H1 when the number of points inside the sample is the largest; Step 5.5: Using the point cloud set PointCloud_4 with semantic information as the reference point cloud and the virtual model point cloud set PointCloud_5 with label information as the point cloud to be registered, the two point cloud sets are registered using the ICP algorithm, including: Step 5.5.1: Use the KD tree nearest neighbor search algorithm to calculate the corresponding close points of each point in the point cloud set PointCloud_5 in the point cloud set PointCloud_4 to obtain the corresponding close point pairs; Step 5.5.2: Calculate the rigid body transformation that minimizes the average distance between the corresponding close point pairs and obtain the rigid body transformation matrix; Step 5.5.3: Use the rigid body transformation matrix obtained in step 5.5.2 to perform spatial transformation on the point cloud set PointCloud_4 to obtain a new point cloud set PointCloud_6; Step 5.5.4: If the transformed new point cloud PointCloud_6 and the point cloud set PointCloud_4 satisfy that the average distance between the corresponding close point pairs of the two point clouds is less than the given threshold C2, or the number of iterations reaches the set maximum value k2, then stop the iterative calculation; Among them, if the average distance between the two point clouds after registration is greater than the given threshold C2, the point clouds with the point cloud semantic element "Column" are deleted from the point cloud set PointCloud_5 and the point cloud set PointCloud_4 at the same time, and steps 5.5.1 to 5.5.4 are repeated; if at this time the average distance between the corresponding near point pairs after the new transformed point cloud pointCloud_6 and the point cloud set PointCloud_4 are still greater than the given threshold C2, the point clouds with the point cloud semantic element "Beam" are deleted from the point cloud set PointCloud_5 and the point cloud set PointCloud_4 at the same time, and steps 5.5.1 to 5.5.4 are repeated; if at this time the average distance between the corresponding near point pairs after the new transformed point cloud PointCloud_6 and the point cloud set PointCloud_4 are still greater than the given threshold C2, then the ratio ratio of the average distance between the corresponding near point pairs of the new transformed point cloud PointCloud_6 and the point cloud set PointCloud_4 that is less than the given threshold C2 is counted, and the result is directly returned to the user.

Images