An indoor pose initialization method, system and terminal for a mobile construction robot

Abstract

The application discloses an indoor pose initialization method, system and terminal of a mobile construction robot, and the method realizes robust pose initialization under a complex construction environment through multi-semantics alignment and a two-stage matching mechanism. Firstly, a BIM model is converted into a global point cloud map with category semantics, an image segmentation is used to extract static structures, and a semantic-enhanced 3D point cloud is generated in combination with real-time point clouds. A point cloud global descriptor is improved by fusing geometric features and category semantic coding, coarse matching is realized by using hash indexing and geometric consistency verification. Finally, pose optimization is performed through an improved ICP algorithm weighted by semantics, and an accurate initial pose of the mobile construction robot is obtained. The application effectively solves the matching failure problem of traditional methods under weak texture and dynamic interference environments, breaks through the dependence of the prior art on a motion trajectory, realizes fast and accurate initialization of an arbitrary initial pose, and provides accurate initial coordinates for autonomous navigation of the construction robot.

Description

Technical Field

[0001] This invention relates to the field of robot autonomous positioning and navigation, and in particular to an indoor pose initialization method, system and terminal for a mobile construction robot. Background Technology

[0002] Initializing the pose of a construction robot involves determining the robot's initial pose (including position and attitude) on the navigation map and rigidly associating the robot's local odometry coordinate system with the global coordinate system of the navigation map. This is a prerequisite for the robot to achieve navigation planning and autonomous inspection on the construction site.

[0003] Traditional indoor pose initialization methods rely on manual specification, leading to significant discrepancies between the robot's actual pose and the initial pose on the map, making subsequent navigation tasks impossible. To address this issue, some methods involve setting positioning tags (such as UWB or RFID) on the construction site or the robot itself and recording the tag positions during robot movement to achieve initial localization. However, this method requires linear motion within a certain range to determine the initial pose and cannot correlate the robot's actual initial pose with the navigation map.

[0004] There are also vision-based pose initialization methods that acquire images of the robot's location using a camera, extract feature points and descriptors from the images, match them with image features stored in a prior navigation map, and use the pose of the feature points in the prior image as the robot's initial pose. However, vision-based pose initialization methods are easily affected by factors such as scene lighting and feature sparsity, limiting the accuracy and applicability of pose initialization.

[0005] Therefore, existing technologies still need to be improved and developed. Summary of the Invention

[0006] The main objective of this invention is to provide an indoor pose initialization method, system, terminal, and computer-readable storage medium for mobile construction robots. This invention aims to solve the problems of traditional indoor pose initialization methods in the prior art, which are prone to matching failure in environments with weak textures and dynamic interference, cannot achieve fast and accurate initialization of arbitrary initial poses, and cannot provide accurate initial coordinates for autonomous navigation of construction robots.

[0007] To achieve the above objectives, the present invention provides an indoor pose initialization method for a mobile construction robot, the method comprising the following steps:

[0008] Collect image data and point cloud data of the pose of the mobile construction robot when it is powered on, and perform point cloud sampling and annotation on a pre-set BIM model to obtain a semantic point cloud map.

[0009] The static structure is segmented from the image data. Based on the spatial relationship between the camera and the LiDAR, the point cloud data is associated with the semantics of the 2D image to obtain point cloud data with assigned semantics. The point cloud data with assigned semantics is then filtered based on the static structure to obtain static structure point cloud data.

[0010] Key points are extracted from the static structural point cloud data or the semantic point cloud map. A 3D point cloud global descriptor is constructed using the key points. The 3D point cloud global descriptor is stored in a hash table to obtain the 3D point cloud global descriptor construction method.

[0011] The 3D point cloud global descriptor construction method is used to extract 3D point cloud descriptors from the static structural point cloud data and the semantic point cloud map, respectively, and encode and coarsely match them to obtain a coarse pose and an optimal point cloud. The improved semantic iterative nearest point algorithm is used to iteratively optimize the coarse pose to obtain the precise initial pose of the mobile construction robot.

[0012] Optionally, the indoor pose initialization method for the mobile construction robot, wherein the acquisition of image data and point cloud data of the pose of the mobile construction robot when it is powered on, further includes:

[0013] A lidar-camera perception system is pre-deployed on the mobile construction robot body, and the extrinsic parameters of the lidar-camera perception system are pre-calibrated using the Kalibr algorithm;

[0014] The LiDAR-camera sensing system is deployed in the same direction as the mobile construction robot's forward movement.

[0015] Optionally, the indoor pose initialization method for the mobile construction robot, wherein the step of sampling and labeling point clouds on a pre-set BIM model to obtain a semantic point cloud map specifically includes:

[0016] The pre-set BIM model is converted from RVT format to IFC format. Static structural elements are extracted from the IFC format BIM model according to IFC category, and the static structural elements are saved as structure category files in OBJ format.

[0017] The types of static structural elements include columns, beams, walls, doors, windows, and stairs;

[0018] Point cloud sampling is performed on the structure category files to obtain PCD point cloud files of different categories, and the PCD point cloud files of different categories are labeled.

[0019] All labeled point cloud pcd files are merged into a complete point cloud with categories, and this point cloud is used as the semantic point cloud map.

[0020] Optionally, the indoor pose initialization method for the mobile construction robot, wherein segmenting the static structure from the image data specifically includes:

[0021] Construct a joint detection-segmentation model, which includes the YOLO-World object detection model and the MobileSAM zero-shot segmentation model;

[0022] The YOLO-World target detection model was quantized using FP16 mixed precision using the TensorRT tool to obtain a quantized detection model, which was then deployed on an edge device.

[0023] The image data is input into the quantized detection model to obtain the target location. This target location is then used as a segmentation cue and input into the MobileSAM zero-shot segmentation model to obtain the target location.

[0024] The static structure semantic mask in an image.

[0025] Optionally, the indoor pose initialization method for the mobile construction robot, wherein the step of associating the point cloud data with 2D image semantics based on the spatial relationship between the camera and the lidar to obtain point cloud data with assigned category semantics, and filtering the point cloud data with assigned category semantics according to the static structure to obtain static structure point cloud data, specifically includes:

[0026] Based on the camera intrinsic parameters and radar-camera extrinsic parameters, the point cloud data is projected onto a 2D pixel coordinate system to obtain projection points, and the semantic mask of the projection points is calculated using a bilinear interpolation algorithm.

[0027] Based on the semantic mask of the projection points, the 3D coordinates of the point cloud data are assigned category semantics to obtain point cloud data with assigned category semantics.

[0028] Based on the static structure semantic mask, point clouds with static structure categories in the point cloud data after being assigned category semantics are filtered to obtain static structure point cloud data.

[0029] Optionally, the indoor pose initialization method for the mobile construction robot, wherein extracting key points from the static structural point cloud data or the semantic point cloud map, constructing a 3D point cloud global descriptor using the key points, and storing the 3D point cloud global descriptor using a hash table specifically includes:

[0030] The intrinsic shape signature algorithm is used to extract multiple key points from the static structural point cloud data or the semantic point cloud map, construct a reference plane, and divide the reference plane into multiple pixel regions according to the preset side length.

[0031] Each key point is projected onto a certain pixel region, and a 3D point cloud global descriptor is constructed based on the principle of triangle invariance.

[0032] The global descriptor of the 3D point cloud is stored using a key-value pair-based hash table;

[0033] The 3D point cloud global descriptor includes a keypoint binary descriptor and a triangle descriptor. The keypoint binary descriptor includes a height descriptor and a category semantic descriptor. The height descriptor is used to encode the local geometric structure of the keypoint, and the category semantic descriptor is used to encode the semantics of the keypoint and the semantic proportion of the keypoint in its nearest neighbors. The vertices of the triangle descriptor are composed of keypoints of three different semantic categories. The side lengths of the triangle descriptors are arranged in ascending order of length. The key of the hash table is calculated by a hash function and the side lengths of the triangles. The index position of the key is a container that stores similar triangle descriptors.

[0034] Optionally, the indoor pose initialization method for the mobile construction robot, wherein the step of extracting 3D point cloud descriptors from the static structural point cloud data and the semantic point cloud map using the 3D point cloud global descriptor construction method, encoding and coarsely matching them to obtain a coarse pose and an optimal point cloud, specifically includes:

[0035] The 3D point cloud global descriptor construction method is used to extract descriptors from the static structure point cloud data to obtain a candidate descriptor set, and the 3D point cloud global descriptor construction method is used to extract descriptors from the semantic point cloud map to form a query descriptor set;

[0036] The key value of each query descriptor in the query descriptor set is obtained by using a hash function and the side length of a triangle. Based on all the key values, the candidate descriptor corresponding to each query descriptor in the candidate descriptor set is obtained.

[0037] Calculate the binary descriptor similarity between all the query descriptors and their respective candidate descriptors, and compare all the binary descriptor similarities with a first preset threshold to obtain multiple target binary descriptor similarities that satisfy the preset threshold;

[0038] The candidate descriptors corresponding to the similarity of the multiple target binary descriptors are used as target candidate descriptors, and the static structural point cloud data fragments corresponding to each of the multiple target candidate descriptors are used as the initial screening target point clouds.

[0039] The number of target candidate descriptors in the multiple preliminary screening target point clouds is compared with a second preset threshold, and the preliminary screening target point cloud with the number of target candidate descriptors exceeding the second preset threshold is taken as the target point cloud;

[0040] Obtain the triangle plane of the query descriptor corresponding to a certain target point cloud, and obtain the target triangle plane that matches the target point cloud based on the triangle plane;

[0041] A point pair matching relationship is established based on the stated triangle plane and the target triangle plane. Based on the point pair matching relationship, a coarse pose between the stated triangle plane and the target triangle plane is obtained by singular value decomposition.

[0042] Using the coarse pose, the number of planes that overlap between the triangular plane in each query point cloud and the target triangular plane in the target point cloud is calculated, resulting in multiple plane overlap numbers. The target point cloud with the largest plane overlap number is selected as the optimal point cloud.

[0043] Optionally, the indoor pose initialization method for the mobile construction robot, wherein the step of iteratively optimizing the coarse pose using an improved semantic iterative nearest-point algorithm to obtain the precise initial pose of the mobile construction robot specifically includes:

[0044] The kD tree is used to obtain the corresponding point in the optimal point cloud for each point in the query descriptor subset, and a matching point pair is established based on each point in the query descriptor subset and the corresponding point.

[0045] Calculate the category semantic consistency of all the matching point pairs, assign weights to all the matching point pairs based on the category semantic consistency, and calculate the normal vectors of all points in the query descriptor subset and the optimal point cloud.

[0046] Based on the multiple weights and multiple normal vectors, the optimal rotation matrix and the optimal translation matrix are calculated, and the precise initial pose of the mobile construction robot is obtained based on the optimal rotation matrix and the optimal translation matrix.

[0047] Furthermore, to achieve the above objectives, the present invention also provides an indoor pose initialization system for a mobile construction robot, wherein the indoor pose initialization system for the mobile construction robot includes:

[0048] The data acquisition and map creation module is used to collect image data and point cloud data of the mobile construction robot's pose when it is powered on, and to perform point cloud sampling and annotation on a pre-set BIM model to obtain a semantic point cloud map.

[0049] The static structure point cloud data acquisition module is used to segment the static structure from the image data, associate the point cloud data with the semantics of the 2D image according to the spatial relationship between the camera and the lidar, obtain point cloud data with assigned semantics, and filter the point cloud data with assigned semantics according to the static structure to obtain static structure point cloud data.

[0050] The global descriptor construction module extracts key points from the static structured point cloud data or the semantic point cloud map, uses the key points to construct a 3D point cloud global descriptor, and uses a hash table to store the 3D point cloud global descriptor, thus obtaining the 3D point cloud global descriptor construction method.

[0051] The initial pose matching module is used to extract 3D point cloud descriptors from the static structural point cloud data and the semantic point cloud map using the 3D point cloud global descriptor construction method, and to encode and coarsely match them to obtain a coarse pose and an optimal point cloud. The improved semantic iterative nearest point algorithm is then used to iteratively optimize the coarse pose to obtain the accurate initial pose of the mobile construction robot.

[0052] In addition, to achieve the above objectives, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and an indoor pose initialization program for a mobile construction robot stored in the memory and executable on the processor, wherein when the indoor pose initialization program for the mobile construction robot is executed by the processor, it implements the steps of the indoor pose initialization method for the mobile construction robot as described above.

[0053] In addition, to achieve the above objectives, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores an indoor pose initialization program for a mobile construction robot, and when the indoor pose initialization program for the mobile construction robot is executed by a processor, it implements the steps of the indoor pose initialization method for the mobile construction robot as described above.

[0054] This invention achieves robust pose initialization in complex construction environments through multi-semantic alignment and a two-stage matching mechanism. First, the BIM model is converted into a global point cloud map with category semantics. Image segmentation is used to extract static structures, which are then combined with real-time point clouds to generate semantically enhanced 3D point clouds. An improved point cloud global descriptor, fusing geometric features and category semantic encoding, is used, employing hash indexing and geometric consistency verification for coarse matching. Finally, a semantically weighted improved ICP algorithm is used for pose fine optimization, ultimately obtaining the precise initial pose of the mobile construction robot. This invention effectively solves the matching failure problem of traditional methods in environments with weak texture and dynamic interference, overcomes the dependence of existing technologies on motion trajectories, and achieves rapid and accurate initial pose initialization for any given pose, providing precise initial coordinates for the autonomous navigation of construction robots. Attached Figure Description

[0055] Figure 1 This is a flowchart of a preferred embodiment of the indoor pose initialization method for the mobile construction robot of the present invention;

[0056] Figure 2 This is a schematic diagram illustrating the method for initializing the indoor pose of the mobile construction robot in this invention, which assigns semantic category information to point clouds through image semantics.

[0057] Figure 3 This is a schematic diagram of the 3D point cloud global descriptor composition in the indoor pose initialization method of the mobile construction robot of the present invention.

[0058] Figure 4 This is a schematic diagram of point cloud coarse matching based on 3D point cloud global descriptor in the indoor pose initialization method of the mobile construction robot of the present invention.

[0059] Figure 5 This is a structural diagram of a preferred embodiment of the indoor pose initialization system for the mobile construction robot of the present invention;

[0060] Figure 6 This is a structural diagram of a preferred embodiment of the terminal of the present invention. Detailed Implementation

[0061] This application provides an indoor pose initialization method, system, and terminal for a mobile construction robot. To make the purpose, technical solution, and effects of this application clearer and more explicit, the following detailed description is provided with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only for explaining this application and are not intended to limit this application.

[0062] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0063] Furthermore, if the embodiments of this invention involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.

[0064] The indoor pose initialization method for a mobile construction robot according to a preferred embodiment of the present invention, such as... Figure 1 As shown, the indoor pose initialization method for the mobile construction robot includes the following steps:

[0065] Step S10: Collect image data and point cloud data of the pose of the mobile construction robot when it is powered on, and perform point cloud sampling and annotation on the pre-set BIM model to obtain a semantic point cloud map.

[0066] It is understandable that the acquisition of image data and point cloud data of the pose of the mobile construction robot when it is powered on also previously included:

[0067] A lidar-camera perception system is pre-deployed on the mobile construction robot, and the extrinsic parameters of the lidar-camera perception system are pre-calibrated using the Kalibr algorithm. The deployment orientation of the lidar-camera perception system is consistent with the forward direction of the mobile construction robot.

[0068] In this embodiment, a lidar-camera perception system is pre-deployed on the mobile construction robot body, and the Kalibr algorithm (which supports calibration of multi-camera systems with non-globally shared overlapping fields of view and can handle various camera models) is used to pre-calibrate the intrinsic parameters of the camera and the extrinsic parameters between the lidar-camera perception system, and ensures that the installation orientation of the lidar-camera perception system is consistent with the robot's forward direction.

[0069] After deploying and calibrating the lidar-camera perception system on the mobile construction robot, the robot system is activated after the mobile construction robot is placed at any position and orientation in the scene to acquire image and point cloud data. The image and point cloud data at this time are regarded as the image and point cloud data under the initial pose.

[0070] Furthermore, the pre-set BIM model (Building Information Modeling, in this application, the BIM model is constructed based on the interior scene, which may be constructed during the design phase or during the construction phase) is converted from RVT format to IFC format. Static structural elements are extracted from the IFC format BIM model according to IFC categories, and the static structural elements are saved as structural category files in OBJ format. The types of the static structural elements include columns, beams, walls, doors, windows, and stairs.

[0071] In this embodiment, the BIM model's RVT format is first converted to IFC format. Then, ifcConvert is used to extract static structural elements from the IFC format model according to IFC categories and save them as OBJ format structural category files. The static structural categories and their formats in IFC include: column—ifcColumn, beam—ifcBeam, wall—ifcWall, door (frame)—ifcDoor, window (frame)—ifcWindow, and staircase—ifcStair.

[0072] Furthermore, point cloud sampling is performed on the structure category files to obtain PCD point cloud files of different categories, and the PCD point cloud files of different categories are labeled.

[0073] It is understood that this embodiment takes the structure category file in obj format as input, uses CloudCompare to perform point cloud sampling on the extracted surface of different categories of structures, and obtains the pcd point cloud file of the structure of that category.

[0074] Based on this, CloudCompare uses unified tags to annotate PCD point cloud files of different categories and structures. The unified tags include: 0: other, 1: column, 2: beam, 3: wall, 4: door, 5: window, and 6: stair.

[0075] Furthermore, all the labeled point cloud pcd files are merged into a complete point cloud with categories, and this point cloud is used as the semantic point cloud map.

[0076] In this embodiment, CloudCompare is used to merge all the labeled point cloud pcd files into a complete point cloud with categories, which serves as the semantic point cloud map.

[0077] As can be seen, this application uses ifcConvert to extract structural elements from the BIM model according to categories and save them as obj format; performs point cloud sampling on the surface of the structural elements in obj format to generate 3D point clouds; performs semantic annotation on point clouds of different structural types and assigns semantic information to point cloud categories; and merges semantic point clouds of different types to generate a global semantic point cloud map.

[0078] Step S20: Segment the static structure from the image data, associate the point cloud data with the semantics of the 2D image according to the spatial relationship between the camera and the lidar, and obtain point cloud data with assigned semantics. Filter the point cloud data with assigned semantics according to the static structure to obtain static structure point cloud data.

[0079] The step of segmenting the static structure from the image data specifically includes:

[0080] Construct a joint detection-segmentation model, which includes the YOLO-World object detection model and the MobileSAM zero-shot segmentation model;

[0081] The YOLO-World target detection model was quantized using FP16 mixed precision using the TensorRT tool to obtain a quantized detection model, which was then deployed on an edge device.

[0082] The image data is input into the quantized detection model to obtain the target location. The target location is then used as a segmentation cue and input into the MobileSAM zero-shot segmentation model to obtain the static structure semantic mask of the static structure in the image.

[0083] In this embodiment, a segmentation model is used to segment static structures from the image data. The segmentation model uses a quantized version of the YOLO-World object detection model + MobileSAM zero-shot segmentation model to perform semantic segmentation on indoor static structures (including columns, beams, walls, door and window openings, stairs, etc.).

[0084] Understandably, in order to further improve the detection accuracy of static structures in indoor construction scenarios, this application may also collect corresponding image datasets to train the segmentation model, such as the DETR series, or adopt a state-of-the-art detection model (such as Grounding DINO).

[0085] like Figure 2As shown, further, based on the camera intrinsic parameters and the radar-camera extrinsic parameters, the point cloud data is projected into a 2D pixel coordinate system to obtain projection points, and the semantic mask of the projection points is calculated using a bilinear interpolation algorithm.

[0086] Specifically, based on camera intrinsic parameters and radar-camera extrinsic parameters, the acquired 3D point cloud is projected onto a pixel coordinate system:

[0087]

[0088] Where s is the camera depth information, (u,v) is the pixel coordinates of the 3D point cloud (x,y,z) projected onto the 2D image, K is the camera intrinsic parameter matrix, and [R|t] is the radar-camera extrinsic parameter matrix.

[0089] Furthermore, based on the semantic mask of the projection points, the 3D coordinates of the point cloud data are assigned category semantics to obtain point cloud data with assigned category semantics;

[0090] Specifically, the semantic mask of the projection point (u,v) is calculated using a bilinear interpolation algorithm:

[0091] round(L(u,v))=(1-α)(1-β)L(u0,v0)+α(1-β)L(u1,v0)+(1-α)βL(u0,v1)+αβL(u1,v1);

[0092] Among them, α=u-u0, β=v-v0, L(u,v) is the semantic label corresponding to pixel (u,v), where L∈{0,1,2,3,4,5,6}, and round(L(u,v)) is the semantic mask of the pixel (u,v) where the projection point is located. It is the floor symbol.

[0093] Based on the semantic mask of the projection points, the 3D coordinates of the point cloud data are assigned (extended) with category semantics, resulting in point cloud data [xyzc] with assigned category semantics. Here, dimension c represents the structural category semantics assigned to the point cloud. Represents natural numbers.

[0094] Furthermore, based on the static structural semantic mask, point clouds with static structural categories in the point cloud data after being assigned category semantics are filtered to obtain static structural point cloud data.

[0095] Specifically, after assigning a category semantic to each point, point clouds with static structural categories are filtered in the 3D semantic point cloud, and point clouds with a category label of 0 are deleted. In this application, to avoid label noise caused by interpolation, the DBSCAN clustering algorithm (Density-Based Spatial Clustering of Applications with Noise, a density-based spatial clustering algorithm that can divide data points into core points, boundary points, and noise points) can also be used to cluster the point clouds of the same object and unify their semantic labels.

[0096] Step S30: Extract key points from the static structural point cloud data or the semantic point cloud map, construct a 3D point cloud global descriptor using the key points, and store the 3D point cloud global descriptor using a hash table.

[0097] like Figure 3 As shown, specifically, an intrinsic shape signature algorithm is used to extract multiple key points from the static structural point cloud data or the semantic point cloud map, construct a reference plane, and divide the reference plane into multiple pixel regions according to a preset side length.

[0098] In this embodiment, the Intrinsic Shape Signature (ISS) algorithm is used to extract key points from the static structural point cloud data. Then, a reference plane is constructed with z=0, and the plane is divided into multiple pixel regions with a preset length d as the side length.

[0099] Furthermore, each key point is projected onto a certain pixel region, and a 3D point cloud global descriptor is constructed based on the principle of triangle invariance.

[0100] It is understood that in this embodiment, each key point is projected onto a certain pixel area of ​​the reference plane to construct a 3D point cloud global descriptor based on the principle of triangle invariance. The 3D point cloud global descriptor consists of two parts: a key point binary descriptor and a triangle descriptor.

[0101] The keypoint binary descriptor includes a height descriptor and a category semantic descriptor. Therefore, in this embodiment, a height descriptor and a category semantic descriptor are constructed, and the height descriptor and the category semantic descriptor are combined into a (5+m) bit binary number to represent the keypoint binary descriptor.

[0102] Among them, the height descriptor is used to encode the local geometry of key points. First, the indoor floor height h is... max The system is divided into m layers based on the resolution Δh in the height direction; taking the reference plane pixels where the keypoint is located as an example, if a certain layer m in the vertical direction... iIf a point cloud exists, the layer is encoded as 1; otherwise, it is encoded as 0. Therefore, the height descriptor of a certain key point P can be represented as [101000011].

[0103] Category semantic descriptors are used to encode the semantics of key points and their semantic proportions in their nearest neighbors. Static structural point cloud labels are encoded using 3 bits, such as

[010] for the point cloud semantics of the beam category. The semantic proportions are divided into discrete intervals (e.g., 0-20%, 21-50%, 51-80%, 81-100%), and 2 bits are used to encode these discrete intervals, such as

[01] for 21-50%, thus avoiding encoding jumps caused by small semantic fluctuations.

[0104] After obtaining the binary descriptor of the keypoints, a triangle descriptor is also needed. Specifically, the vertices of the triangle descriptor are composed of three keypoints (P1, P2, P3) of different semantic categories, and the lengths of the three sides l1, l2, l3 are calculated by ||(P1, P2)||2, ||(P2, P3)||2, and ||(P3, P1)||2, respectively. Therefore, the triangle descriptor TriangleDes can be represented as:

[0105] TriangleDes = [l1, l2, l3];

[0106] In order to quickly obtain the corresponding point pairs after matching the relationship of the three sides, the side lengths are arranged in ascending order of length in the triangle descriptor, i.e., l1≤l2≤l3, which correspond to points P1, P2, and P3 respectively.

[0107] It should be noted that, to improve the efficiency of subsequent pose calculations, when constructing the triangle descriptor, a kD tree is built for the keypoints of each category, and the triangle descriptor is constructed by querying only the k nearest keypoints from other categories each time. Therefore, for a given query keypoint, there will be 10k... 2 A number of candidate triangle descriptors. Furthermore, this invention excludes triangles with any two sides equal, triangles with side lengths that are too short or too long, and duplicate triangle descriptors.

[0108] Furthermore, a key-value pair-based hash table is used to store the global descriptor of the 3D point cloud.

[0109] In this embodiment, a key-value pair-based hash table is used to store the global descriptor of the 3D point cloud. The key is calculated by a hash function and the side length of the triangle, and the value at the key index position is a container that stores descriptors of similar triangles.

[0110] Specifically, the value Hash(l1,l2,l3) at the key index position is calculated using the following formula:

[0111]

[0112] in, Q is a prime number whose value is close to but not greater than the hash table capacity, N is the hash table length, and Δl is the scaling factor. and These are the scaled values ​​of l1, l2, and l3, respectively, and mod() is the modulo function.

[0113] Step S40: Using the 3D point cloud global descriptor construction method, extract 3D point cloud descriptors from the static structural point cloud data and the semantic point cloud map respectively, encode and coarsely match them to obtain a coarse pose and the best point cloud. Use the improved semantic iterative nearest point algorithm to iteratively optimize the coarse pose to obtain the accurate initial pose of the mobile construction robot.

[0114] The step of using the 3D point cloud global descriptor construction method to extract 3D point cloud descriptors from the static structural point cloud data and the semantic point cloud map, respectively, and then encoding and coarsely matching them to obtain a coarse pose and an optimal point cloud, specifically includes:

[0115] The 3D point cloud global descriptor construction method is used to extract descriptors from the static structure point cloud data to obtain a candidate descriptor set, and the 3D point cloud global descriptor construction method is used to extract descriptors from the semantic point cloud map to form a query descriptor set;

[0116] The key value of each query descriptor in the query descriptor set is obtained by using a hash function and the side length of a triangle. Based on all the key values, the candidate descriptor corresponding to each query descriptor in the candidate descriptor set is obtained.

[0117] Calculate the binary descriptor similarity between all the query descriptors and their respective candidate descriptors, and compare all the binary descriptor similarities with a first preset threshold to obtain multiple target binary descriptor similarities that satisfy the preset threshold;

[0118] The candidate descriptors corresponding to the similarity of the multiple target binary descriptors are used as target candidate descriptors, and the static structural point cloud data fragments corresponding to each of the multiple target candidate descriptors are used as the initial screening target point clouds.

[0119] The number of target candidate descriptors in the multiple preliminary screening target point clouds is compared with a second preset threshold, and the preliminary screening target point cloud with the number of target candidate descriptors exceeding the second preset threshold is taken as the target point cloud;

[0120] Obtain the triangle plane of the query descriptor corresponding to a certain target point cloud, and obtain the target triangle plane that matches the target point cloud based on the triangle plane;

[0121] A point pair matching relationship is established based on the stated triangle plane and the target triangle plane. Based on the point pair matching relationship, a coarse pose between the stated triangle plane and the target triangle plane is obtained by singular value decomposition.

[0122] Using the coarse pose, the number of planes that overlap between the triangular plane in each query point cloud and the target triangular plane in the target point cloud is calculated, resulting in multiple plane overlap numbers. The target point cloud with the largest plane overlap number is selected as the optimal point cloud.

[0123] like Figure 4 As shown, the process of obtaining the optimal point cloud is explained in detail below with the help of formulas:

[0124] In this embodiment, the point cloud (static structural point cloud data) obtained from the BIM model is first encoded using a 3D point cloud global descriptor to obtain a candidate descriptor set, and a hash table is constructed to store the descriptor container containing key values ​​and their indices. Then, the descriptor set of the semantic point cloud map (referred to as the query descriptor set, or query point cloud) is calculated, and for each query descriptor... By using a hash function and the side length of the triangle, the corresponding key value can be obtained, thereby indexing the aforementioned candidate descriptor subset with similar characteristics.

[0125] Further, the binary descriptor similarity between the query descriptor and the candidate descriptor for their corresponding key points is calculated. If the similarity meets a threshold, then the candidate descriptor is considered valid. The corresponding BIM semantic point cloud fragments can be recorded as the initial target point clouds. The similarity calculation process is repeated for each query descriptor until the binary descriptor similarity between all the query descriptors and their corresponding candidate descriptors is calculated. When the number of successfully matched candidate descriptors in a point cloud fragment exceeds a certain threshold, that point cloud fragment can be saved as a target point cloud. From this, θ highly similar target point clouds are extracted for the next geometric verification step to find the optimal target point cloud for subsequent pose matching.

[0126] The similarity calculation process is as follows:

[0127]

[0128] Here, '&' represents a bitwise AND operation in binary. It is the Hamming weight, whose value is equal to the number of "1"s in a binary number. This represents the binary descriptor from the i-th plane of the query descriptor subset. This represents a binary descriptor from the i-th plane of the candidate descriptor subset. express and The similarity between them.

[0129] Furthermore, the triangular plane of the query descriptor corresponding to the target point cloud... With the triangle descriptor plane in the target point cloud Perform geometric verification:

[0130] Traverse and query every triangular plane in the point cloud Based on the initial matching, the target point cloud is obtained from the... Matching triangle plane Based on this, a correspondence is established between the vertices of the triangles in the two planes. Based on the established point-pair matching relationship, the singular value decomposition method is used to calculate the transformation relationship T = [R'|t'] between the two triangular planes (i.e., from the query point cloud to the target point cloud). It can be understood that this transformation relationship constitutes the coarse pose. Based on the transformation relationship T, each... The geometric centroid of the plane to The distance (it should be noted that, It is any triangle descriptor plane in the target point cloud, where j represents the ordinal number of the plane, and and The angle between the normal vectors is used to determine whether two planes have geometric consistency, i.e., whether the two planes coincide. If the proportion of planes in the query point cloud that coincide with planes in the target point cloud is the largest among all planes in the query point cloud, it indicates that the target point cloud has the highest similarity to the query point cloud and can be considered the best target point cloud.

[0131] The formulas for calculating the distance from the point to the plane and the normal angle are as follows:

[0132]

[0133] Where μ is the normal vector of the plane, Let be the centroid of the triangle, ε and δ be hyperparameters used as the distance threshold and normal angle difference threshold between the two planes, respectively, R' represent a 3x3 rotation matrix, and t' represent a 3x1 displacement vector. Let represent the normal vector of the j-th plane in the target point cloud c'. This indicates a query for the normal vector of the i-th plane in the point cloud q. This query retrieves the centroid of the i-th plane in the point cloud q. Let represent the centroid of the i-th plane in the target point cloud c'.

[0134] Furthermore, the step of iteratively optimizing the coarse pose using the improved semantic iterative nearest-point algorithm to obtain the precise initial pose of the mobile construction robot specifically includes:

[0135] The kD tree is used to obtain the corresponding point in the optimal point cloud for each point in the query descriptor subset, and a matching point pair is established based on each point in the query descriptor subset and the corresponding point.

[0136] Calculate the category semantic consistency of all the matching point pairs, assign weights to all the matching point pairs based on the category semantic consistency, and calculate the normal vectors of all points in the query descriptor subset and the optimal point cloud.

[0137] Based on the multiple weights and multiple normal vectors, the optimal rotation matrix and the optimal translation matrix are calculated, and the precise initial pose of the mobile construction robot is obtained based on the optimal rotation matrix and the optimal translation matrix.

[0138] Understandably, the objective function of traditional point-to-point ICP methods is optimized only based on the distance between point pairs, without considering the semantic and local geometric information of the points. Therefore, a semantically improved point-to-plane ICP algorithm (PP-ICP) is introduced to optimize the optimal point cloud and obtain an accurate initial pose.

[0139] The process of obtaining the precise initial pose of the mobile construction robot is explained in detail below, using formulas:

[0140] Use a kD-tree to find each point in the query point cloud. The corresponding point in the target point cloud Establish matching pairs

[0141] Furthermore, when determining matching point pairs, it is also necessary to further judge the semantic consistency of the matching point pairs and assign weights to the point pairs in the objective function. The specific calculations are as follows:

[0142]

[0143] Furthermore, when calculating the distance between point pairs using Euclidean distance, points whose distance is greater than a threshold are deleted.

[0144] For each point P t Given t∈{q,c'}, calculate point P with radius r as its neighborhood. t The normal vector μ is used to calculate the optimal rotation and translation matrices to obtain the optimal initial pose, specifically:

[0145]

[0146] in, It is the weighting coefficient, R * Let t represent the optimal rotation matrix. * This represents the optimal translation vector.

[0147] Compared with the prior art, the beneficial effects of the technical method provided by the present invention are:

[0148] (1) Multimodal semantic fusion enhances adaptability in complex indoor environments. This invention achieves cross-modal data complementarity by fusing images, point clouds, and multi-level semantics of BIM. By using image segmentation to filter dynamic interference point clouds and combining the structural prior information of BIM semantic point clouds, it effectively overcomes interference such as feature sparsity and moving obstacles in the construction environment, and significantly improves the robustness and adaptability of pose initialization in complex indoor scenes.

[0149] (2) Semantically Enhanced Global Descriptors Improve Matching Efficiency and Accuracy. This invention solves the problem of low matching efficiency and accuracy caused by the lack of semantic constraints in traditional point cloud matching by constructing a 3D point cloud global descriptor with enhanced structural semantics. Combining hash indexing and geometric consistency verification, the initial pose is quickly calculated in the coarse matching stage, and then fine-tuned by an improved ICP algorithm with semantic weighting.

[0150] (3) Fast and accurate initialization of any position and pose without motion constraints. Existing methods rely on linear, rotational motion or feature position for initialization. This invention achieves fast and accurate initialization of robot pose at any initial position and any pose by cross-modal alignment of static structural semantics, combined with semantically enhanced 3D point cloud global descriptors and a two-stage matching method.

[0151] Furthermore, such as Figure 5 As shown, based on the above-described indoor pose initialization method for a mobile construction robot, the present invention also provides an indoor pose initialization system for a mobile construction robot, wherein the indoor pose initialization system for the mobile construction robot includes:

[0152] The data acquisition and map creation module 51 is used to collect image data and point cloud data of the pose of the mobile construction robot when it is powered on, and to perform point cloud sampling and annotation on the pre-set BIM model to obtain a semantic point cloud map.

[0153] The static structure point cloud data acquisition module 52 is used to segment the static structure from the image data, associate the point cloud data with the semantics of the 2D image according to the spatial relationship between the camera and the lidar, obtain point cloud data with assigned category semantics, and filter the point cloud data with assigned category semantics according to the static structure to obtain static structure point cloud data.

[0154] The global descriptor construction module 53 extracts key points from the static structural point cloud data or the semantic point cloud map, uses the key points to construct a 3D point cloud global descriptor, and uses a hash table to store the 3D point cloud global descriptor, thus obtaining the 3D point cloud global descriptor construction method.

[0155] The initial pose matching module 54 is used to extract 3D point cloud descriptors from the static structural point cloud data and the semantic point cloud map using the 3D point cloud global descriptor construction method, and encode and coarsely match them to obtain a coarse pose and an optimal point cloud. The improved semantic iterative nearest point algorithm is then used to iteratively optimize the coarse pose to obtain the accurate initial pose of the mobile construction robot.

[0156] Furthermore, such as Figure 6 As shown, based on the above-mentioned indoor pose initialization method and system for mobile construction robots, the present invention also provides a terminal, which includes a processor 10, a memory 20 and a display 30. Figure 6 Only some of the terminal components are shown; however, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0157] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory. In other embodiments, the memory 20 may be an external storage device of the terminal, such as a plug-in hard disk, SmartMedia Card (SMC), Secure Digital (SD) card, or Flash Card. Further, the memory 20 may include both internal and external storage devices. The memory 20 is used to store application software and various types of data installed on the terminal, such as program code installed on the terminal. The memory 20 can also be used to temporarily store data that has been output or will be output. In one embodiment, the memory 20 stores an indoor pose initialization program 40 for a mobile construction robot, which can be executed by the processor 10 to implement the indoor pose initialization method for the mobile construction robot in this application.

[0158] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor, or other data processing chip, used to run program code stored in the memory 20 or process data, such as executing the indoor pose initialization method of the mobile construction robot.

[0159] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display 30 is used to display information on the terminal and to display a visual user interface. The components of the terminal communicate with each other via a system bus.

[0160] In one embodiment, when the processor 10 executes the indoor pose initialization program 40 for the mobile construction robot in the memory 20, the following steps are performed:

[0161] Collect image data and point cloud data of the pose of the mobile construction robot when it is powered on, and perform point cloud sampling and annotation on a pre-set BIM model to obtain a semantic point cloud map.

[0162] The static structure is segmented from the image data. Based on the spatial relationship between the camera and the LiDAR, the point cloud data is associated with the semantics of the 2D image to obtain point cloud data with assigned semantics. The point cloud data with assigned semantics is then filtered based on the static structure to obtain static structure point cloud data.

[0163] Key points are extracted from the static structural point cloud data or the semantic point cloud map. A 3D point cloud global descriptor is constructed using the key points. The 3D point cloud global descriptor is stored in a hash table to obtain the 3D point cloud global descriptor construction method.

[0164] The 3D point cloud global descriptor construction method is used to extract 3D point cloud descriptors from the static structural point cloud data and the semantic point cloud map, respectively, and encode and coarsely match them to obtain a coarse pose and an optimal point cloud. The improved semantic iterative nearest point algorithm is used to iteratively optimize the coarse pose to obtain the precise initial pose of the mobile construction robot.

[0165] The acquisition of image data and point cloud data of the mobile construction robot's pose at startup also previously included:

[0166] A lidar-camera perception system is pre-deployed on the mobile construction robot body, and the extrinsic parameters of the lidar-camera perception system are pre-calibrated using the Kalibr algorithm;

[0167] The LiDAR-camera sensing system is deployed in the same direction as the mobile construction robot's forward movement.

[0168] Specifically, the step of sampling and labeling point clouds on a pre-set BIM model to obtain a semantic point cloud map includes:

[0169] The pre-set BIM model is converted from RVT format to IFC format. Static structural elements are extracted from the IFC format BIM model according to IFC category, and the static structural elements are saved as structure category files in OBJ format.

[0170] The types of static structural elements include columns, beams, walls, doors, windows, and stairs;

[0171] Point cloud sampling is performed on the structure category files to obtain PCD point cloud files of different categories, and the PCD point cloud files of different categories are labeled.

[0172] All labeled point cloud pcd files are merged into a complete point cloud with categories, and this point cloud is used as the semantic point cloud map.

[0173] Specifically, the step of segmenting the static structure from the image data includes:

[0174] Construct a joint detection-segmentation model, which includes the YOLO-World object detection model and the MobileSAM zero-shot segmentation model;

[0175] The YOLO-World target detection model was quantized using FP16 mixed precision using the TensorRT tool to obtain a quantized detection model, which was then deployed on an edge device.

[0176] The image data is input into the quantized detection model to obtain the target location. The target location is then used as a segmentation cue and input into the MobileSAM zero-shot segmentation model to obtain the static structure semantic mask of the static structure in the image.

[0177] Specifically, the step of associating the point cloud data with 2D image semantics based on the spatial relationship between the camera and the LiDAR to obtain point cloud data with assigned semantic categories, and then filtering the point cloud data with assigned semantic categories according to the static structure to obtain static structured point cloud data, includes:

[0178] Based on the camera intrinsic parameters and radar-camera extrinsic parameters, the point cloud data is projected onto a 2D pixel coordinate system to obtain projection points, and the semantic mask of the projection points is calculated using a bilinear interpolation algorithm.

[0179] Based on the semantic mask of the projection points, the 3D coordinates of the point cloud data are assigned category semantics to obtain point cloud data with assigned category semantics.

[0180] Based on the static structure semantic mask, point clouds with static structure categories in the point cloud data after being assigned category semantics are filtered to obtain static structure point cloud data.

[0181] Specifically, the step of extracting key points from the static structured point cloud data or the semantic point cloud map, constructing a 3D point cloud global descriptor using the key points, and storing the 3D point cloud global descriptor using a hash table includes:

[0182] The intrinsic shape signature algorithm is used to extract multiple key points from the static structural point cloud data or the semantic point cloud map, construct a reference plane, and divide the reference plane into multiple pixel regions according to the preset side length.

[0183] Each key point is projected onto a certain pixel region, and a 3D point cloud global descriptor is constructed based on the principle of triangle invariance.

[0184] The global descriptor of the 3D point cloud is stored using a key-value pair-based hash table;

[0185] The 3D point cloud global descriptor includes a keypoint binary descriptor and a triangle descriptor. The keypoint binary descriptor includes a height descriptor and a category semantic descriptor. The height descriptor is used to encode the local geometric structure of the keypoint, and the category semantic descriptor is used to encode the semantics of the keypoint and the semantic proportion of the keypoint in its nearest neighbors. The vertices of the triangle descriptor are composed of keypoints of three different semantic categories. The side lengths of the triangle descriptors are arranged in ascending order of length. The key of the hash table is calculated by a hash function and the side lengths of the triangles. The index position of the key is a container that stores similar triangle descriptors.

[0186] Specifically, the step of using the 3D point cloud global descriptor construction method to extract 3D point cloud descriptors from the static structured point cloud data and the semantic point cloud map, respectively, for encoding and coarse matching to obtain a coarse pose and an optimal point cloud, includes:

[0187] The 3D point cloud global descriptor construction method is used to extract descriptors from the static structure point cloud data to obtain a candidate descriptor set, and the 3D point cloud global descriptor construction method is used to extract descriptors from the semantic point cloud map to form a query descriptor set;

[0188] The key value of each query descriptor in the query descriptor set is obtained by using a hash function and the side length of a triangle. Based on all the key values, the candidate descriptor corresponding to each query descriptor in the candidate descriptor set is obtained.

[0189] Calculate the binary descriptor similarity between all the query descriptors and their respective candidate descriptors, and compare all the binary descriptor similarities with a first preset threshold to obtain multiple target binary descriptor similarities that satisfy the preset threshold;

[0190] The candidate descriptors corresponding to the similarity of the multiple target binary descriptors are used as target candidate descriptors, and the static structural point cloud data fragments corresponding to each of the multiple target candidate descriptors are used as the initial screening target point clouds.

[0191] The number of target candidate descriptors in the multiple preliminary screening target point clouds is compared with a second preset threshold, and the preliminary screening target point cloud with the number of target candidate descriptors exceeding the second preset threshold is taken as the target point cloud;

[0192] Obtain the triangle plane of the query descriptor corresponding to a certain target point cloud, and obtain the target triangle plane that matches the target point cloud based on the triangle plane;

[0193] A point pair matching relationship is established based on the stated triangle plane and the target triangle plane. Based on the point pair matching relationship, a coarse pose between the stated triangle plane and the target triangle plane is obtained by singular value decomposition.

[0194] Using the coarse pose, the number of planes that overlap between the triangular plane in each query point cloud and the target triangular plane in the target point cloud is calculated, resulting in multiple plane overlap numbers. The target point cloud with the largest plane overlap number is selected as the optimal point cloud.

[0195] Specifically, the step of using the improved semantic iterative nearest-point algorithm to iteratively optimize the coarse pose to obtain the precise initial pose of the mobile construction robot includes:

[0196] The kD tree is used to obtain the corresponding point in the optimal point cloud for each point in the query descriptor subset, and a matching point pair is established based on each point in the query descriptor subset and the corresponding point.

[0197] Calculate the category semantic consistency of all the matching point pairs, assign weights to all the matching point pairs based on the category semantic consistency, and calculate the normal vectors of all points in the query descriptor subset and the optimal point cloud.

[0198] Based on the multiple weights and multiple normal vectors, the optimal rotation matrix and the optimal translation matrix are calculated, and the precise initial pose of the mobile construction robot is obtained based on the optimal rotation matrix and the optimal translation matrix.

[0199] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores an indoor pose initialization program for a mobile construction robot, and when the indoor pose initialization program for the mobile construction robot is executed by a processor, it implements the steps of the indoor pose initialization method for the mobile construction robot as described above.

[0200] In summary, this invention proposes an indoor pose initialization method, system, and terminal for a mobile construction robot. The method includes: acquiring image data and point cloud data of the pose of the mobile construction robot when it is powered on, and sampling and annotating the point cloud on a pre-set BIM model to obtain a semantic point cloud map; segmenting the static structure from the image data, associating the point cloud data with the semantics of the 2D image to obtain point cloud data with assigned semantic categories, filtering the point cloud data with assigned semantic categories based on the static structure to obtain static structure point cloud data; extracting key points from the static structure point cloud data, constructing a 3D point cloud global descriptor using the key points, and storing the 3D point cloud global descriptor using a hash table; using the 3D point cloud global descriptor to perform coarse matching on the static structure point cloud data and the semantic point cloud map to obtain the optimal point cloud; using the semantic point cloud map to improve the iterative nearest point algorithm, and using the improved iterative nearest point algorithm to iteratively optimize the optimal point cloud to obtain the precise initial pose of the mobile construction robot. This invention effectively solves the matching failure problem of traditional methods in weak texture and dynamic interference environments, breaks through the dependence of existing technology on motion trajectory, realizes rapid and accurate initialization of arbitrary initial pose, and provides accurate initial coordinates for autonomous navigation of construction robots.

[0201] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal that includes that element.

[0202] Of course, those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0203] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. An indoor pose initialization method for a mobile construction robot, characterized in that, The indoor pose initialization method for the mobile construction robot includes: Collect image data and point cloud data of the pose of the mobile construction robot when it is powered on, and perform point cloud sampling and annotation on a pre-set BIM model to obtain a semantic point cloud map. The static structure is segmented from the image data. Based on the spatial relationship between the camera and the LiDAR, the point cloud data is associated with the semantics of the 2D image to obtain point cloud data with assigned semantics. The point cloud data with assigned semantics is then filtered based on the static structure to obtain static structure point cloud data. Key points are extracted from the static structural point cloud data or the semantic point cloud map. A 3D point cloud global descriptor is constructed using the key points. The 3D point cloud global descriptor is stored in a hash table to obtain the 3D point cloud global descriptor construction method. The 3D point cloud global descriptor construction method is used to extract 3D point cloud descriptors from the static structural point cloud data and the semantic point cloud map, respectively, and encode and coarsely match them to obtain a coarse pose and an optimal point cloud. Based on the static structural point cloud and the optimal point cloud, the improved semantic iterative nearest point algorithm is used to iteratively optimize the coarse pose to obtain the accurate initial pose of the mobile construction robot.

2. The indoor pose initialization method for a mobile construction robot according to claim 1, characterized in that, The acquisition of image data and point cloud data of the mobile construction robot's pose at startup also includes, prior to: A lidar-camera perception system is pre-deployed on the mobile construction robot body, and the extrinsic parameters of the lidar-camera perception system are pre-calibrated using the Kalibr algorithm; The LiDAR-camera sensing system is deployed in the same direction as the mobile construction robot's forward movement.

3. The indoor pose initialization method for a mobile construction robot according to claim 1, characterized in that, The step of sampling and labeling point clouds on a pre-set BIM model to obtain a semantic point cloud map specifically includes: The pre-set BIM model is converted from RVT format to IFC format. Static structural elements are extracted from the IFC format BIM model according to IFC category, and the static structural elements are saved as structure category files in OBJ format. The types of static structural elements include columns, beams, walls, doors, windows, and stairs; Point cloud sampling is performed on the structure category files to obtain PCD point cloud files of different categories, and the PCD point cloud files of different categories are labeled. All labeled point cloud pcd files are merged into a complete point cloud with categories, and this point cloud is used as the semantic point cloud map.

4. The indoor pose initialization method for a mobile construction robot according to claim 1, characterized in that, The step of segmenting the static structure from the image data specifically includes: Construct a joint detection-segmentation model, which includes the YOLO-World object detection model and the MobileSAM zero-shot segmentation model; The YOLO-World target detection model was quantized using FP16 mixed precision using the TensorRT tool to obtain a quantized detection model, which was then deployed on an edge device. The image data is input into the quantized detection model to obtain the target location. The target location is then used as a segmentation cue and input into the MobileSAM zero-shot segmentation model to obtain the static structure semantic mask of the static structure in the image.

5. The indoor pose initialization method for a mobile construction robot according to claim 4, characterized in that, The step involves associating the point cloud data with 2D image semantics based on the spatial relationship between the camera and the LiDAR to obtain point cloud data with assigned semantic categories. Then, the point cloud data with assigned semantic categories is filtered according to the static structure to obtain static structured point cloud data. Specifically, this includes: Based on the camera intrinsic parameters and radar-camera extrinsic parameters, the point cloud data is projected onto a 2D pixel coordinate system to obtain projection points, and the semantic mask of the projection points is calculated using a bilinear interpolation algorithm. Based on the semantic mask of the projection points, the 3D coordinates of the point cloud data are assigned category semantics to obtain point cloud data with assigned category semantics. Based on the static structure semantic mask, point clouds with static structure categories in the point cloud data after being assigned category semantics are filtered to obtain static structure point cloud data.

6. The indoor pose initialization method for a mobile construction robot according to claim 1, characterized in that, The step of extracting key points from the static structured point cloud data or the semantic point cloud map, constructing a 3D point cloud global descriptor using the key points, and storing the 3D point cloud global descriptor using a hash table specifically includes: The intrinsic shape signature algorithm is used to extract multiple key points from the static structural point cloud data or the semantic point cloud map, construct a reference plane, and divide the reference plane into multiple pixel regions according to the preset side length. Each key point is projected onto a certain pixel region, and a 3D point cloud global descriptor is constructed based on the principle of triangle invariance. The global descriptor of the 3D point cloud is stored using a key-value pair-based hash table; The 3D point cloud global descriptor includes a keypoint binary descriptor and a triangle descriptor. The keypoint binary descriptor includes a height descriptor and a category semantic descriptor. The height descriptor is used to encode the local geometric structure of the keypoint, and the category semantic descriptor is used to encode the semantics of the keypoint and the semantic proportion of the keypoint in its nearest neighbors. The vertices of the triangle descriptor are composed of keypoints of three different semantic categories. The side lengths of the triangle descriptors are arranged in ascending order of length. The key of the hash table is calculated by a hash function and the side lengths of the triangles. The index position of the key is a container that stores similar triangle descriptors.

7. The indoor pose initialization method for a mobile construction robot according to claim 6, characterized in that, The step of using the 3D point cloud global descriptor construction method to extract 3D point cloud descriptors from the static structural point cloud data and the semantic point cloud map, respectively, and then encoding and coarsely matching them to obtain a coarse pose and an optimal point cloud, specifically includes: The 3D point cloud global descriptor construction method is used to extract descriptors from the static structure point cloud data to obtain a candidate descriptor set, and the 3D point cloud global descriptor construction method is used to extract descriptors from the semantic point cloud map to form a query descriptor set; The key value of each query descriptor in the query descriptor set is obtained by using a hash function and the side length of a triangle. Based on all the key values, the candidate descriptor corresponding to each query descriptor in the candidate descriptor set is obtained. Calculate the binary descriptor similarity between all the query descriptors and their respective candidate descriptors, and compare all the binary descriptor similarities with a first preset threshold to obtain multiple target binary descriptor similarities that satisfy the preset threshold; The candidate descriptors corresponding to the similarity of the multiple target binary descriptors are used as target candidate descriptors, and the static structural point cloud data fragments corresponding to each of the multiple target candidate descriptors are used as the initial screening target point clouds. The number of target candidate descriptors in the multiple preliminary screening target point clouds is compared with a second preset threshold, and the preliminary screening target point cloud with the number of target candidate descriptors exceeding the second preset threshold is taken as the target point cloud; Obtain the triangle plane of the query descriptor corresponding to a certain target point cloud, and obtain the target triangle plane that matches the target point cloud based on the triangle plane; A point pair matching relationship is established based on the stated triangle plane and the target triangle plane. Based on the point pair matching relationship, a coarse pose between the stated triangle plane and the target triangle plane is obtained by singular value decomposition. Using the coarse pose, the number of planes that overlap between the triangular plane in each query point cloud and the target triangular plane in the target point cloud is calculated, resulting in multiple plane overlap numbers. The target point cloud with the largest plane overlap number is selected as the optimal point cloud.

8. The indoor pose initialization method for a mobile construction robot according to claim 7, characterized in that, Based on the static structural point cloud and the optimal point cloud, the step of iteratively optimizing the coarse pose using the improved semantic iterative nearest point algorithm to obtain the precise initial pose of the mobile construction robot specifically includes: The kD-tree is used to obtain the corresponding point in the optimal point cloud for each point in the query descriptor subset, and a matching point pair is established based on each point in the query descriptor subset and the corresponding point. Calculate the category semantic consistency of all the matching point pairs, assign weights to all the matching point pairs based on the category semantic consistency, and calculate the normal vectors of all points in the query descriptor subset and the optimal point cloud. Based on the multiple weights and multiple normal vectors, the optimal rotation matrix and the optimal translation matrix are calculated, and the precise initial pose of the mobile construction robot is obtained based on the optimal rotation matrix and the optimal translation matrix.

9. An indoor pose initialization system for a mobile construction robot, characterized in that, The indoor pose initialization system of the mobile construction robot includes: The data acquisition and map creation module is used to collect image data and point cloud data of the mobile construction robot's pose when it is powered on, and to perform point cloud sampling and annotation on a pre-set BIM model to obtain a semantic point cloud map. The static structure point cloud data acquisition module is used to segment the static structure from the image data, associate the point cloud data with the semantics of the 2D image according to the spatial relationship between the camera and the lidar, obtain point cloud data with assigned semantics, and filter the point cloud data with assigned semantics according to the static structure to obtain static structure point cloud data. The global descriptor construction module extracts key points from the static structured point cloud data or the semantic point cloud map, uses the key points to construct a 3D point cloud global descriptor, and uses a hash table to store the 3D point cloud global descriptor, thus obtaining the 3D point cloud global descriptor construction method. The initial pose matching module is used to extract 3D point cloud descriptors from the static structural point cloud data and the semantic point cloud map using the 3D point cloud global descriptor construction method, and to encode and coarsely match them to obtain a coarse pose and an optimal point cloud. The improved semantic iterative nearest point algorithm is then used to iteratively optimize the coarse pose to obtain the accurate initial pose of the mobile construction robot.

10. A terminal, characterized in that, The terminal includes: a memory, a processor, and an indoor pose initialization program for the mobile construction robot stored in the memory and executable on the processor. When the indoor pose initialization program for the mobile construction robot is executed by the processor, it implements the steps of the indoor pose initialization method for the mobile construction robot as described in any one of claims 1-8.

Images