Automatic inspection method and system for immersed tube prefabrication project based on point cloud and mechanical dog

Abstract

The invention relates to an immersed tube prefabrication engineering automatic inspection method and system based on a point cloud and a mechanical dog, and the method comprises the steps: generating an oblique photography model of a prefabrication site, and converting the oblique photography model into a point cloud model; performing grid segmentation on the point cloud model, determining a plurality of scanning sites according to a grid segmentation result, and performing path planning on the plurality of scanning sites to obtain an optimal path; transmitting the optimal path to a mechanical dog, so that the mechanical dog reaches each scanning site according to the optimal path, and collecting scanning point cloud data of the prefabricated immersed tube at each scanning site through a scanner carried by the mechanical dog; and converting the BIM model of the prefabricated immersed tube into model point cloud data, performing registration and alignment on the model point cloud data and the scanning point cloud data at each scanning site, and calculating the geometric deviation of the prefabricated immersed tube based on a registration and alignment result.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the technical field of inspection of civil precast components, and in particular, to an automated inspection method and system for immersed tube precast projects based on point clouds and robotic dogs. Background Art

[0002] Currently, at the construction site of traditional large precast components such as immersed tubes, the quality monitoring and inspection of immersed tubes mainly rely on manual means, such as the experience and subjective judgment of quality inspectors, which easily leads to missed inspections or misjudgments. The accuracy of the quality inspection results of immersed tubes is low, and the manual operation and recording, as well as the data collection and analysis process, are cumbersome, wasting manpower, and it is difficult to achieve automation and data sharing, resulting in low inspection efficiency. At the same time, the construction site of large precast components such as immersed tubes is usually a complex construction site and construction scenario. Due to site restrictions, it is difficult for personnel to reach relatively dangerous measuring points for construction measurement, which also leads to a decrease in the accuracy of inspection results and inspection efficiency, and low measurement safety. Summary of the Invention

[0003] To solve the above technical problems or at least partially solve the above technical problems, embodiments of the present disclosure provide an automated inspection method and system for immersed tube precast projects based on point clouds and robotic dogs.

[0004] In a first aspect, embodiments of the present disclosure provide an automated inspection method for immersed tube precast projects based on point clouds and robotic dogs, the method comprising: Generating an oblique photography model of the precast site, and converting the oblique photography model into a point cloud model; performing grid segmentation on the point cloud model, determining a plurality of scanning stations according to the grid segmentation result, and performing path planning on the plurality of scanning stations to obtain an optimal path; Transmitting the optimal path to the robotic dog, so that the robotic dog reaches each scanning station according to the optimal path, and collects scanned point cloud data of the precast immersed tube at each scanning station through a scanner carried by itself; Converting the BIM model of the precast immersed tube into model point cloud data, registering and aligning the model point cloud data with the scanned point cloud data at each scanning station, and calculating the geometric deviation of the precast immersed tube based on the registration and alignment result.

[0005] In one embodiment, the converting the oblique photography model into a point cloud model includes: Reconstructing the oblique photography model into a three-dimensional grid model, and sampling the three-dimensional grid model to convert the three-dimensional grid model into a point cloud model.

[0006] In one embodiment, the method further includes: Detect the point cloud density in the point cloud model; when it is determined that the point cloud density is less than or equal to the preset point cloud density, resample the three-dimensional grid model so that the point cloud density of the transformed point cloud model is greater than the preset point cloud density.

[0007] In one embodiment, the method of performing grid segmentation on the point cloud model and determining multiple scanning stations according to the grid segmentation result includes: Set a preset grid size to divide the point cloud in the point cloud model into multiple sub-regions according to the spatial range, and each sub-region corresponds to a grid; Select the grid center point in each grid as multiple candidate stations; optimize the multiple candidate stations based on the scanning coverage rate of the prefabricated site at the position of each candidate station to remove redundant candidate stations to obtain multiple scanning stations.

[0008] In one embodiment, the method further includes: When it is determined that the path planning is the first planning after the first component of the prefabricated immersed tube is completed, transmit the point cloud map to the robotic dog, so that the robotic dog navigates to the designated station based on the position of each scanning station marked in the point cloud map, and at the designated station, obtain the accurate coordinate position of the designated station through the positioning system carried by itself, so as to determine the accurate coordinate position of each scanning station; wherein the point cloud map is determined by the point cloud model, and the designated station is any one of the multiple scanning stations; When controlling the robotic dog to perform multi-point navigation to the target station based on the accurate coordinate position of each scanning station, after calibrating the position of the robotic dog based on the accurate coordinate position of the target station, set the residence time of the target station, and control the scanner to collect the scanned point cloud data of the prefabricated immersed tube based on the residence time.

[0009] In one embodiment, the method further includes: When it is determined that the path planning is not the first planning after the first component of the prefabricated immersed tube is completed, skip the step of transmitting the point cloud map to the robotic dog, and enable the robotic dog to directly navigate to the designated station.

[0010] In one embodiment, the method further includes: Denoise the scanned point cloud data at each scanning station, and then extract key feature points; judge whether it is the first time to complete the point cloud scanning of the first component of the prefabricated immersed tube based on the key feature points; If so, register and align the model point cloud data with the scanned point cloud data at each scanning station after denoising, and calculate the geometric deviation of the components of the prefabricated immersed tube based on the registration and alignment result; Otherwise, based on the target scanned point cloud data and the scanned point cloud data at each scanned site after denoising processing, perform point cloud stitching to obtain updated scanned point cloud data; perform registration alignment based on the updated scanned point cloud data and the model point cloud data, and calculate the geometric deviation of the precast immersed tube based on the registration alignment result; wherein the target scanned point cloud data is the scanned point cloud data of the next component after the first component is manufactured.

[0011] In one embodiment, the method further includes: Obtain time series data obtained from multiple point cloud scans, where the time series data includes the geometric deviation of the precast immersed tube obtained after each point cloud scan; Input the time series data into an LSTM model trained in advance by machine learning to obtain the predicted deviation value at a future time point, and determine whether the deviation exceeds the tolerance range based on the predicted deviation value; if so, generate information for prompting to adjust the construction process.

[0012] In a second aspect, an embodiment of the present disclosure provides an automated inspection system for precast immersed tube engineering based on point cloud and robotic dog, including: A path planning module, configured to generate an oblique photography model of the precast site, convert the oblique photography model into a point cloud model; perform grid segmentation on the point cloud model, determine multiple scanned sites according to the grid segmentation result, and perform path planning on the multiple scanned sites to obtain an optimal path; A point cloud acquisition module, configured to transmit the optimal path to the robotic dog, so that the robotic dog reaches each scanned site according to the optimal path, and collect the scanned point cloud data of the precast immersed tube at each scanned site through a scanner carried by itself; A deviation calculation module, configured to convert the BIM model of the precast immersed tube into model point cloud data, perform registration alignment on the model point cloud data and the scanned point cloud data at each scanned site, and calculate the geometric deviation of the precast immersed tube based on the registration alignment result.

[0013] In a third aspect, an embodiment of the present disclosure provides an electronic device, including: A processor; and A memory for storing a computer program; Wherein, the processor is configured to execute the automated inspection method for precast immersed tube engineering based on point cloud and robotic dog in any of the above embodiments by executing the computer program.

[0014] The technical solutions provided by the embodiments of the present disclosure have the following advantages compared with the prior art: The automatic inspection method and system for the precast immersed tube project based on point cloud and robotic dog provided by the embodiments of the present disclosure generate an oblique photography model of the precast site, and convert the oblique photography model into a point cloud model; perform grid segmentation on the point cloud model, determine multiple scanning stations according to the grid segmentation result, and perform path planning on the multiple scanning stations to obtain an optimal path; transmit the optimal path to the robotic dog, so that the robotic dog reaches each scanning station according to the optimal path, and collect the scanned point cloud data of the precast immersed tube at each scanning station through the scanner carried by itself; convert the BIM model of the precast immersed tube into model point cloud data, register and align the model point cloud data with the scanned point cloud data at each scanning station, and calculate the geometric deviation of the precast immersed tube based on the registration and alignment result. In the solution of this embodiment, a point cloud model is generated through an oblique photography model. The scanning stations can be accurately determined by grid segmentation of the point cloud model. Based on this, the optimal path for robotic dog navigation is generated through path planning. The component point cloud during the precast process of the immersed tube is collected by using the 3D laser scanner carried by the robotic dog, and registered with the model point cloud converted from the BIM model to accurately measure the geometric deviation of the immersed tube during the precast process. This solution can significantly improve the automation measurement efficiency and accuracy of the geometric deviation of the immersed tube components during inspection in the complex construction environment of large components such as immersed tubes, realize the whole-process dynamic monitoring of large precast components such as immersed tubes, realize an automated, less or unmanned intervention monitoring process, make up for the problems of low data accuracy, low efficiency, and limited by site factors in traditional manual monitoring, reduce the intervention of construction personnel in monitoring, improve the safety of construction measurement, and provide a more efficient, accurate, and safe solution for engineering projects. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings herein are incorporated into the specification and constitute a part of this specification, showing embodiments consistent with the present disclosure and, together with the specification, are used to explain the principles of the present disclosure.

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art, the following will briefly introduce the accompanying drawings required for use in the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, other drawings can also be obtained based on these drawings without creative efforts.

[0017] Figure 1 It is a flowchart of the automatic inspection method for the precast immersed tube project based on point cloud and robotic dog according to the embodiments of the present disclosure; Figure 2 It is a flowchart of the automatic inspection method for the precast immersed tube project based on point cloud and robotic dog according to another embodiment of the present disclosure; Figure 3 It is a schematic diagram of the automatic inspection system for the precast immersed tube project based on point cloud and robotic dog according to the embodiments of the present disclosure; Figure 4 Schematic diagram of an electronic device according to an embodiment of the present disclosure. Detailed implementation manners

[0018] In order to be able to more clearly understand the above-mentioned objects, features, and advantages of the present disclosure, the solution of the present disclosure will be further described below. It should be noted that, without conflict, the embodiments of the present disclosure and the features in the embodiments may be combined with each other.

[0019] In the following description, many specific details are set forth in order to fully understand the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only a part of the embodiments of the present disclosure, rather than all the embodiments.

[0020] It should be understood that in the following text, "at least one (item)" means one or more, and "a plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" may mean: only A exists, only B exists, and both A and B exist at the same time. Among them, A and B may be singular or plural. The character " / " generally means that the associated objects before and after are in an "or" relationship. "At least one (one) of the following" or its similar expression means any combination of these items, including any combination of single item (one) or plural items (ones). For example, at least one (one) of a, b, or c may mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c may be single or plural.

[0021] Figure 1 Flowchart of an automatic inspection method for immersed tube prefabrication project based on point cloud and robotic dog according to an embodiment of the present disclosure. This method can be executed by a computing device such as a computer or a mobile terminal, and specifically may include the following steps: Step S101: Generate an oblique photography model of the prefabrication site, convert the oblique photography model into a point cloud model; perform grid segmentation on the point cloud model, determine multiple scanning stations according to the grid segmentation result, and perform path planning on the multiple scanning stations to obtain an optimal path.

[0022] Exemplarily, the prefabrication site can be the site of a prefabrication factory, which can include components of prefabricated immersed tubes and environmental information such as background and obstacle information. Each immersed tube can include multiple components, and the multiple components are prefabricated in sequence one by one. After each component is prefabricated, three-dimensional scanning can be performed to obtain scanned point cloud data. Before that, it is necessary to accurately plan to obtain the optimal path for the legged robot to walk. The Oblique Photogrammetry Model is a three-dimensional model generated using aerial images taken at an oblique angle. This technology obtains high-precision images of the ground surface and the target object by taking pictures from multiple angles, and combines these images to generate a high-precision three-dimensional digital model through photogrammetry technology.

[0023] In this embodiment, a drone can be used to take oblique images of the prefabrication site to be operated from multiple angles, and an oblique photogrammetry model is generated based on the multi-angle images covering the entire site. The specific generation process can be understood with reference to the prior art. Then, the oblique photogrammetry model is converted into a point cloud model. As shown in Figure 2 , the point cloud model is continuously divided into grids. For example, Delaunay triangulation or Poisson surface reconstruction is performed on the point cloud to generate a grid model, and multiple scanning stations are determined by screening according to the grid division result such as the grid model. For example, the density of scanning stations can be increased in complex terrain areas and decreased in flat areas. Then, path planning is performed on the multiple screened scanning stations. For example, the RRT (Rapidly-exploring Random Tree) algorithm is used for path planning, and the positions of the scanning stations are marked by considering the terrain complexity and obstacle distribution to obtain the global optimal path.

[0024] In this embodiment, the stations are not directly determined using the oblique photogrammetry model. The advantage is that after the point cloud is processed into grids, the data volume is significantly reduced, reducing the computational complexity of subsequent processing, and thus improving the automation measurement efficiency of the geometric deviation of the inspected immersed tube components. At the same time, grid division can better retain key geometric information, improve the positioning accuracy of subsequent scanning stations, and thus make the planned optimal path more accurate, make the collected scanned point cloud data more accurate, and ultimately improve the automation measurement accuracy of the geometric deviation of the inspected immersed tube components.

[0025] Step S102: Transmit the optimal path to the legged robot so that the legged robot reaches each scanning station according to the optimal path and collects the scanned point cloud data of the prefabricated immersed tube at each scanning station through the scanner carried by itself.

[0026] Exemplarily, a quadruped robot, as a four-legged mobile robot, has flexible movement ability and stable load capacity, and can replace manual labor to perform inspection tasks in complex, dangerous or inaccessible environments. After determining the optimal path in this embodiment, it can be transmitted to the quadruped robot, and wireless communication such as 4G or 5G communication can be used between the quadruped robot and the computing device. The quadruped robot walks along the optimal path to reach each scanning site, and at each scanning site, it collects the scanned point cloud data of the precast immersed tube through the scanner carried by itself. The scanner can be a three-dimensional laser scanner. Three-dimensional laser scanning can obtain the accurate three-dimensional coordinates, shape and surface features of an object. In this embodiment, the accurate scanned point cloud data of the precast immersed tube can be collected by using the three-dimensional laser scanner carried by the quadruped robot.

[0027] Step S103: Convert the BIM model of the precast immersed tube into model point cloud data, register and align the model point cloud data with the scanned point cloud data at each scanning site, and calculate the geometric deviation of the precast immersed tube based on the registration and alignment result.

[0028] Exemplarily, the BIM (Building Information Modeling) model belongs to the building information modeling technology and is the most advanced modeling method with the highest degree of digitalization in the current construction field. The BIM model contains rich geometric information of components, and the components of the target component such as the immersed tube contain fine digital information, which is used to assist in construction quality control. In this embodiment, the BIM model of the precast immersed tube is converted into model point cloud data, and then registered and aligned with the scanned point cloud data of the precast immersed tube obtained by scanning. For example, the random sample consensus algorithm RANSAC (Random Sample Consensus) is used for registration and alignment. Finally, the geometric deviation of the precast immersed tube is calculated based on the registration and alignment result. For example, the geometric deviation is calculated by the distance from a point to a plane. In some embodiments, a deviation heat map can be generated for visualization subsequently, and the out-of-tolerance areas can be marked.

[0029] In the above solution of this embodiment, a point cloud model is generated through an oblique photography model. When the point cloud model is meshed and segmented, the scanning station can be accurately determined. Accordingly, the optimal path for the navigation of the robotic dog is generated through path planning. The component point cloud during the precast process of the immersed tube is collected by using the 3D laser scanner carried by the robotic dog, and is registered with the model point cloud converted from the BIM model to accurately measure the geometric deviation of the immersed tube precast process. This solution can significantly improve the automation measurement efficiency and accuracy of the geometric deviation of the immersed tube components during inspection in the complex construction environment of large components such as immersed tubes, realize the whole-process dynamic monitoring of large precast components such as immersed tubes, realize an automated, less-manpower or unmanned intervention monitoring process, make up for the problems of low data accuracy, low efficiency, and being restricted by site factors in traditional manual monitoring, reduce the intervention of construction personnel in monitoring, improve the safety of construction measurement, and provide a more efficient, accurate, and safe solution for engineering projects.

[0030] On the basis of the above implementation, in one embodiment, converting the oblique photography model into a point cloud model in step S101 includes: reconstructing the oblique photography model into a 3D mesh model, and sampling the 3D mesh model to convert the 3D mesh model into a point cloud model.

[0031] Exemplarily, the Pix4Dmapper software can be used to reconstruct the oblique photography model into a 3D mesh model. The data of the 3D mesh model can be stored in the OBJ format. The 3D mesh model is imported into the CloudCompare processing software and the sampling tool is used to convert the 3D mesh model into a point cloud model. The point cloud model can be stored in the PCD format, but it is not limited to this.

[0032] It should be noted that directly generating a point cloud based on the oblique photography model is prone to edge blurring due to image distortion, while the reconstruction of the 3D mesh model can correct such distortion. The 3D mesh reconstruction process will also fuse multi-view texture information, reduce the perspective blind area problem during direct point cloud generation. At the same time, the 3D mesh model can retain the geometric topological relationship of the ground objects, and the sampled point cloud can inherit such structured features. During the reconstruction process of the 3D mesh model, the holes (such as occluded areas) in the original point cloud can be filled by connecting triangular patches. The spatial distribution of the sampled point cloud is more uniform, the quality of the point cloud is higher, and the accuracy is higher. Therefore, the solution of this embodiment provides a more stable and complete data basis for point cloud generation, making the generated point cloud model more accurate, so that the determined scanning station position is more accurate, and then the planned path is the most accurate, making the scanned point cloud data collected by the robotic dog more accurate, and further improving the automation measurement accuracy of the geometric deviation of the immersed tube components during inspection.

[0033] Based on the above embodiments, in one embodiment, the method further includes: detecting the point cloud density in the point cloud model; when determining that the point cloud density is less than or equal to a preset point cloud density, resampling the three-dimensional grid model so that the point cloud density of the converted point cloud model is greater than the preset point cloud density.

[0034] Exemplarily, the preset point cloud density can be set according to needs. By checking the point cloud density, it can be ensured that the resolution of the converted point cloud model is sufficient to cover the details of the site, making the generated point cloud model more accurate, thereby making the determined scanning site location more accurate, and further making the planned path most accurate, so that the scanned point cloud data collected by the legged robot is more accurate, and further improving the automatic measurement accuracy of the geometric deviation of the immersed tube components during inspection.

[0035] Based on the above implementation, in one embodiment, referring to Figure 2 as shown, in step S101, performing grid segmentation on the point cloud model, and determining a plurality of scanning sites according to the grid segmentation result, including: Setting a preset grid size to divide the point clouds in the point cloud model into multiple sub-regions according to the spatial range, and each sub-region corresponds to a grid; Selecting the grid center point in each grid as a plurality of candidate sites; optimizing the plurality of candidate sites based on the scanning coverage rate of the prefabricated site at the position of each candidate site to remove redundant candidate sites to obtain a plurality of scanning sites.

[0036] Exemplarily, the Point Cloud Library (PCL) can be used to perform grid segmentation on the point cloud model. Set a preset grid size to divide the point clouds in the point cloud model into multiple sub-regions according to the spatial range. Each sub-region, that is, the spatial region, corresponds to a grid, and each grid can contain multiple point clouds. Select the grid center point in each grid as a candidate site, generate candidate site coordinates, and use a coverage rate optimization algorithm (based on the laser scanner field of view model) to optimize the distribution of candidate sites. Check the scanning coverage rate of each candidate site for the site, and remove redundant sites such as candidate sites with a scanning coverage rate less than the preset value. In this way, the determined scanning site location can be made more accurate, and further the planned path can be made most accurate, so that the scanned point cloud data collected by the legged robot is more accurate, and further the automatic measurement accuracy of the geometric deviation of the immersed tube components during inspection can be improved.

[0037] Based on any of the above embodiments, in one embodiment, referring to Figure 2 as shown, the method may further include the following steps: When it is determined that the path planning is the first planning after the production of the first component of the precast immersed tube is completed, the point cloud map is transmitted to the robotic dog, so that the robotic dog navigates to the designated site based on the positions of each scanning site marked in the point cloud map, and obtains the precise coordinate position of the designated site through the positioning system carried by itself at the designated site, thereby determining the precise coordinate positions of each scanning site; wherein the point cloud map is determined by the point cloud model, and the designated site is any one of the multiple scanning sites; When controlling the robotic dog to perform multi-point navigation to the target site based on the precise coordinate positions of each scanning site, after calibrating the position of the robotic dog based on the precise coordinate position of the target site, set the residence time of the target site, and control the scanner to collect the scanned point cloud data of the precast immersed tube based on the residence time.

[0038] Exemplarily, the point cloud map can be a two-dimensional (2D) map. In this embodiment, a planning scan measurement can be performed once after the production of each component of the precast immersed tube is completed. Determine whether the first site planning after the production of the first component is completed. If so, the point cloud map that has been converted into a PCD file can be input to the map_server in the operating system ROS (Robot Operating System) of the robotic dog. Subsequently, the robotic dog uses the Navigation function to reach the designated site, that is, any one of the multiple scanning sites, according to the scanning site positions marked in the input point cloud map, and obtains the precise coordinate position of the designated site through the positioning system carried by itself, such as the Real Time Kinematic (RTK) module. In this way, the precise coordinate positions of each scanning site can be obtained.

[0039] After that, the scanning task is executed. First, perform the multi-point navigation function. Input the precise coordinate positions of all scanning sites into the multi-point navigation CPP file written in the robotic dog. Use the Simultaneous Localization and Mapping (SLAM) module carried by the robotic dog to perform real-time dynamic obstacle avoidance to reach the target site. When reaching the target site, perform position calibration in real time through RTK, and then ROS outputs the arrival at the target point. Set the residence time of the robotic dog from when the scanner starts to when the scanning is completed after reaching the target point. The length of the residence time considers the site coverage area and the completion time of the scanner. After the residence time reaches a certain threshold, activate the scanner to scan and collect the scanned point cloud data of the precast immersed tube. After the scanning time ends, start the navigation to the next site.

[0040] In this embodiment, by obtaining the accurate coordinate positions of all scanning stations, the station position calibration can be performed during the multi-point navigation of the robotic dog. Combining with setting a reasonable residence time, it ensures the integrity of the coverage of the scanned data and improves the detection accuracy of the scan, which can make the scanned point cloud data of the precast immersed tube collected more accurate, so that the calculation result of the geometric deviation after registration is more accurate, and further improves the automatic measurement accuracy of the geometric deviation of the inspected immersed tube components. In addition, based on multi-point navigation, multi-task execution can be carried out, which improves the data collection efficiency as a whole, and further improves the automatic measurement efficiency of the geometric deviation of the inspected immersed tube components.

[0041] Based on the above embodiment, in one embodiment, referring to Figure 2 As shown, the method further includes: when it is determined that the path planning is not the first planning after the first component of the precast immersed tube is completed, skip the step of transmitting the point cloud map to the robotic dog, and directly navigate the robotic dog to the specified station. The point cloud map has been transmitted to the robotic dog after the first component is completed, and there is no need to transmit the point cloud map during the subsequent scanning of the components. Just navigate and walk directly. That is, if it is not the first station planning after the completion of the production of the first component, directly use the Navigation function to navigate the robotic dog to the specified station and skip the step of inputting the point cloud map. Then the robotic dog executes the steps of determining the accurate coordinate position of each scanning station and subsequent multi-point navigation, controlling the scanned point cloud, etc.

[0042] In one embodiment, referring to Figure 2 As shown, the method may further include the following steps: Denoise the scanned point cloud data at each scanning station, and then extract key feature points; based on the key feature points, determine whether it is the first time to complete the point cloud scanning of the first component of the precast immersed tube; If so, register and align the model point cloud data with the scanned point cloud data at each scanning station after denoising, and calculate the geometric deviation of the components of the precast immersed tube based on the registration and alignment results; If not, perform point cloud stitching on the target scanned point cloud data and the scanned point cloud data at each scanning station after denoising to obtain the updated scanned point cloud data; register and align the updated scanned point cloud data with the model point cloud data, and calculate the geometric deviation of the components of the precast immersed tube based on the registration and alignment results; where the target scanned point cloud data is the scanned point cloud data of the next component after the completion of the production of the first component.

[0043] Exemplarily, first, the collected scanned point cloud data is denoised to clean redundant points and abnormal points. Then, the key feature points such as edge points and corner points in the denoised scanned point cloud data can be extracted using the feature extraction module of PCL. Subsequently, based on the key feature points, it is determined whether it is the point cloud scan of the first component of the precast immersed tube completed for the first time. If so, the RANSAC algorithm can be used to register and align the denoised scanned point cloud data with the model point cloud data, calculate the geometric deviation based on the registration and alignment results through the distance from point to plane, and then a deviation heat map can be generated for visualization to mark the out-of-tolerance area.

[0044] If it is not the point cloud scan of the first component completed for the first time but the scan of the subsequent next component (i.e., both the first component and the next component have been scanned), then point cloud stitching such as dynamic point cloud stitching is performed, and the scanned point cloud data of the next component is gradually stitched with the scanned point cloud data of the first component. For example, the incremental ICP algorithm is used to dynamically update the global point cloud of the component, and then the registration and alignment with the model point cloud data for BIM model conversion are performed, and the geometric deviation of the precast immersed tube component is calculated based on the registration and alignment results. A deviation heat map can also be generated for visualization to mark the out-of-tolerance area in one step.

[0045] Based on any one of the above embodiments, in one embodiment, referring to Figure 2 as shown, the method may further include the following steps: Obtain the time series data obtained from multiple point cloud scans, where the time series data includes the geometric deviation of the precast immersed tube obtained after each point cloud scan; Input the time series data into the pre-trained LSTM model of machine learning to obtain the predicted deviation value at a future time point, and judge whether the deviation exceeds the tolerance range according to the predicted deviation value; if so, generate information for prompting to adjust the construction process.

[0046] Exemplarily, in this embodiment, the time series data of multiple scans can be analyzed to extract the deviation gradient, and for the high-risk deviation areas, a trend curve can be generated. The LSTM (Long Short-Term Memory) model pre-trained by machine learning can also be used to predict the time series data, predict the deviation value at a future time point, and determine whether the deviation exceeds the tolerance range according to the prediction result, i.e., the predicted deviation value. If so, there is a serious problem, and information for prompting to adjust the construction process is generated to adjust the construction process and optimize the construction flow. Among them, the trained LSTM model is pre-trained based on the sample time series data for the original LSTM model, and it can accurately predict the deviation value at a future time point brought by the current scanning measurement method for the precast immersed tube inspection. For example, it can predict in advance the predicted deviation value of the next component, predict possible problems, and facilitate adjusting the construction process in advance to make the geometric deviation of the precast next component meet the standards and avoid rework.

[0047] Finally, the method of this embodiment can determine whether the scanning measurement after the precast of all components of the precast immersed tube is completed. If so, it ends; if not, it returns to the candidate point selection step to continue the scanning measurement process of the next component.

[0048] It should be noted that although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that these steps must be executed in this specific order, or that all the shown steps must be executed to achieve the desired result. Additionally or alternatively, some steps can be omitted, multiple steps can be combined into one step for execution, and / or one step can be decomposed into multiple steps for execution, etc. Also, it is easy to understand that these steps can be executed synchronously or asynchronously in, for example, multiple modules / processes / threads.

[0049] As Figure 3 shown, the embodiment of the present disclosure provides an automated inspection system for the immersed tube prefabrication project based on point cloud and mechanical dog, including: A path planning module 401, configured to generate an oblique photography model of the prefabrication site, convert the oblique photography model into a point cloud model; perform grid segmentation on the point cloud model, determine multiple scanning stations according to the grid segmentation result, and perform path planning on the multiple scanning stations to obtain an optimal path; A point cloud acquisition module 402, configured to transmit the optimal path to the mechanical dog, so that the mechanical dog reaches each scanning station according to the optimal path, and collect the scanned point cloud data of the precast immersed tube at each scanning station through the scanner carried by itself; The deviation calculation module 403 is configured to convert the BIM model of the precast immersed tube into model point cloud data, register and align the model point cloud data with the scanned point cloud data at each scanning site, and calculate the geometric deviation of the precast immersed tube based on the registration and alignment results.

[0050] In one embodiment, the path planning module 401 converts the oblique photography model into a point cloud model, including: reconstructing the oblique photography model into a three-dimensional grid model, and sampling the three-dimensional grid model to convert the three-dimensional grid model into a point cloud model.

[0051] In one embodiment, the system further includes a point cloud detection module, configured to: detect the point cloud density in the point cloud model; when it is determined that the point cloud density is less than or equal to a preset point cloud density, resample the three-dimensional grid model so that the point cloud density of the converted point cloud model is greater than the preset point cloud density.

[0052] In one embodiment, the path planning module 401 performs grid segmentation on the point cloud model and determines multiple scanning sites according to the grid segmentation result, including: Setting a preset grid size to divide the point cloud in the point cloud model into multiple sub-regions according to the spatial range, and each sub-region corresponds to a grid; Selecting the grid center point in each grid as multiple candidate sites; optimizing the multiple candidate sites based on the scanning coverage rate of the prefabrication site at the position of each candidate site to remove redundant candidate sites to obtain multiple scanning sites.

[0053] In one embodiment, the system further includes a navigation control module, configured to: when it is determined that the path planning is the first planning after the first component of the precast immersed tube is completed, transmit the point cloud map to the robotic dog, so that the robotic dog navigates to the designated site based on the position of each scanning site marked in the point cloud map, and obtains the accurate coordinate position of the designated site through the positioning system carried by itself at the designated site, so as to determine the accurate coordinate position of each scanning site; wherein the point cloud map is determined by the point cloud model, and the designated site is any one of the multiple scanning sites; when controlling the robotic dog to perform multi-point navigation to the target site based on the accurate coordinate position of each scanning site, after calibrating the position of the robotic dog based on the accurate coordinate position of the target site, set the residence time of the target site, and control the scanner to collect the scanned point cloud data of the precast immersed tube based on the residence time.

[0054] In one embodiment, the navigation control module is further configured to: when it is determined that the path planning is not the first planning after the first component of the precast immersed tube is completed, skip the step of transmitting the point cloud map to the robotic dog, and enable the robotic dog to directly navigate to the designated site.

[0055] In one embodiment, the system further includes a point cloud processing module for: denoising the scanned point cloud data at each scanning site, and then extracting key feature points; determining whether the point cloud scanning of the first component of the precast immersed tube is completed for the first time based on the key feature points; If so, the deviation calculation module 403 registers and aligns the model point cloud data with the scanned point cloud data at each scanning site after denoising, and calculates the geometric deviation of the components of the precast immersed tube based on the registration and alignment results; If not, the point cloud processing module stitches the point clouds based on the target scanned point cloud data and the scanned point cloud data at each scanning site after denoising to obtain updated scanned point cloud data; the deviation calculation module 403 registers and aligns the updated scanned point cloud data with the model point cloud data, and calculates the geometric deviation of the components of the precast immersed tube based on the registration and alignment results; wherein the target scanned point cloud data is the scanned point cloud data of the next component after the first component is manufactured.

[0056] In one embodiment, the system may further include an error prediction module for: Obtaining time series data obtained from multiple point cloud scans, where the time series data includes the geometric deviation of the precast immersed tube obtained after each point cloud scan; Inputting the time series data into a pre-trained LSTM model of machine learning to obtain a predicted deviation value at a future time point, and determining whether the deviation exceeds the tolerance range based on the predicted deviation value; if so, generating information for prompting to adjust the construction process.

[0057] Regarding the system in the above embodiments, the specific ways in which each module performs operations and the corresponding technical effects have been described in detail in the embodiments related to the method, and will not be elaborated here.

[0058] It should be noted that although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of the two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units. The components shown as modules or units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of the present disclosure. Those of ordinary skill in the art can understand and implement it without creative work.

[0059] Embodiments of the present disclosure also provide a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the automated inspection method for immersed tube prefabrication projects based on point cloud and robotic dog described in any of the above embodiments.

[0060] Exemplarily, the readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (non-exhaustive list) of the readable storage medium include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0061] The computer-readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries the readable program code. Such a propagated data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable storage medium can also be any readable medium other than the readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium can be transmitted using any appropriate medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination of the above.

[0062] Embodiments of the present disclosure also provide an electronic device, including a processor and a memory, where the memory is used to store a computer program. Among them, the processor is configured to execute the automated inspection method for immersed tube prefabrication projects based on point cloud and robotic dog in any of the above embodiments by executing the computer program.

[0063] Next, refer to Figure 4 to describe the electronic device 600 according to this embodiment of the present invention. Figure 4 The shown electronic device 600 is only an example and should not impose any limitations on the functions and usage scope of the embodiments of the present invention.

[0064] As Figure 4 shown, the electronic device 600 is presented in the form of a general-purpose computing device. The components of the electronic device 600 may include but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different system components (including the storage unit 620 and the processing unit 610), a display unit 640, etc.

[0065] Among them, the storage unit stores program codes, and the program codes can be executed by the processing unit 610, so that the processing unit 610 executes the steps according to various exemplary embodiments of the present invention described in the method embodiment part of the present specification above. For example, the processing unit 610 can execute steps such as Figure 1 shown in the method.

[0066] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 6201 and / or a cache storage unit 6202, and may further include a read-only storage unit (ROM) 6203.

[0067] The storage unit 620 may further include a program / utilities 6204 having a set (at least one) of program modules 6205. Such program modules 6205 include but are not limited to: an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include the implementation of a network environment.

[0068] The bus 630 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of a variety of bus structures.

[0069] The electronic device 600 can also communicate with one or more external devices 700 (such as a keyboard, a pointing device, a Bluetooth device, etc.), can also communicate with one or more devices that enable a user to interact with the electronic device 600, and / or communicate with any device that enables the electronic device 600 to communicate with one or more other computing devices (such as a router, a modem, etc.). Such communication can be carried out through the input / output (I / O) interface 650. And the electronic device 600 can also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through the network adapter 660. The network adapter 660 can communicate with other modules of the electronic device 600 through the bus 630. It should be understood that although not shown in the figure, other hardware and / or software modules can be used in combination with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0070] Through the description of the above embodiments, those skilled in the art can easily understand that the example embodiments described herein can be implemented by software or by a combination of software and necessary hardware. Therefore, the technical solutions according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (such as a personal computer, a server, or a network device, etc.) to execute the method steps of the above various embodiments according to the embodiments of the present disclosure.

[0071] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover a non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of additional identical elements in the process, method, article or device including the said element.

[0072] The above are only specific embodiments of the present disclosure, enabling those skilled in the art to understand or implement the present disclosure. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to these embodiments described herein, but rather to the broadest scope consistent with the principles and novel features disclosed herein.

Claims

1. An automated inspection method for immersed tube prefabrication engineering based on point cloud and mechanical dog, characterized in that: The method includes: Generate an oblique photography model of the prefabrication site, and convert the oblique photography model into a point cloud model; perform grid segmentation on the point cloud model, determine a plurality of scanning sites according to the grid segmentation result, and perform path planning on the plurality of scanning sites to obtain an optimal path; The optimal path is transmitted to the robot dog, so that the robot dog reaches each scanning site according to the optimal path, and collects scanning point cloud data of the prefabricated immersed tube at each scanning site through the scanner carried by the robot dog; The BIM model of the prefabricated immersed tube is converted into model point cloud data, the model point cloud data is aligned with the scanning point cloud data at each scanning site, and the geometric deviation of the prefabricated immersed tube is calculated based on the alignment result.

2. The method according to claim 1, characterized in that The step of converting the oblique photography model into a point cloud model comprises: The oblique photography model is reconstructed into a three-dimensional mesh model, and the three-dimensional mesh model is sampled to convert the three-dimensional mesh model into a point cloud model.

3. The method according to claim 2, characterized in that The method further includes: The point cloud density in the point cloud model is detected; when it is determined that the point cloud density is less than or equal to a preset point cloud density, the three-dimensional grid model is resampled so that the point cloud density of the converted point cloud model is greater than the preset point cloud density.

4. The method according to claim 1, characterized in that: The step of performing grid segmentation on the point cloud model and determining a plurality of scanning sites according to the grid segmentation result includes: Setting a preset grid size to divide the point cloud in the point cloud model into a plurality of sub-regions according to the spatial range, each sub-region corresponding to a grid; A grid center point is selected in each grid as a plurality of candidate sites; based on the scanning coverage of the prefabrication site by the position of each candidate site, the plurality of candidate sites are optimized to remove redundant candidate sites to obtain a plurality of scanning sites.

5. The method according to any one of claims 1 to 4, characterized in that: The method further includes: When it is determined that the path planning is the first planning after the first component of the prefabricated immersed tube is completed, the point cloud map is transmitted to the robot dog, so that the robot dog navigates to the designated site based on the position of each scanning site marked in the point cloud map, and obtains the precise coordinate position of the designated site through the positioning system carried by itself at the designated site, thereby determining the precise coordinate position of each scanning site; wherein the point cloud map is determined by the point cloud model, and the designated site is any one of the multiple scanning sites; When the robot dog is controlled to perform multi-point navigation to reach the target site based on the precise coordinate position of each scanning site, the position of the robot dog is calibrated based on the precise coordinate position of the target site, and the dwell time of the target site is set. Based on the dwell time, the scanner is controlled to collect scanning point cloud data of the prefabricated immersed tube.

6. The method according to claim 5, characterized in that The method further includes: When it is determined that the path planning is not the first planning after the first component of the prefabricated immersed tube is completed, the step of transmitting the point cloud map to the robot dog is skipped, so that the robot dog directly navigates to the designated site.

7. The method according to any one of claims 1 to 4, characterized in that: The method further includes: De-noising the scanned point cloud data at each scanning station, and then extracting key feature points; judging whether the point cloud scan of the first component of the prefabricated immersed tube is completed for the first time based on the key feature points; If yes, the model point cloud data is registered and aligned with the scan point cloud data at each scanning station after denoising, and the component geometric deviation of the prefabricated immersed tube is calculated based on the registration and alignment results; If not, point cloud stitching is performed based on the target scanning point cloud data and the scanning point cloud data at each scanning site after denoising to obtain updated scanning point cloud data; registration and alignment are performed based on the updated scanning point cloud data and the model point cloud data, and the component geometric deviation of the prefabricated immersed tube is calculated based on the registration and alignment results; wherein the target scanning point cloud data is the scanning point cloud data of the next component after the first component is completed.

8. The method according to any one of claims 1 to 4, characterized in that: The method further includes: Acquire time series data obtained by multiple point cloud scans, wherein the time series data includes geometric deviations of the prefabricated immersed tube obtained after each point cloud scan; The time series data is input into the LSTM model that has been pre-trained by machine learning to obtain the deviation value of the predicted future time point, and whether the deviation exceeds the tolerance range is determined based on the predicted deviation value; if so, information prompting adjustment of the construction process is generated.

9. An automated inspection system for immersed tube prefabrication engineering based on point cloud and mechanical dog, characterized in that: include: A path planning module, used to generate an oblique photography model of the prefabrication site and convert the oblique photography model into a point cloud model; Performing grid segmentation on the point cloud model, determining a plurality of scanning sites according to the grid segmentation result, and performing path planning on the plurality of scanning sites to obtain an optimal path; A point cloud acquisition module, used for transmitting the optimal path to the robot dog, so that the robot dog reaches each scanning station according to the optimal path, and acquires scanning point cloud data of the prefabricated immersed tube at each scanning station through a scanner carried by the robot dog; The deviation calculation module is used to convert the BIM model of the prefabricated immersed tube into model point cloud data, align the model point cloud data with the scanning point cloud data at each scanning site, and calculate the geometric deviation of the prefabricated immersed tube based on the alignment result.

10. An electronic device, characterized in that: include: processor; as well as Memory for storing computer programs; Wherein, the processor is configured to execute the automated inspection method for immersed tube prefabrication projects based on point cloud and mechanical dog as described in any one of claims 1 to 8 by executing the computer program.

Images