Automated inspection method and system for immersed tube prefabrication engineering based on point cloud and robotic dog

Abstract

The present disclosure relates to an automated inspection method and system for immersed tube prefabrication projects based on point clouds and a robotic dog, wherein the method comprises: generating an oblique photography model of the prefabrication site, and converting the oblique photography model into a point cloud model; gridding the point cloud model, determining a plurality of scanning sites according to the gridding segmentation result, and performing path planning for the plurality of scanning sites to obtain an optimal path; transmitting the optimal path to a robotic dog so that the robotic dog reaches each scanning site according to the optimal path, and collects scanning point cloud data of the prefabricated immersed tube at each scanning site through a scanner carried by the robotic dog; converting the BIM model of the prefabricated immersed tube into model point cloud data, registering and aligning the model point cloud data with the scanning point cloud data at each scanning site, and calculating the geometric deviation of the prefabricated immersed tube based on the registration and alignment results.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the technical field of inspection of civil prefabricated components, and in particular to an automated inspection method and system for immersed tube prefabrication projects based on point clouds and a robotic dog. Background Art

[0002] Currently, quality monitoring and inspection at traditional construction sites for large prefabricated components, such as immersed tubes, primarily rely on manual methods, such as the experience and subjective judgment of quality inspectors. This can easily lead to missed inspections or misjudgments, resulting in low accuracy of quality inspection results. Furthermore, manual operation and recording, data collection, and analysis are cumbersome, wasteful, and difficult to automate and share data, resulting in low inspection efficiency. Furthermore, construction sites for large prefabricated components, such as immersed tubes, often feature complex construction sites and scenarios. Due to site restrictions, it's difficult for personnel to reach more dangerous measurement points for construction measurements. This results in reduced accuracy and efficiency of inspection results, as well as low measurement safety. Summary of the Invention

[0003] In order to solve the above technical problems or at least partially solve the above technical problems, the embodiments of the present disclosure provide an automated inspection method and system for immersed tube prefabrication projects based on point clouds and a robotic dog.

[0004] In a first aspect, an embodiment of the present disclosure provides an automated inspection method for immersed tube prefabrication projects based on point clouds and a robotic dog, the method comprising:

[0005] Generate an oblique photography model of the prefabrication site, convert the oblique photography model into a point cloud model; perform grid segmentation on the point cloud model, determine multiple scanning sites based on the grid segmentation results, and perform path planning on the multiple scanning sites to obtain an optimal path;

[0006] Transmitting the optimal path to the robot dog so that the robot dog reaches each scanning station according to the optimal path and collects scanning point cloud data of the prefabricated immersed tube at each scanning station through the scanner carried by the robot dog;

[0007] The BIM model of the prefabricated immersed tube is converted into model point cloud data, the model point cloud data is aligned with the scan point cloud data at each scanning station, and the geometric deviation of the prefabricated immersed tube is calculated based on the registration and alignment results.

[0008] In one embodiment, converting the oblique photography model into a point cloud model includes:

[0009] The oblique photography model is reconstructed into a three-dimensional mesh model, and the three-dimensional mesh model is sampled to convert the three-dimensional mesh model into a point cloud model.

[0010] In one embodiment, the method further comprises:

[0011] Detecting the point cloud density in the point cloud model; and when determining that the point cloud density is less than or equal to a preset point cloud density, resampling the three-dimensional grid model so that the point cloud density of the converted point cloud model is greater than the preset point cloud density.

[0012] In one embodiment, the gridding and segmenting the point cloud model and determining a plurality of scanning sites according to the gridding and segmentation results include:

[0013] Setting a preset grid size to divide the point cloud in the point cloud model into a plurality of sub-regions according to the spatial range, each sub-region corresponding to a grid;

[0014] A grid center point is selected in each grid as a plurality of candidate sites; based on the scanning coverage of the prefabrication site by the position of each candidate site, the plurality of candidate sites are optimized to remove redundant candidate sites to obtain a plurality of scanning sites.

[0015] In one embodiment, the method further comprises:

[0016] When it is determined that the path planning is the first planning after the first component of the prefabricated immersed tube is completed, the point cloud map is transmitted to the robotic dog, so that the robotic dog navigates to a designated station based on the position of each scanning station marked in the point cloud map, and obtains the precise coordinate position of the designated station at the designated station through its own positioning system, thereby determining the precise coordinate position of each scanning station; wherein the point cloud map is determined by the point cloud model, and the designated station is any one of the multiple scanning stations;

[0017] When controlling the robotic dog to perform multi-point navigation to reach the target site based on the precise coordinate position of each scanning site, after calibrating the position of the robotic dog based on the precise coordinate position of the target site, set the stay time at the target site, and control the scanner to collect scanning point cloud data of the prefabricated immersed tube based on the stay time.

[0018] In one embodiment, the method further comprises:

[0019] When it is determined that the path planning is not the first planning after the first component of the prefabricated immersed tube is completed, the step of transmitting the point cloud map to the robotic dog is skipped, so that the robotic dog directly navigates to the designated site.

[0020] In one embodiment, the method further comprises:

[0021] De-noising the scanned point cloud data at each scanning station, and then extracting key feature points; judging based on the key feature points whether the point cloud scan of the first component of the prefabricated immersed tube has been completed for the first time;

[0022] If yes, register and align the model point cloud data with the scan point cloud data at each scanning station after denoising, and calculate the component geometric deviation of the prefabricated immersed tube based on the registration and alignment results;

[0023] If not, point cloud stitching is performed based on the target scanning point cloud data and the scanning point cloud data at each scanning site after denoising to obtain updated scanning point cloud data; registration and alignment are performed based on the updated scanning point cloud data and the model point cloud data, and the component geometric deviation of the prefabricated immersed tube is calculated based on the registration and alignment results; wherein the target scanning point cloud data is the scanning point cloud data of the next component after the first component is completed.

[0024] In one embodiment, the method further comprises:

[0025] Acquiring time series data obtained from multiple point cloud scans, wherein the time series data includes geometric deviations of the prefabricated immersed tube obtained after each point cloud scan;

[0026] The time series data is input into the LSTM model that has been pre-trained by machine learning to obtain the deviation value of the predicted future time point, and whether the deviation exceeds the tolerance range is determined based on the predicted deviation value; if so, a message prompting adjustment of the construction process is generated.

[0027] In a second aspect, the present disclosure provides an automated inspection system for immersed tube prefabrication projects based on point clouds and a robotic dog, including:

[0028] A path planning module is used to generate an oblique photography model of the prefabrication site, convert the oblique photography model into a point cloud model, perform grid segmentation on the point cloud model, determine multiple scanning sites based on the grid segmentation results, and perform path planning on the multiple scanning sites to obtain an optimal path;

[0029] a point cloud acquisition module, configured to transmit the optimal path to the robot dog, so that the robot dog reaches each scanning station according to the optimal path, and acquires scanning point cloud data of the prefabricated immersed tube at each scanning station using a scanner carried by the robot dog;

[0030] The deviation calculation module is used to convert the BIM model of the prefabricated immersed tube into model point cloud data, align the model point cloud data with the scanning point cloud data at each scanning station, and calculate the geometric deviation of the prefabricated immersed tube based on the alignment results.

[0031] In a third aspect, an embodiment of the present disclosure provides an electronic device, including:

[0032] processor; and

[0033] Memory for storing computer programs;

[0034] The processor is configured to execute the automated inspection method for immersed tube prefabrication projects based on point cloud and robotic dog of any of the above embodiments by executing the computer program.

[0035] The technical solution provided by the embodiments of the present disclosure has the following advantages over the prior art:

[0036] The embodiment of the present disclosure provides an automated inspection method and system for immersed tube prefabrication projects based on point clouds and a robotic dog, which generates an oblique photography model of the prefabrication site and converts the oblique photography model into a point cloud model; performs grid segmentation on the point cloud model, determines multiple scanning sites based on the grid segmentation results, and performs path planning on the multiple scanning sites to obtain an optimal path; transmits the optimal path to the robotic dog, so that the robotic dog reaches each scanning site according to the optimal path, and collects scanning point cloud data of the prefabricated immersed tube at each scanning site through the scanner carried by the robotic dog; converts the BIM model of the prefabricated immersed tube into model point cloud data, aligns the model point cloud data with the scanning point cloud data at each scanning site, and calculates the geometric deviation of the prefabricated immersed tube based on the alignment results. In this embodiment, the solution generates a point cloud model through an oblique photography model, and the scanning site can be accurately determined by gridding the point cloud model. Based on this, the optimal path for the robot dog navigation is generated through path planning, and the three-dimensional laser scanner carried by the robot dog is used to collect the component point cloud during the immersed tube prefabrication process, and it is aligned with the model point cloud converted from the BIM model to accurately measure the geometric deviation of the immersed tube prefabrication process. In the face of the complex construction environment of large components such as immersed tubes, this solution can significantly improve the efficiency and accuracy of the automated measurement of the geometric deviations of the immersed tube components during inspection, realize dynamic monitoring of the entire process of large prefabricated components such as immersed tubes, and realize automation with little or no human intervention in the monitoring process, which makes up for the problems of low accuracy, low efficiency and site limitations of traditional manual monitoring data, reduces the intervention of construction personnel in monitoring, improves the safety of construction measurement, and provides more efficient, accurate and safe solutions for engineering projects. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0038] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0039] Figure 1This is a flow chart of the automated inspection method for immersed tube prefabrication projects based on point cloud and a robotic dog according to an embodiment of the present disclosure;

[0040] Figure 2 This is a flow chart of an automated inspection method for immersed tube prefabrication projects based on point cloud and a robotic dog according to another embodiment of the present disclosure;

[0041] Figure 3 Schematic diagram of an automated inspection system for immersed tube prefabrication projects based on point cloud and a robotic dog according to an embodiment of the present disclosure;

[0042] Figure 4 Schematic diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0043] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features therein can be combined with each other in the absence of conflict.

[0044] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0045] It should be understood that, in the following text, "at least one (item)" refers to one or more, and "plurality" refers to two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0046] Figure 1 This is a flow chart of an automated inspection method for immersed tube prefabrication projects based on point clouds and a robotic dog according to an embodiment of the present disclosure. The method can be executed by a computing device such as a computer or a mobile terminal and may specifically include the following steps:

[0047] Step S101: Generate an oblique photography model of the prefabrication site, convert the oblique photography model into a point cloud model; perform grid segmentation on the point cloud model, determine multiple scanning sites based on the grid segmentation results, and perform path planning on the multiple scanning sites to obtain an optimal path.

[0048] For example, the prefabrication site can be a prefabrication factory, which includes prefabricated immersed tube components and environmental information such as background and obstacle information. Each immersed tube can consist of multiple components, which are prefabricated sequentially. After each component is prefabricated, a 3D scan can be performed to obtain point cloud data. Prior to this, the optimal path for the robot dog must be accurately planned. An oblique photogrammetry model is a 3D model generated using aerial imagery captured at oblique angles. This technology captures high-precision images of the ground and target from multiple angles, combines these images, and generates a high-precision 3D digital model using photogrammetry.

[0049] In this embodiment, a drone can be used to take multi-angle oblique images of the prefabricated site where the work is required, and an oblique photography model can be generated based on the multi-angle images covering the entire site. The specific generation process can be understood by referring to the existing technology. Then the oblique photography model is converted into a point cloud model. Figure 2 As shown, the point cloud model is then meshed, for example, using Delaunay triangulation or Poisson surface reconstruction to generate a mesh model. Based on the mesh segmentation results, multiple scanning stations are selected and determined. For example, the density of scanning stations can be increased in complex terrain areas and decreased in flat areas. Path planning is then performed for these selected scanning stations, for example using the Rapidly Exploring Random Tree (RRT) algorithm. This algorithm considers terrain complexity and obstacle distribution, selects and marks scanning station locations, and determines the global optimal path.

[0050] This embodiment does not directly use the oblique photography model to determine the site. The advantage is that the gridding of the point cloud significantly reduces the data volume, reducing the computational complexity of subsequent processing, thereby improving the efficiency of automated measurement of geometric deviations of immersed tube components during inspection. Gridding also better preserves key geometric information, improving the accuracy of subsequent scanning site positioning, thereby making the planned optimal path more accurate and the collected scanning point cloud data more accurate, ultimately improving the accuracy of automated measurement of geometric deviations of immersed tube components during inspection.

[0051] Step S102: The optimal path is transmitted to the robot dog, so that the robot dog reaches each scanning site according to the optimal path, and collects scanning point cloud data of the prefabricated immersed tube at each scanning site through the scanner carried by itself.

[0052] For example, a quadruped robot, as a four-legged mobile robot, has flexible movement capabilities and stable load capacity, and can replace human labor in performing inspection tasks in complex, dangerous, or inaccessible environments. In this embodiment, after the optimal path is determined, it can be transmitted to the robot dog, and wireless communication such as 4G or 5G communication can be achieved between the robot dog and the computing device. The robot dog walks according to the optimal path to each scanning station and uses its own scanner to collect scanning point cloud data of the prefabricated immersed tube at each scanning station. The scanner can be a 3D laser scanner. 3D laser scanning can obtain the precise three-dimensional coordinates, shape, and surface features of an object. In this embodiment, the 3D laser scanner carried by the robot dog can collect precise scanning point cloud data of the prefabricated immersed tube.

[0053] Step S103: converting the BIM model of the prefabricated immersed tube into model point cloud data, registering and aligning the model point cloud data with the scan point cloud data at each scanning station, and calculating the geometric deviation of the prefabricated immersed tube based on the registration and alignment results.

[0054] For example, the BIM (Building Information Modeling) model belongs to the building information modeling technology and is the most advanced modeling method with the highest degree of digitization in the construction field. The BIM model contains rich geometric information of components, and contains detailed digital information for target components such as immersed tubes, which is used to assist in construction quality control. In this embodiment, the BIM model of the prefabricated immersed tube is converted into model point cloud data, which is then aligned with the scanned point cloud data of the prefabricated immersed tube obtained by scanning, for example, using the random sample consensus algorithm RANSAC (Random Sample Consensus) for alignment. Finally, the geometric deviation of the prefabricated immersed tube is calculated based on the alignment results, for example, by calculating the geometric deviation through the distance from the point to the surface. In some embodiments, a deviation heat map can then be generated for visualization to mark the out-of-tolerance areas.

[0055] The above-mentioned scheme in this embodiment generates a point cloud model through an oblique photography model, and the scanning site can be accurately determined by gridding the point cloud model. Based on this, the optimal path for the robot dog navigation is generated through path planning, and the three-dimensional laser scanner carried by the robot dog is used to collect the component point cloud during the immersed tube prefabrication process, and it is aligned with the model point cloud converted from the BIM model to accurately measure the geometric deviation of the immersed tube prefabrication process. In the face of the complex construction environment of large components such as immersed tubes, this scheme can significantly improve the efficiency and accuracy of the automated measurement of the geometric deviations of the immersed tube components during inspection, realize dynamic monitoring of the entire process of large prefabricated components such as immersed tubes, and realize automation with little or no human intervention in the monitoring process, which makes up for the problems of low accuracy, low efficiency and site limitations of traditional manual monitoring data, reduces the intervention of construction personnel in monitoring, improves the safety of construction measurement, and provides more efficient, accurate and safe solutions for engineering projects.

[0056] Based on the above implementation, in one embodiment, the step S101 converts the oblique photography model into a point cloud model, including: reconstructing the oblique photography model into a three-dimensional mesh model, and sampling the three-dimensional mesh model to convert the three-dimensional mesh model into a point cloud model.

[0057] For example, the oblique photography model can be reconstructed into a three-dimensional mesh model using Pix4Dmapper software. The three-dimensional mesh model data can be stored in OBJ format. The three-dimensional mesh model is imported into CloudCompare processing software and the sampling tool is used to convert the three-dimensional mesh model into a point cloud model. The point cloud model can be stored in PCD format, but is not limited to this.

[0058] It should be noted that directly generating point clouds based on oblique photography models is prone to blurred edges due to image distortion, while the reconstruction of a three-dimensional mesh model can correct such distortion. The three-dimensional mesh reconstruction process also integrates multi-perspective texture information to reduce the perspective blind spot problem when directly generating point clouds. At the same time, the three-dimensional mesh model can retain the geometric topological relationship of the ground objects, and the sampled point cloud can inherit such structured features. During the reconstruction of the three-dimensional mesh model, the holes in the original point cloud (such as occluded areas) can also be filled by connecting triangular facets. The spatial distribution of the sampled point cloud is more uniform, and the point cloud quality and accuracy are higher. Therefore, the solution of this embodiment provides a more stable and complete data foundation for point cloud generation, making the generated point cloud model more accurate, thereby determining the scanning station location more accurately, and thus planning the path most accurately, making the scanning point cloud data collected by the robot dog more accurate, and further improving the automated measurement accuracy of the geometric deviation of the submerged pipe components during inspection.

[0059] Based on the above embodiments, in one embodiment, the method further includes: detecting the point cloud density in the point cloud model; when determining that the point cloud density is less than or equal to the preset point cloud density, resampling the three-dimensional mesh model so that the point cloud density of the converted point cloud model is greater than the preset point cloud density.

[0060] For example, the preset point cloud density can be set as needed. By checking the point cloud density, it can be ensured that the resolution of the converted point cloud model is sufficient to cover the details of the site, making the generated point cloud model more accurate, thereby determining the scanning site position more accurately, and planning the path most accurately, making the scanning point cloud data collected by the robot dog more accurate, thereby further improving the automated measurement accuracy of the geometric deviations of the inspected immersed tube components.

[0061] Based on the above implementation, in one embodiment, reference Figure 2 As shown, in step S101, the point cloud model is gridded and segmented, and a plurality of scanning sites are determined according to the gridded segmentation result, including:

[0062] Setting a preset grid size to divide the point cloud in the point cloud model into a plurality of sub-regions according to the spatial range, each sub-region corresponding to a grid;

[0063] A grid center point is selected in each grid as a plurality of candidate sites; based on the scanning coverage of the prefabrication site by the position of each candidate site, the plurality of candidate sites are optimized to remove redundant candidate sites to obtain a plurality of scanning sites.

[0064] For example, the point cloud model can be gridded using the Point Cloud Library (PCL). A preset grid size is set to divide the point cloud in the point cloud model into multiple sub-regions based on spatial range. Each sub-region, or spatial region, corresponds to a grid, and each grid can contain multiple point clouds. The center point of each grid is selected as a candidate site, and the coordinates of the candidate site are generated. A coverage optimization algorithm (based on the laser scanner field of view model) is used to optimize the distribution of candidate sites. The scan coverage of each candidate site over the site is checked, and redundant sites, such as those with scan coverage less than a preset value, are removed. This makes the determined scanning site location more accurate, leading to the most accurate path planning and more accurate scanning point cloud data collected by the robot dog, further improving the automated measurement accuracy of geometric deviations of submerged tube components during inspection.

[0065] Based on any one of the above embodiments, in one embodiment, reference Figure 2 As shown, the method may further include the following steps:

[0066] When it is determined that the path planning is the first planning after the first component of the prefabricated immersed tube is completed, the point cloud map is transmitted to the robotic dog, so that the robotic dog navigates to a designated station based on the position of each scanning station marked in the point cloud map, and obtains the precise coordinate position of the designated station at the designated station through its own positioning system, thereby determining the precise coordinate position of each scanning station; wherein the point cloud map is determined by the point cloud model, and the designated station is any one of the multiple scanning stations;

[0067] When controlling the robotic dog to perform multi-point navigation to reach the target site based on the precise coordinate position of each scanning site, after calibrating the position of the robotic dog based on the precise coordinate position of the target site, set the stay time at the target site, and control the scanner to collect scanning point cloud data of the prefabricated immersed tube based on the stay time.

[0068] Exemplarily, the point cloud map can be a two-dimensional (2D) map. In this embodiment, a planning scan measurement can be performed after each prefabricated immersed tube component is completed. After the first component is completed, the initial site planning is determined to be complete. If so, the point cloud map, previously converted into a PCD file, is input to the map service map_server in the robot's operating system (ROS). The robot then uses navigation to reach a designated site, i.e., any one of multiple scanning sites, based on the scan site locations marked in the input point cloud map. At the designated site, the robot uses its onboard positioning system, such as a real-time kinematic (RTK) module, to obtain the precise coordinates of the designated site. This process is repeated to determine the precise coordinates of each scan site.

[0069] After that, the scanning task is executed. First, the multi-point navigation function is performed. The precise coordinate positions of all scanning sites are input into the multi-point navigation CPP file written in the robot dog. The SLAM (Simultaneous Localization and Mapping) module carried by the robot dog is used to perform real-time dynamic obstacle avoidance to reach the target site. After arriving at the target site, the position is calibrated in real time through RTK and the ROS output reaches the target point. The stay time of the robot dog from the start of the scanner after reaching the target point to the completion of the scan is set. The length of the stay time takes into account the site coverage area and the scanner completion time. After the stay time reaches a certain threshold, the scanner is activated to scan and collect the scanning point cloud data of the prefabricated immersed tube. After the scanning time ends, navigation to the next site begins.

[0070] In this embodiment, by obtaining the precise coordinates of all scanning stations, the robot can calibrate their positions during multi-point navigation. This, combined with a reasonable dwell time, ensures complete coverage of the scanned data and improves detection accuracy. This results in more accurate point cloud data collected for prefabricated immersed tubes, and thus more accurate geometric deviation calculations after registration. This further enhances the accuracy of automated measurement of geometric deviations for immersed tube components during inspection. Furthermore, multi-point navigation allows for multi-tasking, improving overall data collection efficiency and, consequently, automated measurement of geometric deviations for immersed tube components during inspection.

[0071] Based on the above embodiments, in one embodiment, reference is made to Figure 2 As shown, the method also includes: when it is determined that the path planning is not the first planning after the first component of the prefabricated immersed tube is completed, the step of transmitting the point cloud map to the robot dog is skipped, and the robot dog is directly navigated to the designated site. The point cloud map has been transmitted to the robot dog after the first component is completed. When scanning subsequent components, there is no need to transmit the point cloud map, and navigation can be done between them. That is, if it is not the first site planning after the completion of the production of the first component, the Navigation function is directly used to navigate the robot dog to the designated site, skipping the step of inputting the point cloud map. After that, the robot dog executes the step of determining the precise coordinate position of each scanning site and the subsequent steps of multi-point navigation, controlling the scanning point cloud, etc.

[0072] In one embodiment, reference Figure 2 As shown, the method may further include the following steps:

[0073] De-noising the scanned point cloud data at each scanning station, and then extracting key feature points; judging based on the key feature points whether the point cloud scan of the first component of the prefabricated immersed tube has been completed for the first time;

[0074] If yes, register and align the model point cloud data with the scan point cloud data at each scanning station after denoising, and calculate the component geometric deviation of the prefabricated immersed tube based on the registration and alignment results;

[0075] If not, point cloud stitching is performed based on the target scanning point cloud data and the scanning point cloud data at each scanning site after denoising to obtain updated scanning point cloud data; registration and alignment are performed based on the updated scanning point cloud data and the model point cloud data, and the component geometric deviation of the prefabricated immersed tube is calculated based on the registration and alignment results; wherein the target scanning point cloud data is the scanning point cloud data of the next component after the first component is completed.

[0076] For example, the collected scanned point cloud data is first denoised to remove redundant and outliers. Then, the PCL feature extraction module can be used to extract key feature points, such as edge points and corner points, from the denoised scanned point cloud data. Based on these key feature points, a determination is made as to whether the scan represents the first completed point cloud scan of the prefabricated immersed tube component. If so, the RANSAC algorithm is used to align the denoised scanned point cloud data with the model point cloud data. Based on the alignment results, the geometric deviation is calculated using the point-to-surface distance. This can then be visualized to generate a deviation heat map, marking areas of deviation.

[0077] If the point cloud scan is not the first completed component, but rather the scan of the subsequent component (i.e., both the first and next components have been scanned), point cloud stitching, such as dynamic point cloud stitching, is performed. The scanned point cloud data of the next component is gradually stitched together with the scanned point cloud data of the first component. For example, an incremental ICP algorithm is used to dynamically update the component's global point cloud. Subsequently, the point cloud data is aligned with the model point cloud data converted from the BIM model. Based on the alignment results, the geometric deviation of the prefabricated immersed tube component is calculated. Deviation heatmaps can also be generated and visualized in a single step to mark areas of deviation.

[0078] Based on any one of the above embodiments, in one embodiment, reference Figure 2 As shown, the method may further include the following steps:

[0079] Acquiring time series data obtained from multiple point cloud scans, wherein the time series data includes geometric deviations of the prefabricated immersed tube obtained after each point cloud scan;

[0080] The time series data is input into the LSTM model that has been pre-trained by machine learning to obtain the deviation value of the predicted future time point, and whether the deviation exceeds the tolerance range is determined based on the predicted deviation value; if so, a message prompting adjustment of the construction process is generated.

[0081] For example, this embodiment can analyze time series data from multiple scans, extract deviation gradients, and generate trend curves for high-risk deviation areas. A pre-trained LSTM (Long Short-Term Memory) model can also be used to predict the time series data, predicting deviation values at future time points. Based on the predicted deviation value, the prediction result, i.e., the predicted deviation value, determines whether the deviation exceeds the tolerance range. If so, a serious problem exists, and a message is generated prompting adjustments to the construction process to optimize the construction process. The trained LSTM model is pre-trained on the original LSTM model based on sample time series data. It can accurately predict the deviation values of prefabricated immersed tube inspections at future time points caused by the current scanning measurement method. For example, it can predict the predicted deviation value of the next component in advance, anticipating potential problems and facilitating early adjustments to the construction process to ensure that the geometric deviation of the next prefabricated component meets standards, avoiding rework.

[0082] Finally, the method of this embodiment can determine whether the scanning measurement of all components of the prefabricated immersed tube is completed. If so, the method ends; if not, the method returns to the candidate point selection step to continue the scanning measurement process of the next component.

[0083] It should be noted that although the steps of the method disclosed herein are depicted in a particular order in the accompanying drawings, this does not require or imply that the steps must be performed in that particular order, or that all steps must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into a single step, and / or a single step may be decomposed into multiple steps. Furthermore, it is readily understood that these steps may be executed synchronously or asynchronously, for example, in multiple modules / processes / threads.

[0084] like Figure 3 As shown, the embodiment of the present disclosure provides an automated inspection system for immersed tube prefabrication projects based on point cloud and a robotic dog, comprising:

[0085] Path planning module 401 is used to generate an oblique photography model of the prefabrication site, convert the oblique photography model into a point cloud model, perform grid segmentation on the point cloud model, determine multiple scanning sites based on the grid segmentation results, and perform path planning on the multiple scanning sites to obtain an optimal path;

[0086] The point cloud acquisition module 402 is configured to transmit the optimal path to the robot dog, so that the robot dog reaches each scanning station according to the optimal path and acquires scanning point cloud data of the prefabricated immersed tube at each scanning station using its own scanner;

[0087] The deviation calculation module 403 is used to convert the BIM model of the prefabricated immersed tube into model point cloud data, align the model point cloud data with the scanning point cloud data at each scanning station, and calculate the geometric deviation of the prefabricated immersed tube based on the alignment results.

[0088] In one embodiment, the path planning module 401 converts the oblique photography model into a point cloud model, including: reconstructing the oblique photography model into a three-dimensional mesh model, and sampling the three-dimensional mesh model to convert the three-dimensional mesh model into a point cloud model.

[0089] In one embodiment, the system also includes a point cloud detection module, which is used to: detect the point cloud density in the point cloud model; when it is determined that the point cloud density is less than or equal to the preset point cloud density, resample the three-dimensional grid model so that the point cloud density of the converted point cloud model is greater than the preset point cloud density.

[0090] In one embodiment, the path planning module 401 performs grid segmentation on the point cloud model and determines a plurality of scanning sites according to the grid segmentation result, including:

[0091] Setting a preset grid size to divide the point cloud in the point cloud model into a plurality of sub-regions according to the spatial range, each sub-region corresponding to a grid;

[0092] A grid center point is selected in each grid as a plurality of candidate sites; based on the scanning coverage of the prefabrication site by the position of each candidate site, the plurality of candidate sites are optimized to remove redundant candidate sites to obtain a plurality of scanning sites.

[0093] In one embodiment, the system also includes a navigation control module, which is used to: when it is determined that the path planning is the first planning after the first component of the prefabricated immersed tube is completed, the point cloud map is transmitted to the mechanical dog, so that the mechanical dog navigates to the designated site based on the position of each scanning site marked in the point cloud map, and obtains the precise coordinate position of the designated site through the positioning system carried by itself at the designated site, thereby determining the precise coordinate position of each scanning site; wherein the point cloud map is determined by the point cloud model, and the designated site is any one of the multiple scanning sites; when controlling the mechanical dog to perform multi-point navigation based on the precise coordinate position of each scanning site to reach the target site, after calibrating the position of the mechanical dog based on the precise coordinate position of the target site, setting the dwell time of the target site, and controlling the scanner to collect the scanning point cloud data of the prefabricated immersed tube based on the dwell time.

[0094] In one embodiment, the navigation control module is also used to: when it is determined that the path planning is not the first planning after the first component of the prefabricated immersed tube is completed, skip the step of transmitting the point cloud map to the robotic dog, so that the robotic dog directly navigates to the designated site.

[0095] In one embodiment, the system further includes a point cloud processing module for: performing denoising processing on the scanned point cloud data at each scanning station, and then extracting key feature points; and determining, based on the key feature points, whether the point cloud scan of the first component of the prefabricated immersed tube has been completed for the first time;

[0096] If yes, the deviation calculation module 403 registers and aligns the model point cloud data with the scan point cloud data at each scanning station after denoising, and calculates the component geometric deviation of the prefabricated immersed tube based on the registration and alignment results;

[0097] If not, the point cloud processing module performs point cloud splicing based on the target scanning point cloud data and the scanning point cloud data at each scanning site after denoising to obtain updated scanning point cloud data; the deviation calculation module 403 performs registration and alignment based on the updated scanning point cloud data and the model point cloud data, and calculates the component geometric deviation of the prefabricated immersed tube based on the registration and alignment results; wherein the target scanning point cloud data is the scanning point cloud data of the next component after the first component is completed.

[0098] In one embodiment, the system may further include an error prediction module for:

[0099] Acquiring time series data obtained from multiple point cloud scans, wherein the time series data includes geometric deviations of the prefabricated immersed tube obtained after each point cloud scan;

[0100] The time series data is input into the LSTM model that has been pre-trained by machine learning to obtain the deviation value of the predicted future time point, and whether the deviation exceeds the tolerance range is determined based on the predicted deviation value; if so, a message prompting adjustment of the construction process is generated.

[0101] Regarding the system in the above embodiment, the specific manner in which each module performs operations and the corresponding technical effects brought about have been described in detail in the embodiment of the method, and will not be elaborated here.

[0102] It should be noted that although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiment of the present disclosure, the features and functions of two or more modules or units described above can be concretized in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units for concretization. The components displayed as modules or units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the disclosed solution. Those of ordinary skill in the art can understand and implement it without paying any creative work.

[0103] An embodiment of the present disclosure also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for automated inspection of immersed tube prefabrication projects based on point cloud and a robotic dog as described in any of the above embodiments is implemented.

[0104] Exemplarily, the readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0105] The computer-readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, wherein the readable program code is carried. The data signal propagated may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or component. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.

[0106] The present disclosure also provides an electronic device comprising a processor and a memory for storing a computer program, wherein the processor is configured to execute the computer program to perform the automated inspection method for immersed tube prefabrication projects based on point clouds and a robotic dog according to any of the above-mentioned embodiments.

[0107] Refer to the following Figure 4 An electronic device 600 according to this embodiment of the present invention will be described. Figure 4 The electronic device 600 shown is merely an example and should not limit the functions and scope of use of the embodiments of the present invention.

[0108] like Figure 4 As shown, electronic device 600 is implemented as a general-purpose computing device. Components of electronic device 600 may include, but are not limited to, at least one processing unit 610, at least one storage unit 620, a bus 630 connecting various system components (including storage unit 620 and processing unit 610), and a display unit 640.

[0109] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 performs the steps according to various exemplary embodiments of the present invention described in the above method embodiment section of this specification. For example, the processing unit 610 can perform the following steps: Figure 1 The steps of the method shown in .

[0110] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .

[0111] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each of which or some combination may include an implementation of a network environment.

[0112] Bus 630 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0113] The electronic device 600 can also communicate with one or more external devices 700 (e.g., a keyboard, pointing device, Bluetooth device, etc.), one or more devices that enable a user to interact with the electronic device 600, and / or any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., a router, modem, etc.). This communication can occur via an input / output (I / O) interface 650. Furthermore, the electronic device 600 can communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via a network adapter 660. The network adapter 660 can communicate with other modules of the electronic device 600 via the bus 630. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with the electronic device 600, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0114] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented through software or through a combination of software and necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes a number of instructions to enable a computing device (such as a personal computer, a server, or a network device, etc.) to execute the method steps of the above-mentioned embodiments according to the embodiments of the present disclosure.

[0115] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0116] The foregoing description is intended only to provide specific embodiments of the present disclosure, intended to enable those skilled in the art to understand and implement the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the embodiments described herein, but rather to be construed in the broadest manner consistent with the principles and novel features disclosed herein.

Claims

1. An automated inspection method for immersed tube prefabrication projects based on point cloud and robotic dog, characterized in that: The method includes: Generate an oblique photography model of the prefabrication site, convert the oblique photography model into a point cloud model; perform grid segmentation on the point cloud model, determine multiple scanning sites based on the grid segmentation results, and perform path planning on the multiple scanning sites to obtain an optimal path; Transmitting the optimal path to the robot dog so that the robot dog reaches each scanning station according to the optimal path and collects scanning point cloud data of the prefabricated immersed tube at each scanning station through the scanner carried by the robot dog; Converting the BIM model of the prefabricated immersed tube into model point cloud data, registering and aligning the model point cloud data with the scan point cloud data at each scanning station, and calculating the geometric deviation of the prefabricated immersed tube based on the registration and alignment results; The step of performing grid segmentation on the point cloud model and determining a plurality of scanning sites according to the grid segmentation result includes: Setting a preset grid size to divide the point cloud in the point cloud model into a plurality of sub-regions according to the spatial range, each sub-region corresponding to a grid; A grid center point is selected in each grid as a plurality of candidate sites; based on the scanning coverage of the prefabrication site by the position of each candidate site, the plurality of candidate sites are optimized to remove redundant candidate sites to obtain a plurality of scanning sites.

2. The method according to claim 1, characterized in that The step of converting the oblique photography model into a point cloud model comprises: The oblique photography model is reconstructed into a three-dimensional mesh model, and the three-dimensional mesh model is sampled to convert the three-dimensional mesh model into a point cloud model.

3. The method according to claim 2, characterized in that The method further includes: Detecting the point cloud density in the point cloud model; and when determining that the point cloud density is less than or equal to a preset point cloud density, resampling the three-dimensional grid model so that the point cloud density of the converted point cloud model is greater than the preset point cloud density.

4. The method according to any one of claims 1 to 3, characterized in that The method further includes: When it is determined that the path planning is the first planning after the first component of the prefabricated immersed tube is completed, the point cloud map is transmitted to the robotic dog, so that the robotic dog navigates to a designated station based on the position of each scanning station marked in the point cloud map, and obtains the precise coordinate position of the designated station at the designated station through its own positioning system, thereby determining the precise coordinate position of each scanning station; wherein the point cloud map is determined by the point cloud model, and the designated station is any one of the multiple scanning stations; When controlling the robotic dog to perform multi-point navigation to reach the target site based on the precise coordinate position of each scanning site, after calibrating the position of the robotic dog based on the precise coordinate position of the target site, set the stay time at the target site, and control the scanner to collect scanning point cloud data of the prefabricated immersed tube based on the stay time.

5. The method according to claim 4, characterized in that The method further includes: When it is determined that the path planning is not the first planning after the first component of the prefabricated immersed tube is completed, the step of transmitting the point cloud map to the robotic dog is skipped, so that the robotic dog directly navigates to the designated site.

6. The method according to any one of claims 1 to 3, characterized in that The method further includes: De-noising the scanned point cloud data at each scanning station, and then extracting key feature points; judging based on the key feature points whether the point cloud scan of the first component of the prefabricated immersed tube has been completed for the first time; If yes, register and align the model point cloud data with the scan point cloud data at each scanning station after denoising, and calculate the component geometric deviation of the prefabricated immersed tube based on the registration and alignment results; If not, point cloud stitching is performed based on the target scanning point cloud data and the scanning point cloud data at each scanning site after denoising to obtain updated scanning point cloud data; registration and alignment are performed based on the updated scanning point cloud data and the model point cloud data, and the component geometric deviation of the prefabricated immersed tube is calculated based on the registration and alignment results; wherein the target scanning point cloud data is the scanning point cloud data of the next component after the first component is completed.

7. The method according to any one of claims 1 to 3, characterized in that The method further includes: Acquiring time series data obtained from multiple point cloud scans, wherein the time series data includes geometric deviations of the prefabricated immersed tube obtained after each point cloud scan; The time series data is input into the LSTM model that has been pre-trained by machine learning to obtain the deviation value of the predicted future time point, and whether the deviation exceeds the tolerance range is determined based on the predicted deviation value; if so, a message prompting adjustment of the construction process is generated.

8. An automated inspection system for immersed tube prefabrication projects based on point cloud and robotic dog, characterized in that: include: A path planning module, configured to generate an oblique photography model of the prefabrication site and convert the oblique photography model into a point cloud model; Performing grid segmentation on the point cloud model, determining a plurality of scanning sites according to the grid segmentation result, and performing path planning on the plurality of scanning sites to obtain an optimal path; a point cloud acquisition module, configured to transmit the optimal path to the robot dog, so that the robot dog reaches each scanning station according to the optimal path, and acquires scanning point cloud data of the prefabricated immersed tube at each scanning station using a scanner carried by the robot dog; a deviation calculation module, configured to convert the BIM model of the prefabricated immersed tube into model point cloud data, register and align the model point cloud data with the scan point cloud data at each scanning station, and calculate the geometric deviation of the prefabricated immersed tube based on the registration and alignment results; The path planning module is specifically used to: Setting a preset grid size to divide the point cloud in the point cloud model into a plurality of sub-regions according to the spatial range, each sub-region corresponding to a grid; A grid center point is selected in each grid as a plurality of candidate sites; based on the scanning coverage of the prefabrication site by the position of each candidate site, the plurality of candidate sites are optimized to remove redundant candidate sites to obtain a plurality of scanning sites.

9. An electronic device, characterized in that: include: processor; as well as memory for storing computer programs; Wherein, the processor is configured to execute the automated inspection method for immersed tube prefabrication projects based on point cloud and robotic dog as described in any one of claims 1 to 7 by executing the computer program.

Images