Construction progress intelligent dynamic identification and modeling method based on laser radar fused binocular and depth camera data

Abstract

The invention provides a construction progress intelligent dynamic identification and modeling method based on laser radar fused binocular and depth camera data. The method comprises the following steps: data acquisition: acquiring a construction site two-dimensional RBG image and three-dimensional point cloud data containing construction progress information through a multi-mode construction progress data acquisition system; performing data processing and analysis: performing fusion processing on the acquired two-dimensional RBG images and three-dimensional point cloud data by using a construction state recognition algorithm based on artificial intelligence, recognizing type information, spatial position information and time feature information of the components, performing construction progress quantification based on spatial volume difference to obtain construction progress information of each component, and performing data processing and analysis; forming a component point cloud data set; and performing three-dimensional progress dynamic reconstruction based on the building information model: dynamically reconstructing the progress information through the building information model BIM by using the component point cloud data set. According to the invention, dynamic and accurate evaluation and scientific guidance of the construction progress are realized through unmanned, intelligent and three-dimensional visualization means.

Description

Technical Field

[0001] The present invention relates to the field of engineering construction, and specifically to an intelligent dynamic recognition and modeling method for construction progress based on the fusion of lidar, binocular and depth camera data. Background Art

[0002] In engineering construction, the construction stage is the link with the highest investment and the greatest management difficulty, which mainly involves the planning, organization, command, supervision and regulation of production activities. Accurately evaluating the construction progress and scientifically guiding the construction management are crucial for the advancement of engineering projects.

[0003] The existing construction progress evaluation usually relies on the manual statistical method of construction logs, and records the progress in the forms of daily reports, weekly reports and monthly reports. This method is limited by the following problems:

[0004] Strong subjectivity: The statistical results rely on the experience and judgment of management personnel and are easily affected by subjective factors.

[0005] Single spatial dimension: The construction progress data is usually recorded in a one-dimensional form, lacking a three-dimensional description of the spatio-temporal characteristics of the building.

[0006] Insufficient dynamics: The dynamic changes of the construction progress cannot be reflected in real time, affecting the scientific nature of construction management decisions.

[0007] With the increasing complexity of engineering construction, the existing methods are difficult to meet the needs of digital and intelligent construction management. Summary of the Invention

[0008] To solve the deficiencies in the prior art, the present invention proposes an intelligent dynamic recognition and modeling method for construction progress based on the fusion of lidar, binocular and depth camera data, aiming to achieve dynamic and accurate evaluation of construction progress and scientific guidance through unmanned, intelligent and three-dimensional visualization means.

[0009] An intelligent dynamic recognition and modeling method for construction progress based on the fusion of lidar, binocular and depth camera data, comprising the following steps:

[0010] Data acquisition: Collect two-dimensional RBG images and three-dimensional point cloud data of the construction site containing construction progress information through a multi-modal construction progress data acquisition system. The multi-modal construction progress data acquisition system includes an outdoor macro-scale data acquisition system and an indoor meso-scale data acquisition system. The outdoor macro-scale data acquisition system includes dynamic fixed measurement points and unmanned aerial vehicles, and the indoor meso-scale data acquisition system includes bionic robots. The dynamic fixed measurement points, unmanned aerial vehicles and bionic robots are respectively equipped with multi-modal data acquisition devices. The multi-modal data acquisition devices include binocular cameras, depth cameras or lidar;

[0011] Data processing and analysis: Using the artificial intelligence-based construction status recognition algorithm, the collected 2D RBG images and 3D point cloud data are fused and processed to identify the type information, spatial position information, and time characteristic information of the components. The construction progress is quantified based on the spatial volume difference, and the construction progress information of each component is obtained to form a component point cloud data set;

[0012] Dynamically reconstruct three-dimensional progress based on building information model: utilize the component point cloud data set to dynamically reconstruct progress information through building information model BIM.

[0013] Furthermore, the data collection step includes outdoor macro-scale data collection and indoor micro-scale data collection.

[0014] Furthermore, the outdoor macro-scale data collection includes:

[0015] Through dynamic fixed measuring points, fixed monitoring measuring points are arranged on the facades of buildings near the construction site, and the positions of the measuring points are dynamically adjusted based on the multi-objective optimization algorithm to achieve the largest range of image data collection in the construction site;

[0016] By using unmanned aerial vehicles equipped with binocular cameras and lidars, combined with multi-objective optimization algorithms, the optimal flight path for different construction stages is determined, and linked with dynamic fixed measuring points to supplement data collection.

[0017] Furthermore, the indoor micro-scale data is collected by a bionic robot equipped with a binocular camera, a depth camera and a lidar, and the construction data of the indoor scene is collected according to an optimized detection path.

[0018] Furthermore, the data processing and analysis steps include:

[0019] Based on the multimodal 3D target detection algorithm, target detection and segmentation are performed on the collected 3D point cloud data to identify the component point cloud, where the components include beams, plates, columns, and walls;

[0020] Label the corresponding component point cloud with type information, spatial location information, and time information;

[0021] According to the component point cloud data collected at different times, the spatial three-dimensional feature comparison of the components is carried out on the basis of the alignment type information, spatial position information and time information. By calculating the volume difference of the components, the component progress quantification based on the spatial volume is realized, and finally the construction progress information of each component is obtained;

[0022] The component point cloud data set is formed based on type information, spatial position information, time information and construction progress information of each component.

[0023] Furthermore, the three-dimensional progress dynamic reconstruction based on the building information model specifically includes:

[0024] Based on the time information of the point cloud data of each component, the type information and spatial position information of the annotation of the component point cloud dataset are compared with the components in the building information model BIM in terms of type and spatial position to align the component point cloud with its corresponding component model.

[0025] According to the point cloud model corresponding to each component model, reconstruct the true three-dimensional geometric features of the component at this moment, update the three-dimensional model in the building information model BIM, and replace the corresponding component model at the previous moment.

[0026] The components that have not started construction are represented by virtual component models in the building information model, and the components that have been constructed are represented by actual component models.

[0027] Furthermore, it also includes a result display step: presenting the construction progress through a three-dimensional dynamic visualization platform, including the time, space, and progress status information of the components.

[0028] The present invention can realize the unmanned, intelligent, and three-dimensional visual expression of the construction progress, so as to realize the dynamic and refined evaluation of the construction progress and achieve the purpose of scientifically guiding construction management. Description of the Drawings

[0029] Figure 1 is a schematic flow chart of the construction progress intelligent dynamic recognition and modeling method based on the fusion of lidar, binocular and depth camera data of the present invention;

[0030] Figure 2 is a schematic structural diagram of the multi-mode construction progress data acquisition system of the present invention;

[0031] Figure 3 is a flow chart of the construction state recognition algorithm based on artificial intelligence of the present invention;

[0032] Figure 4 is a flow chart of the three-dimensional progress dynamic reconstruction based on the building information model of the present invention.

[0033] In the figure: 1: Dynamic fixed measurement point; 2: Unmanned aerial vehicle; 3: Building to be built; 4: Bionic robot; 5: Interior of the building to be built; 6: Outdoor field of view of the dynamic fixed measurement point; 7: Outdoor field of view range of the unmanned aerial vehicle; 8: Field of view range of the bionic robot; 9: Adjacent existing building. Detailed Embodiments

[0034] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Apparently, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0035] Please refer to Figures 1-4 , the embodiments of the present invention provide a method for intelligent dynamic recognition and modeling of construction progress based on the fusion of lidar, binocular, and depth camera data, including the following steps:

[0036] (1) Data acquisition: Collect two-dimensional RBG images and three-dimensional point cloud data of the construction site containing construction progress information through a multi-modal construction progress data acquisition system. Specifically, use acquisition platforms such as dynamic fixed measurement points, unmanned aerial vehicles, and bionic robots, and perform data acquisition based on multi-modal acquisition terminals such as binocular cameras and lidar.

[0037] The main function of this step is to use a multi-method fusion acquisition method to achieve the acquisition of raw data on the construction status at multiple scales. For construction projects, it mainly includes sub-projects at the macroscopic scale and sub-items at the microscopic scale. Among them, the sub-items also include several specific operation tasks; in addition, at the macroscopic scale, the construction site will be divided into several construction sections, and at the microscopic scale, construction teams will also be divided according to the flow rhythm. Therefore, when collecting construction progress data, it should include the construction status at different scales to truly depict the multi-scale characteristics of the construction process. The present invention specifically divides the multi-modal construction progress data acquisition system into an outdoor macroscopic scale data acquisition system and an indoor microscopic scale data acquisition system.

[0038] The main framework of this multi-modal construction progress data acquisition system is as Figure 2 shown, where the forms of the acquisition terminals mainly include dynamic fixed measurement points 1, unmanned aerial vehicles 2, and bionic robots 4; and a multi-modal data acquisition device integrating binocular cameras, depth cameras, and lidar is carried by the above acquisition terminals; the collected data is stored according to the component association relationship of two-dimensional RBG images - three-dimensional point cloud data.

[0039] The outdoor macroscopic scale data acquisition system includes dynamic fixed measurement points and unmanned aerial vehicles.

[0040] 1) Dynamic fixed measurement points: This acquisition terminal is a fixed monitoring measurement point placed on the outer facade of the existing building 9 adjacent to the construction site. Its position is fixed and will not change easily, so the measurement range is fixed, but it can be adjusted in a timely manner according to the construction progress and management requirements.

[0041] Specific location layout method: Before the construction starts, investigate the adjacent existing buildings 9 near the project construction site, and establish a building information model including the adjacent buildings. Through visual simulation, simulate and layout fixed monitoring points on the adjacent buildings; then, based on the monitoring equipment parameters and the visual acquisition range, use the multi-objective optimization method to solve the optimal layout plan of the monitoring points at each construction stage; furthermore, according to the simulated and optimized layout plan, dynamically adjust the actual positions of the monitoring points to achieve the monitoring of the largest range of the construction site and collect construction image data in a larger range.

[0042] 2) Unmanned aerial vehicle: Considering that in actual construction, there is uncertainty about whether there are existing buildings near the construction site that can be used to layout fixed monitoring points, and there must be acquisition blind spots for the fixed monitoring points on the construction site status. Therefore, it is necessary to use the unmanned aerial vehicle 2 to achieve the supplementary acquisition of large-scale data and cooperate with the dynamic fixed monitoring points to supplement the measurement of their blind spots.

[0043] The specific acquisition strategy includes: First, according to the performance parameters of the multi-mode data acquisition devices such as the binocular camera and lidar carried by the unmanned aerial vehicle 2, simulate and analyze the field-of-view characteristics and rules of the UAV acquisition positions through visual simulation. Then, use the multi-objective optimization algorithm to solve the optimal detection path of the unmanned aerial vehicle at different construction stages to achieve the linkage measurement with the dynamic fixed monitoring points.

[0044] The indoor meso-scale data acquisition system includes the bionic robot 4. The macroscopic progress of the construction project can be detected by outdoor sensors, but for the indoor meso-scale progress, due to conditions such as narrow space and visual field occlusion, it cannot be detected by outdoor means. Therefore, the present invention uses the bionic robot 4 (such as a robotic dog) as a carrier platform to achieve the acquisition of the construction progress data of the indoor scene. The main process includes: First, according to the performance parameters of the multi-mode data acquisition devices such as the binocular camera, depth camera, and lidar carried by the robotic dog, simulate and analyze the field-of-view characteristics and rules of the data acquisition positions of the robotic dog in the indoor scene at different construction stages. Then, use the multi-objective optimization algorithm to solve the optimal detection path of the bionic robot at different construction stages to achieve the all-round data acquisition of the indoor scene.

[0045] Through the above multi-mode construction progress data acquisition system, two-dimensional RBG images and three-dimensional point cloud data containing construction progress information at different construction stages can be collected, as well as the spatial position information of the components corresponding to the images, and the components include beams, slabs, columns, walls, etc.

[0046] (2) Data processing and analysis: Using the construction state recognition algorithm based on artificial intelligence (such as Figure 3As shown, the collected two-dimensional RGB images and three-dimensional point cloud data are fused to identify the type information, spatial position information, and time feature information of components. Based on the spatial volume difference, the construction progress is quantified to obtain the construction progress information of each component, forming a component point cloud data set.

[0047] Specifically, according to the data collected by the multi-modal construction progress data acquisition system, using the multi-modal three-dimensional object detection algorithm that fuses two-dimensional RGB images and three-dimensional point cloud data, the three-dimensional point cloud data collected is subjected to object detection and segmentation to identify and segment component point clouds such as beams, slabs, columns, and walls, and label the corresponding component point clouds with type information, spatial position information, and time information. Further, based on the component point cloud data collected at different times, on the basis of aligning the attribute information and spatial position information, the three-dimensional spatial features of the components are compared. By calculating the volume difference of the components, the progress quantification of the components based on the spatial volume is realized, and finally the construction progress information of each component is obtained.

[0048] Through the above operations, the collected point cloud data can be segmented and identified, and the spatial position, time, type, and progress information of the point cloud data of each segmented component can be labeled, forming a component point cloud data set.

[0049] (3) Three-dimensional progress dynamic reconstruction based on building information model

[0050] As Figure 4 shown, using the above-mentioned component point cloud data set updated in real time, the progress information is dynamically reconstructed through the building information model BIM: First, according to the time information of each component point cloud data, based on the type information and spatial position information labeled in the component point cloud data set at that moment, the types and spatial positions of the components in the building information model BIM are compared to align the component point cloud with its corresponding component model. Subsequently, the time information and progress information labeled in the component point cloud are updated in the component model. Then, according to the point cloud model corresponding to each component model, the real three-dimensional geometric features of the components at that moment are reconstructed, and the three-dimensional model in the building information model BIM is updated to replace the corresponding component model at the previous moment; the components that have not started construction are represented by virtual component models in the building information model, and the components that have been constructed are represented by actual component models.

[0051] Through the above operations, the three-dimensional progress dynamic reconstruction based on the building information model is realized.

[0052] The present invention has the following characteristics and effects:

[0053] (1) A method for collecting construction progress image data of a multi-mode vision acquisition terminal integrating dynamic fixed measurement points, unmanned aerial vehicles, and bionic robots is proposed. A dynamic optimization strategy for the layout position of fixed measurement points, the acquisition paths of unmanned aerial vehicles and bionic robots based on construction progress is established, realizing the acquisition of indoor and outdoor macro- and micro-scale data.

[0054] (2) A method for identifying component spatio-temporal and progress information based on multi-mode image data fusion is proposed. A three-dimensional dynamic characterization of construction progress based on building information models is constructed, realizing the intelligent evaluation of construction progress.

[0055] As described above, only the specific embodiments of the present invention are provided, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed by the present invention should be covered by the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. A construction progress intelligent dynamic recognition and modeling method based on laser radar fusion binocular and depth camera data, characterized in that: The following steps are involved: Data collection: The two-dimensional RBG image and three-dimensional point cloud data of the construction site containing the construction progress information are collected through a multi-mode construction progress data collection system, wherein the multi-mode construction progress data collection system includes an outdoor macro-scale data collection system and an indoor micro-scale data collection system. The outdoor macro-scale data collection system includes dynamic fixed measuring points and unmanned aerial vehicles, and the indoor micro-scale data collection system includes a bionic robot. The dynamic fixed measuring points, the unmanned aerial vehicle and the bionic robot are respectively equipped with multi-mode data collection equipment, and the multi-mode data collection equipment includes a binocular camera, a depth camera or a laser radar; Data processing and analysis: Using the artificial intelligence-based construction status recognition algorithm, the collected 2D RBG images and 3D point cloud data are fused and processed to identify the type information, spatial position information, and time characteristic information of the components. The construction progress is quantified based on the spatial volume difference, and the construction progress information of each component is obtained to form a component point cloud data set; Dynamically reconstruct three-dimensional progress based on building information model: utilize the component point cloud data set to dynamically reconstruct progress information through building information model BIM.

2. The construction progress intelligent dynamic recognition and modeling method based on laser radar fusion binocular and depth camera data according to claim 1 is characterized in that: The data collection step includes outdoor macro-scale data collection and indoor micro-scale data collection.

3. The construction progress intelligent dynamic recognition and modeling method based on laser radar fusion binocular and depth camera data according to claim 2 is characterized in that: The outdoor macro-scale data collection includes: Through dynamic fixed measuring points, fixed monitoring measuring points are arranged on the facades of buildings near the construction site, and the positions of the measuring points are dynamically adjusted based on the multi-objective optimization algorithm to achieve the largest range of image data collection in the construction site; By using unmanned aerial vehicles equipped with binocular cameras and lidars, combined with multi-objective optimization algorithms, the optimal flight path for different construction stages is determined, and linked with dynamic fixed measuring points to supplement data collection.

4. The construction progress intelligent dynamic recognition and modeling method based on laser radar fusion binocular and depth camera data according to claim 2 is characterized in that: The indoor micro-scale data collection is carried out by using a bionic robot equipped with a binocular camera, a depth camera and a lidar to collect construction data of indoor scenes according to an optimized detection path.

5. The construction progress intelligent dynamic recognition and modeling method based on laser radar fusion binocular and depth camera data according to claim 1 is characterized in that: The data processing and analysis steps include: Based on the multimodal 3D target detection algorithm, target detection and segmentation are performed on the collected 3D point cloud data to identify the component point cloud, where the components include beams, plates, columns, and walls; Label the corresponding component point cloud with type information, spatial location information, and time information; According to the component point cloud data collected at different times, the spatial three-dimensional feature comparison of the components is carried out on the basis of the alignment type information, spatial position information and time information. By calculating the volume difference of the components, the component progress quantification based on the spatial volume is realized, and finally the construction progress information of each component is obtained; The component point cloud data set is formed based on type information, spatial position information, time information and construction progress information of each component.

6. The construction progress intelligent dynamic recognition and modeling method based on laser radar fusion binocular and depth camera data according to claim 1 is characterized in that: The three-dimensional progress dynamic reconstruction based on the building information model specifically includes: Based on the time information of each component point cloud data, the type information and spatial position information of the annotation of the component point cloud data set are compared with the type and spatial position of the components in the building information model BIM, and the component point cloud is aligned with its corresponding component model; According to the point cloud model corresponding to each component model, the real 3D geometric features of the component at that moment are reconstructed, and the 3D model in the building information model BIM is updated to replace the construction model corresponding to the previous moment; Components that have not yet been constructed are represented by virtual component models in the building information model, and components that have been completed are represented by actual component models.

7. The construction progress intelligent dynamic recognition and modeling method based on laser radar fusion binocular and depth camera data according to claim 1 is characterized in that: It also includes a result display step: presenting the construction progress through a three-dimensional dynamic visualization platform, including the time, space and progress status information of the components.

Images