Project progress automatic identification method based on multi-modal computer vision

Abstract

The invention discloses an automatic engineering progress identification method based on multi-modal computer vision. The method comprises the steps of data acquisition, preprocessing, multi-task parallel model reasoning (including at least three tasks of object counting, state analysis and the like), data post-processing, BIM and plan progress data fusion, visualization and early warning. Through multi-source time sequence data integration, Kalman filtering / Bayesian fusion and BIM space-time alignment technologies, full-process automatic identification of the progress is realized, limitation of a single task is broken through, monitoring real-time performance and accuracy are improved, a digital management and control system is constructed, a complex field environment is adapted, and visual support is provided for progress decision making.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, deep learning, and engineering management technology, specifically to an automatic engineering progress recognition method based on multimodal computer vision. Background Technology

[0002] Project schedule management is a core control link in the entire life cycle of a construction project, directly affecting cost control, quality assurance, safety management, and on-time delivery. Especially in large and complex projects, schedule deviations can trigger a chain reaction, causing huge economic losses. Traditional project schedule monitoring mainly relies on manual inspections, paper-based records, and periodic acceptance reports. This not only requires a large investment of manpower but also has obvious shortcomings: manual inspections have limited coverage and cannot capture real-time dynamic changes on the construction site, resulting in data lag; relying on personnel experience to judge key information such as component completion status and material arrival quantities is highly subjective and prone to statistical errors and information omissions; and untimely data transmission among participating parties creates information silos, leading to delays in schedule analysis and decision-making adjustments.

[0003] With the deepening of digital and intelligent transformation in the construction industry, computer vision technology, with its advantages of non-contact monitoring and efficient data acquisition, has gradually become an important technical means for on-site engineering monitoring. However, existing technologies still have many problems that urgently need to be solved: First, their functions are limited, with most technologies focusing only on a single monitoring task, such as locating construction machinery, counting precast components, or providing safety warnings for construction areas. They lack the ability to integrate and analyze multi-dimensional progress-related data such as component status, construction activities, and quantity calculations, making it difficult to comprehensively reflect the actual progress of the project. Second, the depth of data fusion is insufficient. Existing methods have failed to fully achieve deep integration with Building Information Modeling (BIM) and project schedule data, lacking an effective spatiotemporal alignment mechanism. They cannot accurately correlate multi-source data collected on-site with digital models and project plans, resulting in discrepancies between the physical site and the digital data. The existing technologies suffer from several problems: First, they are disconnected from the established models, resulting in a lack of systematic and coherent progress analysis results. Second, they lack robustness. Construction site environments are complex and variable, and multi-source image and video data are susceptible to factors such as dust, lighting changes, occlusion, and equipment vibration. Current technologies are inadequate in noise suppression and data preprocessing adaptability, and the fusion methods for multi-model inference results are not perfect, leading to insufficient accuracy and stability to meet the refined management needs of actual projects. Third, they lack practicality. Existing technologies are mostly still in the laboratory verification stage, lacking adaptation and optimization for actual engineering scenarios. The output results are also limited in format, making it difficult to provide managers with intuitive and actionable progress decision support. These issues prevent existing technologies from truly replacing traditional manual monitoring methods and fail to meet the urgent needs of the construction industry for automated, precise, and real-time control of project progress. Therefore, there is an urgent need for an automatic project progress identification method that can integrate multi-source data, achieve multi-task collaborative analysis, and deeply fuse digital models and planning data to address these industry pain points. Summary of the Invention

[0004] This invention aims to address the technical shortcomings of existing engineering progress monitoring methods, such as high reliance on manual labor, poor real-time performance, insufficient integration of multi-dimensional data, weak integration with digital models, and weak robustness of recognition results. It provides an automatic engineering progress recognition method based on computer vision, which realizes automated, multi-dimensional, accurate recognition and dynamic early warning of engineering progress.

[0005] To achieve the above objectives, the present invention employs the following technical means:

[0006] An automatic engineering progress recognition method based on multimodal computer vision includes the following steps:

[0007] Data acquisition steps: Using various image acquisition devices deployed at the construction site, multi-source time-series images and video data of the construction site are acquired at preset time intervals;

[0008] Data preprocessing steps: Perform preprocessing operations including denoising, correction, annotation and data augmentation on the acquired multi-source time-series image and video data to form a standardized training set and inference dataset;

[0009] Model inference steps: Using a pre-trained deep learning model, perform at least three of the following analysis tasks in parallel on the pre-processed data: object counting task, state analysis task, behavior understanding task, and volume measurement task.

[0010] Data post-processing and fusion steps: The multi-dimensional identification results output by the model inference steps are post-processed, including deduplication, correlation and confidence calibration, and the processed results are spatiotemporally aligned and fused with the building information model and the project schedule data to generate unified and structured current progress information;

[0011] Progress visualization and reporting steps: Based on the fused current progress information, automatically generate a visualized progress report, a 3D progress comparison model, and a progress deviation warning.

[0012] Preferably, in the object counting task, YOLO, Faster R-CNN or RetinaNet object detection models are used to identify and count prefabricated components, steel bar bundles, formwork and construction machinery, and the statistical results are used for material arrival and installation efficiency analysis.

[0013] Preferably, in the state analysis task, DeepLab and Mask R-CNN segmentation models are used to classify the state of components. The state includes not started, installation in progress, completed, and pending acceptance. The pouring and curing status of beams, slabs, and columns is determined through pixel-level analysis.

[0014] Preferably, the volume measurement task specifically includes:

[0015] Dense point clouds are generated from image sequences using structure-of-motion reconstruction technology and multi-view stereo vision algorithms.

[0016] Register the point cloud with the design model;

[0017] By calculating the volume change of point cloud within a specific area or the volume difference with the design model, the volume of earthwork excavation / backfill and concrete pouring can be automatically calculated.

[0018] Preferably, in the behavior understanding task, a 3D CNN or temporal Transformer model is used to analyze video clips to identify the types of construction activities, including excavation, hoisting, pouring, and welding, and the intensity and efficiency of the construction activities are evaluated in combination with the object counting results.

[0019] Preferably, in the data post-processing and fusion step, Kalman filtering or Bayesian fusion methods are used to fuse the identification results from different models and different time points to improve the accuracy and robustness of the data; and the fusion results are associated one by one with the components in the BIM model to achieve synchronous updates between the physical site and the digital model.

[0020] Preferably, the report generated in the progress visualization and reporting step includes at least: a comparison chart of planned progress and actual progress displayed in the form of a Gantt chart, a component completion status displayed in color coding on the 3D BIM model, a resource consumption and efficiency analysis chart generated based on volume measurement and behavior understanding results, and an early warning notification automatically triggered when a progress deviation exceeds a threshold.

[0021] Preferably, in the data preprocessing step, the denoising operation uses Gaussian filtering or median filtering algorithm, the correction operation uses perspective transformation to correct image distortion, and the data enhancement operation includes image flipping, scaling, color gamut transformation, and random cropping.

[0022] Preferably, in the model inference step, the parallel execution of multiple analysis tasks is achieved through a distributed computing framework, which dynamically allocates computing resources according to the computational complexity of different tasks to improve inference efficiency.

[0023] The present invention has the following beneficial effects:

[0024] 1. Achieve fully automated identification of project progress. Through multi-source time-series data acquisition and multi-task parallel reasoning, core progress monitoring tasks such as component counting, status judgment, quantity calculation, and construction activity analysis can be completed without manual intervention. This significantly reduces the labor costs of manual inspection and data statistics, avoids human error and information omission, and shortens the progress data update cycle from several days in the traditional manual process to hours, significantly improving the real-time performance and efficiency of progress monitoring.

[0025] 2. Breaking through the limitations of existing single-task monitoring technologies, this technology integrates multi-dimensional analysis tasks such as object counting, state analysis, behavior understanding, and volume measurement to comprehensively cover the core data dimensions related to project progress, forming a full-chain progress perception of "components-activities-workload". This solves the problem that existing technologies cannot fully reflect the actual progress of projects and provides more systematic and complete data support for progress assessment.

[0026] 3. Achieve deep integration of multi-source data and digital models. Improve the accuracy and robustness of multi-model inference results through methods such as Kalman filtering and Bayesian fusion. At the same time, with the help of BIM model spatiotemporal alignment and component association technology, realize the dynamic synchronous update of physical site data and digital models, and build a three-in-one progress management system of "site-model-plan". This solves the pain points of physical and digital disconnect and lack of correlation in progress analysis in existing technologies, and lays the foundation for digital progress control.

[0027] 4. It possesses strong environmental adaptability and practical value. Through targeted preprocessing solutions (noise suppression, distortion correction, and data augmentation), it adapts to the complex environment of the construction site, improving the stability of the technology in scenarios such as dust, changes in lighting, and occlusion. At the same time, it outputs diverse results such as Gantt chart comparison, BIM color coding visualization, resource efficiency analysis, and deviation early warning, directly providing managers with intuitive and operable decision support, effectively supporting timely adjustment of schedule deviations, reducing the risk of project delays, and meeting the needs of refined engineering management. Detailed Implementation

[0028] The technical solution of the present invention will be clearly and completely described below. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] This invention provides an automatic project progress identification method based on multimodal computer vision. The invention constructs an automatic project progress identification system integrating "multi-source perception, intelligent analysis, digital fusion, and precise control," achieving fully automated and precise progress management through five core steps, specifically including the following steps:

[0030] 1. Data Acquisition Steps: Scenario-based Device Deployment and Dynamic Acquisition Strategies

[0031] Core objective: To achieve "no blind spots, all-time, and highly adaptable" data collection at the construction site, providing a high-quality data source for subsequent analysis.

[0032] - Equipment selection and deployment specifications:

[0033] Fixed high-definition camera: Select an industrial-grade camera with 2 megapixels or higher (supports IP67 dust and water resistance, and wide operating temperature range of -20℃ to 60℃). The lens focal length should be adjusted according to the monitoring range (8mm focal length for close-range monitoring at 10-20m, and 25mm focal length for wide-range monitoring at 50-100m). The deployment location should avoid the tower crane's rotation radius and material transport channels, and be fixed with embedded parts (3-5m above the working surface) to ensure that the shooting angle covers the component installation point and the core area of ​​the material stacking area.

[0034] - Mobile data collection equipment: The inspection robot is equipped with a panoramic camera (360° field of view, supports automatic obstacle avoidance) and completes one inspection of the work surface per hour along the preset track; the drone is a quadcopter industrial-grade model (endurance ≥40 minutes, equipped with a 1-inch CMOS camera) and takes aerial photos according to the "grid route" (flying altitude 50-80m above the work surface, overlap rate ≥80%). In severe weather (wind speed >5, heavy rain), it switches to ground mobile data collection.

[0035] - Auxiliary equipment: Add supplementary lights in backlit areas (supporting automatic light-sensing on / off), and equip cameras with automatic cleaning devices (air jet dust removal every 2 hours) in areas with high dust concentration (such as earthwork operation areas).

[0036] Dynamic data acquisition strategy:

[0037] Time intervals: every 30 minutes during the foundation construction phase (earthwork, rebar tying); every 15 minutes during the main structure construction phase (concrete pouring, component hoisting); every 2 hours during the decoration and finishing phase; real-time data collection mode is activated for key processes (such as large-volume concrete pouring).

[0038] Data is categorized and stored as follows: Image data is named according to "Project-Building-Date-Time Period" (e.g., "Commercial Center-Building A-20251208-0900"), and video data is compressed using H.265 encoding (to save storage space). The original data is stored on the local edge server for 30 days and backed up to the cloud database for long-term storage.

[0039] 2. Data preprocessing steps: Engineering-based data cleaning and standardization

[0040] Core objective: To solve the problem of data noise at construction sites, output standardized data that meets the model input requirements, and improve the accuracy of subsequent inference.

[0041] - Scene-specific preprocessing solutions:

[0042] Noise reduction: Median filtering (window size 5×5) is used in dusty environments, bilateral filtering (preserving details) is used in low-light nighttime environments, and guided filtering is used to remove raindrop noise in rainy images.

[0043] - Correction processing: Camera intrinsic parameters are obtained through Zhang Zhengyou calibration method, and perspective transformation is used to correct tilted images (correcting the construction plane to a front view) to ensure that the component size measurement error is ≤2cm; for images with uneven lighting, the CLAHE algorithm is used to enhance contrast (with a contrast threshold of 2.0).

[0044] - Labeling and Enhancement: A "semi-automatic labeling + manual review" model is adopted - LabelStudio tool is used to automatically pre-label (based on historical labeling models), and manual correction is only used to correct erroneous labels (efficiency improvement of 60%); the labeling content must include component ID (consistent with the BIM model), status, location and other information; data enhancement adopts a "scene-adaptive" strategy: sand and dust simulation enhancement is added to the earthwork operation area, low light enhancement is added to the night operation area, and snow scene simulation enhancement is added to the winter construction area.

[0045] Data quality inspection: Set quality thresholds (images with low clarity or occlusion area > 50% are directly removed), and automatically generate a preprocessing report daily (including the amount of valid data and statistics on the reasons for data removal) to ensure that the pass rate of training set data is ≥ 95%.

[0046] 3. Model inference steps: Multi-task parallelism and adaptation to engineering scenarios

[0047] Core objective: To improve inference efficiency and meet the real-time requirements of engineering while ensuring accuracy.

[0048] - Model selection and parameter engineering configuration:

[0049] The core parameters of the model for task adaptation (engineering adaptation) and application scenario optimization are as follows: Object counting: YOLOv9 (dense component scene), Faster R-CNN (wide road scene). Input resolution: 640×640, confidence threshold: 0.55, NMS threshold: 0.3. Rebar bundle counting: Add morphological feature assistance (avoid misjudging a single bar as a bundle). State analysis: Mask R-CNN (building components), DeepLabv3+ (road structure layer). Backbone network: ResNet101, output channels: 4 (corresponding to 4 states). Concrete pouring state: Add humidity feature recognition (combined with infrared data). Behavior understanding: Temporal Transformer (complex actions), 3D CNN (simple actions). Video clip length: 20 frames, frame interval: 3 frames, learning rate: 1e-4. Lifting behavior: Add judgment of the connection status between the lifting device and the component (to prevent fall risk). Volume measurement: SfM+ multi-view stereo vision. Point cloud density: 0.03m / point, registration error: ≤2cm. Earthwork calculation: Add terrain slope correction coefficient.

[0050] - Parallel inference and resource scheduling: Based on the Kubernetes distributed framework deployment model, task priorities are set (volume measurement > state analysis > behavior understanding > object counting). During peak construction periods (such as 8-10 am), the number of computing nodes is automatically expanded (the number of GPUs increases from 4 to 8), and the inference latency is controlled within 5 minutes; during off-peak periods, the number of nodes is automatically reduced to reduce hardware costs.

[0051] 4. Data post-processing and fusion steps: Digital twin linkage and progress calibration

[0052] Core objective: To achieve precise integration of multi-source data and digital models, and generate structured progress data that meets the needs of engineering management.

[0053] - Multi-dimensional result optimization:

[0054] Post-processing: Duplicate identification results were removed using an IOU threshold of 0.7, and confidence level calibration was performed (results below 0.8 were marked as "to be reviewed"); the volume measurement results were smoothed by using a moving average method (3 measurements within a window).

[0055] - Fusion method: Short-term data (within 1 hour) is filtered by Kalman filtering (prediction + update) to remove equipment jitter error, and long-term data (within 1 day) is fused by Bayesian fusion (combined with historical accuracy weights) to improve stability; for example, component counting results are fused with "fixed camera + drone" data, and the weights are dynamically allocated according to the monitoring distance (0.7 weight for close-range cameras and 0.3 weight for drones).

[0056] Deep BIM Integration and Spatiotemporal Alignment:

[0057] Format compatibility: Supports mainstream BIM software formats such as Revit, Civil 3D, and Navisworks, and enables lightweight model import through API interface (removing redundant information and improving loading speed).

[0058] Spatiotemporal alignment: Based on the "component ID + time axis" of the BIM model, the identification results are associated with the BIM components one by one (e.g., "column-C301" corresponds to the "completed" status of the identification), with a time alignment error of ≤10 minutes; the "planned progress - actual progress - deviation rate" data of the components are updated in real time in the BIM model.

[0059] Integration with management systems: Structured progress data is pushed to the smart construction site platform and Project management system via HTTP interface, supporting custom data formats (to meet the data needs of different organizations).

[0060] 5. Progress visualization and reporting steps: hierarchical control and precise early warning

[0061] Core objective: To output intuitive and hierarchical progress information to achieve "early problem detection and early accountability".

[0062] - Multi-dimensional visualization:

[0063] BIM 3D visualization: Navisworks uses color coding to display the status of components in real time (gray for not started, yellow for installation in progress, green for completed, blue for pending acceptance, and red for overdue), and supports clicking on components to view detailed information (such as installation time, construction team, and acceptance records).

[0064] - Progress comparison charts: Generate "Plan vs. Actual" Gantt charts (support filtering by building and process), bar charts of project completion status (such as daily statistics of concrete pouring volume), and construction efficiency trend charts (such as hourly statistics of hoisting efficiency).

[0065] - Mobile adaptation: A lightweight mini-program has been developed, allowing construction workers to view real-time progress data for their assigned areas, and managers to receive alerts via the app.

[0066] Tiered reporting and early warning mechanism:

[0067] Report Type | Recipients | Core Content | Push Frequency | Team Daily Report | Construction Team Leader | Daily Completed Workload, Unfinished Tasks, and Tomorrow's Plan | Daily 18:00 | Project Weekly Report | Project Manager and Supervisor | Progress Deviation Analysis, Resource Allocation Efficiency, and Quality Issue Statistics | Sunday 20:00 | Early Warning Notification | Relevant Responsible Persons (Tiered Push) | Deviation Content, Scope of Impact, Rectification Suggestions, and Time Limits | Real-time Push

[0068] Early warning and graded response:

[0069] Level 1 warning (deviation ≤ 5%): Push to the construction team leader, and provide a rectification plan within 24 hours;

[0070] Level 2 warning (5% < deviation ≤ 10%): Notified to project manager and supervisor, and a special meeting should be organized within 48 hours;

[0071] Level 3 warning (deviation > 10%): The warning is sent to the construction unit and the quality supervision station, and a special plan to correct the progress deviation is initiated.

[0072] Engineering support system

[0073] 1. Hardware maintenance and support

[0074] Establish a "daily inspection + automatic monitoring" operation and maintenance mechanism: assign one dedicated person to check the equipment power supply and network connection every day (ensure bandwidth ≥100Mbps), and calibrate the camera parameters once a week; the system automatically monitors the equipment status, and if a fault occurs (such as offline or blurry image quality), immediately push the operation and maintenance work order to the person in charge, with a repair response time of ≤2 hours.

[0075] 2. Personnel Training System

[0076] Specialized training is conducted for different positions: construction workers are trained on "equipment avoidance standards" and "key points for shooting at the work site"; technicians are trained on "model parameter adjustment" and "data quality analysis"; and management personnel are trained on "visual platform operation" and "early warning response procedures" to ensure that personnel at all levels are proficient in the relevant skills.

[0077] 3. Data security protection

[0078] It adopts dual encryption storage of "local + cloud" and data transmission uses SSL encryption protocol; it sets hierarchical access (construction personnel can only view data in their local area, while management personnel can view data for the entire project); and it performs regular (monthly) data backups and security audits to prevent data leakage or tampering.

[0079] Specific experimental verification

[0080] I. Experimental Objective

[0081] This study verifies the accuracy, real-time performance, environmental adaptability, and data fusion reliability of the multimodal computer vision-based automatic engineering progress recognition method in real-world engineering scenarios. It also quantitatively evaluates the performance of the method in core functions such as component counting, status recognition, quantity calculation, and progress deviation early warning, and verifies its advantages over traditional methods and existing single-task recognition technologies.

[0082] II. Experimental Environment and Data

[0083] 1. Experimental Scenario

[0084] Parallel experiments were conducted using three typical engineering scenarios:

[0085] Scenario 1: Construction phase of the main structure of a 30-story high-rise building (including components such as beams, slabs, columns, and shear walls, involving activities such as hoisting, pouring, and welding);

[0086] Scenario 2: 5km municipal road reconstruction and expansion project (including roadbed, base course, and surface course construction, involving paving, compaction, and road marking activities);

[0087] Scenario 3: Construction phase of 8 prefabricated residential buildings (including installation of prefabricated exterior wall panels and composite slabs, involving activities such as hoisting and grouting at joints).

[0088] 2. Hardware Deployment

[0089] Deploy data acquisition equipment according to the technical solution:

[0090] Fixed high-definition cameras: resolution 2560×1440 (scene 1 / 3), 1920×1080 (scene 2), with 12, 10, and 8 units deployed respectively;

[0091] Mobile devices: Inspection robots equipped with mobile cameras (3 units / Scene 1, 4 units / Scene 2, 2 units / Scene 3) and multi-rotor drones (2 units / Scene 1, 1 unit / Scene 2, 1 unit / Scene 3).

[0092] Computing equipment: Edge computing server equipped with GPU (NVIDIA A100) to support distributed parallel inference.

[0093] 3. Experimental Data

[0094] Data collection: Data was collected continuously for 30 days in each of the three scenarios, resulting in a total of 150,000 multi-source time-series images and 8,000 video clips, covering complex environments such as dust, strong light, obstruction, and nighttime construction.

[0095] Annotated data: Three senior engineering technicians jointly annotated key information such as component type, status, and construction activities, and constructed a validation set containing 24,000 annotated images (annotation consistency ≥ 95%).

[0096] Benchmark data: The actual values ​​of component counts, measured quantities, and planned progress benchmark values ​​are obtained through manual hourly inspection records, total station measurements, and BIM design model export, and are used as the basis for verification and comparison.

[0097] III. Experimental Evaluation Indicators

[0098] Design a quantitative evaluation index system:

[0099]

[0100] IV. Experimental Procedure

[0101] 1. Model Training and Parameter Configuration

[0102] Training deep learning models according to technical solutions:

[0103] Object counting: YOLOv9 (scene 1 / 3), Faster R-CNN (scene 2), confidence threshold 0.5-0.55;

[0104] State recognition: Mask R-CNN (scenes 1 / 3), DeepLabv3+ (scene 2), with ResNet50 / 101 as the backbone network;

[0105] Behavioral understanding: Temporal Transformer (Scenes 1 / 3), 3D CNN (Scene 2), video clip length 16-22 frames;

[0106] Volume measurement: SfM technology + multi-view stereo vision algorithm, point cloud density 0.03-0.05m / point;

[0107] Data fusion: Kalman filtering + Bayesian fusion method, spatiotemporal alignment based on BIM component ID association.

[0108] 2. Experimental Procedure

[0109] Data preprocessing: Perform noise reduction, correction, and enhancement operations on the collected data to generate a standardized inference dataset;

[0110] Multi-task parallel reasoning: Initiate the model reasoning process and execute component counting, state analysis, behavior understanding, and volume measurement tasks in parallel;

[0111] Data fusion and schedule generation: Post-process the inference results and merge them with BIM and planned schedule data to generate structured schedule information and early warnings;

[0112] Result comparison and verification: The identification results and progress data are compared with the manually measured baseline data one by one, and the evaluation indicators are recorded.

[0113] Environmental robustness testing: Targeted testing of index changes under scenarios such as dust, strong light, shading, and nighttime to verify adaptability.

[0114] V. Experimental Results and Analysis

[0115] 1. Core performance indicator results

[0116]

[0117] 2. Comparison of experimental results

[0118] Compared with traditional manual methods: the progress data update cycle is shortened from "several days" to "within 30 minutes", labor costs are reduced by more than 75%, and counting / status recognition errors are reduced by 60%;

[0119] Compared with existing single-task technologies: After multi-dimensional data integration, the comprehensiveness of progress assessment is improved by 40%, the data correlation after BIM integration is improved by 55%, and the robustness in complex environments is improved by 30%.

[0120] 3. Results Analysis

[0121] Experimental results show that:

[0122] The method of this invention achieves high accuracy in component counting, status identification, and quantity calculation in three typical engineering scenarios, with the calculation error controlled within 5.3%, meeting the needs of refined engineering management.

[0123] Data update delay ≤35 minutes, significantly improving the real-time performance of progress monitoring and solving the lag problem of traditional methods;

[0124] Despite complex environments such as dust and strong light, the core indicators did not show a significant decline. The fusion method effectively improved the reliability of the results, with a confidence level exceeding 91%.

[0125] With an accuracy rate exceeding 90%, the early warning system can accurately identify progress deviations and provide effective support for decision-making adjustments.

[0126] VI. Experimental Conclusions

[0127] The method of this invention, through multi-source data acquisition, multi-task parallel reasoning, deep BIM integration, and targeted preprocessing design, demonstrates high accuracy, strong real-time performance, excellent robustness, and high practicality in actual engineering scenarios. Its core performance indicators are superior to traditional manual methods and existing single-task recognition technologies. It can effectively solve the pain points of manual dependence, data disconnect, and incomplete coverage in engineering progress monitoring, and fully meet the actual needs of digital and refined progress management of building engineering.

[0128] Example 1

[0129] Automatic recognition of the progress of high-rise building construction

[0130] 1. Data Acquisition Deployment

[0131] For a 30-story high-rise building project, 12 fixed high-definition cameras (2560×1440 resolution) were deployed at key locations on the construction site, including the top of the tower cranes, the edges of each floor, the material storage area, and the entrance to the construction elevator. Three mobile inspection cameras (mounted on an inspection robot) and two multi-rotor drones were also deployed. Data collection intervals were dynamically adjusted according to the construction phases: every 40 minutes during the foundation construction phase, every 1.5 hours during the main structure construction phase, and every 2.5 hours during the decoration and finishing phase. The drones conducted full-coverage aerial photography of the entire site twice a week, focusing on capturing the progress of the facade construction and the status of the roof work.

[0132] 2. Data Preprocessing

[0133] Median filtering was used to remove image noise caused by dust and smoke at the construction site, and perspective transformation was used to correct image distortion caused by camera installation angle deviations, ensuring the accuracy of component geometric dimension recognition. The LabelMe annotation tool was used to annotate targets such as precast beams, precast slabs, rebar bundles, formwork, tower cranes, concrete pump trucks, and construction workers, constructing a training set containing 8000 annotated images. Data augmentation operations, including horizontal / vertical flipping, scaling (0.7-1.3x), HSV color gamut adjustment (saturation ±15%, brightness ±20%), random cropping (cropping ratio 0.6-1.0), and Gaussian noise addition, generated 24000 enhanced images to improve the model's generalization ability, ultimately forming a standardized training and inference dataset.

[0134] 3. Model Training and Inference

[0135] Object counting task: Using the YOLOv9 object detection model, the input resolution is set to 800×800, the batch size is 24, the number of training iterations is 400 rounds, and the confidence threshold is 0.55. The task is to identify and count prefabricated components, steel bar bundles, formwork and other materials and construction machinery, and output daily material arrival verification reports and installation efficiency analysis results.

[0136] State analysis task: Using the Mask R-CNN segmentation model (backbone network is ResNet101), with 300 training iterations, pixel-level state classification (not started, under installation, completed, awaiting acceptance) is performed on components such as beams, slabs, columns, and shear walls. The pouring and curing status is determined by analyzing the coverage of the concrete pouring surface and the integrity of the curing layer.

[0137] Behavior understanding task: Using the temporal Transformer model, the input video clip length is set to 20 frames, the frame interval is 3 frames, the training iteration is 200 rounds, the model identifies construction activities such as hoisting, pouring, welding, and formwork, and calculates key indicators such as hoisting efficiency and pouring area per unit time by combining the object counting results.

[0138] Volume measurement task: Process UAV aerial image sequences using structure of motion restoration (SfM) technology to generate dense point clouds (point cloud density 0.03m / point), register the point clouds with the Revit format BIM design model (registration error ≤2cm), calculate earthwork excavation / backfill volume and concrete pouring volume, and update engineering quantity statistics in real time.

[0139] 4. Data Post-processing and Fusion

[0140] Kalman filtering is used to smooth the object counting and volume measurement results at different time points, removing outliers caused by equipment vibration and occlusion. The results of multi-model inference are integrated through Bayesian fusion method, and the progress confidence of each component is calculated (confidence ≥ 0.85 is considered a valid result). Based on the component ID of the BIM model, the fusion results are associated one-to-one with the digital model, and the progress status and engineering completion status of the components in the BIM model are dynamically updated to achieve real-time synchronization between the physical site and the digital model.

[0141] 5. Progress visualization and reporting

[0142] Daily automatic generation of multi-dimensional progress reports: displaying the comparison between planned and actual progress in Gantt chart format (supports export to Project and Primavera P6); displaying the component completion status of the 3D BIM model in Navisworks software using color coding (gray for not started, yellow for installation in progress, green for completed, and blue for pending acceptance); generating comparison charts of steel reinforcement usage and concrete pouring volume with planned values; when the progress deviation of a component exceeds 8%, sending SMS and platform message alerts to the project manager and construction supervisor through the project management platform.

[0143] Example 2

[0144] Automatic recognition of municipal road construction progress

[0145] 1. Data Acquisition Deployment

[0146] For a 5km-long urban main road reconstruction and expansion project, one fixed high-definition camera (1920×1080 resolution) was deployed every 500 meters along the road's centerline, along with four mobile inspection cameras (mounted on engineering vehicles) and one long-endurance UAV. Data collection intervals were set as follows: every hour during the roadbed construction phase, every 30 minutes during the pavement paving phase, and every two hours during the ancillary works construction phase. The UAV conducted a full-section aerial survey once daily, focusing on roadbed compaction, pavement paving progress, and intersection construction status.

[0147] 2. Data Preprocessing

[0148] Gaussian filtering combined with bilateral filtering algorithms was used to remove image noise caused by dust and vehicle exhaust at road construction sites; perspective transformation was used to correct distortion caused by camera tilt installation to ensure the accuracy of road cross-section dimension measurement; the LabelMe annotation tool was used to annotate construction machinery such as road rollers, pavers, and loaders, road structural layers such as subgrade, base course, and surface course, as well as auxiliary components such as manhole covers and guardrails, to construct a training set containing 6,000 annotated images; through data augmentation operations such as image flipping, scaling (0.8-1.2 times), brightness adjustment, random cropping, and affine transformation, 18,000 augmented images were generated to form a standardized training set and inference dataset.

[0149] 3. Model Training and Inference

[0150] Object counting task: The Faster R-CNN object detection model is used with an input resolution of 640×640, a batch size of 16, 350 training iterations, and a confidence threshold of 0.5. The model counts the number of construction machinery and the number of auxiliary components installed to support the analysis of construction resource allocation efficiency.

[0151] State analysis task: Using DeepLabv3 + segmentation model, trained for 250 rounds, the state of road structure layers is classified (not under construction, under construction, completed, to be detected), and the subgrade compaction range, pavement paving thickness and smoothness are determined through pixel-level analysis;

[0152] Behavior understanding task: Using a 3D CNN model, the input video clip is 16 frames long with a frame interval of 2 frames. The training iterations are 180 rounds. The model identifies construction activities such as roadbed compaction, pavement paving, road marking, and guardrail installation. The construction intensity is assessed in conjunction with the number of construction machines.

[0153] Volume measurement task: By processing drone aerial images and images acquired by fixed cameras through multi-view stereo vision algorithms, a dense point cloud along the road is generated. After registering the point cloud with the road design model, the volume of roadbed excavation / backfill and pavement paving is calculated to achieve real-time calculation of engineering quantities.

[0154] 4. Data Post-processing and Fusion

[0155] A Bayesian fusion method is used to fuse the inference results of multiple models, improving the accuracy of construction machinery counting and road structure layer status identification. Kalman filtering is used to smooth the engineering quantity calculation results and reduce the interference of environmental factors. The fusion results are associated with the stationing and cross-section information of the road BIM model (Civil 3D format) to achieve accurate matching between progress data and road linearity and structural layer design, and dynamically update the construction progress information in the BIM model.

[0156] 5. Progress visualization and reporting

[0157] Daily road construction progress reports are generated: segmented Gantt charts are used to compare the planned and actual progress of each construction segment; the completion status of each structural layer is displayed in color coding on the 3D road BIM model; comparison charts are generated to show the efficiency of construction machinery shifts, the completion status of the project, and the planned values; when the progress deviation of a construction segment exceeds 10% or the pavement paving thickness deviation exceeds the allowable range of the specification, an early warning notification is automatically triggered and pushed to the project management personnel and supervision unit.

[0158] Example 3

[0159] Automatic recognition of the progress of prefabricated building projects

[0160] 1. Data Acquisition Deployment

[0161] For a prefabricated residential community project (comprising 8 18-story residential buildings), 8 fixed high-definition cameras (2560×1440 resolution) were deployed at key locations such as the top of the tower cranes, component stacking areas, installation work surfaces, and transportation channels in each building. Two mobile inspection cameras and one drone were also deployed. Data collection intervals were set as follows: every 45 minutes during the component hoisting phase, every 1.5 hours during the node connection phase, and every 2 hours during the waterproofing and decoration phase. The drone conducted aerial photography of each building three times a week, focusing on capturing the installation progress of prefabricated components on the facade and the status of the roof construction.

[0162] 2. Data Preprocessing

[0163] Median filtering was used to remove image noise caused by dust from the construction site and dust generated during component transportation. Perspective transformation was used to correct distortion caused by aerial camera shooting to ensure accurate identification of prefabricated component dimensions and installation precision. The LabelMe annotation tool was used to annotate prefabricated components such as prefabricated exterior wall panels, composite slabs, stairs, and balcony slabs, as well as targets such as hoisting machinery, connection nodes, and waterproof membranes, to build a training set containing 7,000 annotated images. Data augmentation operations such as image flipping, scaling (0.75-1.25 times), color gamut transformation, random cropping, and mirror transformation were used to generate 21,000 enhanced images, forming a standardized training set and inference dataset.

[0164] 3. Model Training and Inference

[0165] Object counting task: Using the RetinaNet object detection model, with an input resolution of 720×720, a batch size of 20, 380 training iterations, and a confidence threshold of 0.52, the task counts the number of prefabricated components arriving on site and the number of components completed in installation, supporting the analysis of component arrival acceptance and installation progress.

[0166] State analysis task: Using the Mask R-CNN segmentation model (backbone network is ResNet50), with 280 training iterations, the installation status of prefabricated components (not installed, hoisting, installation completed, node connection completed, waterproofing completed) is classified, and the integrity of the connection nodes and the coverage of the waterproof layer are judged through pixel-level analysis.

[0167] Behavior understanding task: Using a temporal Transformer model, the input video clip is 22 frames long with a frame interval of 3 frames. The training iterations are 220 rounds. The model identifies activities such as hoisting of prefabricated components, grouting of nodes, and waterproofing construction. The installation efficiency is evaluated by combining the component counting results.

[0168] Volume measurement task: Process multi-view images using motion recovery structure technology to generate a point cloud model of the prefabricated component installation area. After registration with the BIM design model, verify the component installation position deviation and splicing gap size to ensure that the installation quality meets the specifications.

[0169] 4. Data Post-processing and Fusion

[0170] A method combining Kalman filtering and Bayesian fusion is adopted to fuse the results of prefabricated component counting, installation status identification, and installation deviation measurement, thereby improving data accuracy and robustness. Based on the unique component code of the BIM model, the fusion results are accurately associated with the prefabricated components in the digital model, and the component installation progress and quality inspection results are dynamically updated, realizing digital management and control of the entire construction process of prefabricated buildings.

[0171] 5. Progress visualization and reporting

[0172] Daily prefabricated building progress reports are generated: the planned installation progress and actual progress of each building and component type are compared in the form of Gantt charts; the installation and node processing status of prefabricated components are displayed in color coding on the 3D BIM model; a full-process traceability report of prefabricated component arrival, installation and acceptance is generated; when the component installation progress deviation exceeds 7% or the installation deviation exceeds the allowable value of the specification, an early warning is automatically triggered and pushed to the construction management personnel, supervision unit and component manufacturer to ensure that the problem is rectified in a timely manner.

[0173] Example 4

[0174] Automatic recognition of subway station construction progress

[0175] 1. Data Acquisition Deployment

[0176] For a transfer station project of Metro Line 10 (3 underground levels, 26m deep foundation pit), the data acquisition equipment was deployed according to the principle of "zonal coverage + key focus": 16 low-light enhanced fixed high-definition cameras (resolution 2560×1440, supporting 0.01 lux low-light imaging) were deployed on the top of the retaining piles around the foundation pit (1 unit per 20m), at the entrance and exit of the shield tunnel shaft (2 units each), and on the working face of the main structure of the station (4 units per level); 4 explosion-proof mobile inspection cameras (mounted on a rail-mounted inspection vehicle) were configured to adapt to the damp and dusty underground environment; a long-endurance vertical take-off and landing UAV (60-minute flight time) was selected to conduct aerial photography of the foundation pit excavation face and the ground material yard 3 times a week, and in severe weather (such as a foundation pit rainstorm warning), the underground inspection vehicle was used for full coverage data acquisition.

[0177] Data acquisition intervals are dynamically adjusted: every 20 minutes during the excavation phase; every 10 minutes during the tunnel boring machine (TBM) phase (simultaneously capturing segment installation); every 15 minutes during the main structure pouring phase; and every hour during the finishing and equipment installation phase. The UAV focuses on acquiring image data related to the excavation pit slope displacement observation points, the TBM's attitude, and the segment assembly quality.

[0178] 2. Data Preprocessing

[0179] A combined algorithm of "median filtering + adaptive threshold denoising" is used to remove image noise caused by underground construction dust and water mist. To address the problem of uneven underground lighting, a brightness equalization algorithm is used to correct image grayscale deviation. The camera tilt is corrected by combining Zhang Zhengyou calibration method and perspective transformation (caused by the slope of the foundation pit working surface) to ensure that the measurement error of key dimensions such as the spacing of pipe segments and steel bars is ≤3mm.

[0180] The annotation adopts the "BIM-associated annotation" mode: the LabelMe tool is used to annotate targets such as tunnel boring machines, tunnel segments, steel mesh, formwork supports, and construction personnel. The annotation information is forcibly associated with the component IDs in the BIM model (such as "tunnel segment-T1008" and "retaining pile-W25"). Targeted data augmentation design: low-light environment simulation (brightness reduced by 30%-50%), pit water accumulation reflection simulation, and dust concentration gradient enhancement are added to construct a training set containing 9,000 annotated images. 27,000 enhanced images are generated through flipping, scaling (0.6-1.4 times), and affine transformation to ensure the model adapts to complex underground environments.

[0181] 3. Model Training and Inference

[0182] - Object counting task: Using the YOLOv9-Tiny lightweight model (adapted to underground edge computing devices), the input resolution is 640×640, the batch size is 32, the number of training iterations is 350, and the confidence threshold is 0.58. The task focuses on counting the number of shield tunnel segments (number of rings in a single excavation), steel bundles, and formwork. The daily output is a segment arrival acceptance report and steel consumption calculation results. The counting accuracy is ≥94%.

[0183] - State analysis task: The Mask R-CNN model (backbone network ResNet101) was used for 300 training iterations to classify the state of components as "not under construction, rebar tying in progress, formwork installation in progress, concrete pouring in progress, and completed". The state classification accuracy was ≥93% through pixel-level analysis to identify the filling status of the segment assembly sealant, the development of cracks on the foundation pit slope, and the compactness of the concrete pouring.

[0184] - Behavioral understanding task: Using a temporal Transformer model, the input video clip is 24 frames long (adapted to the continuous operation characteristics of the tunnel boring machine), with a frame interval of 2 frames and 220 training iterations. It identifies construction activities such as tunnel boring, segment assembly, concrete vibration, and rebar binding, and calculates indicators such as tunneling efficiency per unit time and segment installation speed by combining the tunnel boring machine's operating parameters (accessed through API interface).

[0185] - Volume measurement task: Using SfM technology combined with laser point cloud data (to supplement underground visual blind spots), generate dense point clouds (point cloud density 0.04m / point) of the foundation pit excavation surface and the main structure pouring area, register them with the Revit format subway BIM model (registration error ≤3cm), automatically calculate the foundation pit excavation volume, concrete pouring volume and segment assembly gap, with a relative calculation error ≤5%.

[0186] 4. Data Post-processing and Fusion

[0187] Kalman filtering was used to smooth time-series data such as tunnel boring machine (TBM) progress and segment counts, removing measurement anomalies caused by TBM vibration. A Bayesian fusion method was employed to integrate multi-source data from fixed cameras, inspection vehicles, and laser point clouds, calculating the progress confidence score for each component (a confidence score ≥ 0.88 was considered a valid result). Based on the BIM model's dual-dimensional correlation and fusion results of "component ID + construction zone," the progress data of the underground main structure, the TBM section, and surface ancillary facilities were updated in a coordinated manner, simultaneously generating a segment assembly quality traceability ledger (linked to construction teams and acceptance personnel information).

[0188] 5. Progress visualization and reporting

[0189] Daily subway station progress reports are generated: Gantt charts are used to compare the planned and actual progress of excavation, main structure, and tunnel boring machine (TBM) excavation (export to Primavera P6 is supported); the BIMFusion platform displays the component status of the 3D BIM model using color coding (gray for no construction, yellow for construction, green for completed, and red for quality abnormalities). Clicking on a segment component allows viewing details such as assembly time and sealant testing data; comparison charts of TBM shift efficiency and concrete consumption with planned values ​​are generated. When the TBM excavation speed deviation exceeds 15%, the segment assembly gap exceeds the specification (≥5mm), or the excavation slope displacement exceeds the warning value, an immediate SMS and APP warning is sent to the project manager, supervisor, and TBM team leader via the smart subway platform, requiring a response and rectification within 2 hours.

[0190] Example 5

[0191] Automatic progress identification of integrated utility tunnel projects

[0192] 1. Data Acquisition Deployment

[0193] For a certain city's underground integrated utility tunnel project (3.2km in length, including power, water and telecommunications, and gas compartments), one explosion-proof fixed high-definition camera (1920×1080 resolution, IP68 protection level, suitable for underground flammable and explosive environments) is deployed every 100m along the tunnel axis. Key deployments are made at compartment intersections, hoisting openings (3 cameras each), and the pipe jacking machine working face (2 cameras). Three tracked inspection robots (equipped with panoramic cameras and gas sensors) are configured to inspect the tunnel every 40 minutes along its internal tracks. Wind-resistant drones (50-minute flight time) are used to conduct aerial photography of the ground construction section and foundation pit excavation area once a day, focusing on collecting data on the progress of the tunnel foundation pit, prefabricated compartment hoisting, and pipeline installation.

[0194] Data acquisition intervals: every 30 minutes during the foundation pit excavation stage; every 15 minutes during the pipe jacking construction stage; every 20 minutes during the prefabricated module installation stage; and every hour during the pipeline insertion stage. All equipment data is transmitted to a ground-based edge server via fiber optic cable to avoid interference from underground wireless signals.

[0195] 2. Data Preprocessing

[0196] A combined algorithm of "Gaussian filtering + morphological opening operation" is used to remove image noise caused by dust and mud adhesion during underground pipe gallery construction. To address the problem of dim lighting at the pipe jacking working face, Gamma correction (Gamma value 1.8) is used to enhance image brightness. Perspective transformation is combined to correct the tilt shooting angle of the pipe gallery compartment to ensure that the measurement accuracy of the compartment size is ≤2cm.

[0197] LabelImg was used as the annotation tool to focus on annotating targets such as pipe jacking machines, prefabricated cabins, power cables, water supply pipes, and gas pipelines. The annotation information was linked to the cabin number in the BIM model (such as "Power Cabin-025") and the pipeline specifications. Data augmentation adopted a "scene customization" strategy: adding simulation of shadows inside the pipe gallery, image enhancement of mud splashes, and annotation of gas sensor data (such as image markings during gas leak warnings). A training set of 7,500 annotated images was constructed, and 22,500 enhanced images were generated by flipping, scaling (0.7-1.3 times), and color gamut transformation to improve the model's environmental adaptability.

[0198] 3. Model Training and Inference

[0199] - Object counting task: Using the Faster R-CNN model (adapted for long-distance pipe gallery target detection), with an input resolution of 720×720, a batch size of 20, 380 training iterations, and a confidence threshold of 0.52, the task counts the number of prefabricated sections, pipelines, and pipe jacking machine accessories, supporting the verification of prefabricated components entering the site and the analysis of pipeline laying progress, with a counting accuracy of ≥93.5%.

[0200] - Status Analysis Task: Using the DeepLabv3+ segmentation model, with 280 training iterations, the status of the utility tunnel structure and pipelines is classified as "not installed, hoisting, installation and fixing, pressure testing, and acceptance". The status of the prefabricated compartment splice joints, the integrity of pipeline interface connections, and the coverage of the waterproof layer of the compartment are identified through pixel-level analysis. The status classification accuracy is ≥92.8%.

[0201] - Behavioral understanding task: Using a 3D CNN model, the input video clip is 18 frames long with a frame interval of 2 frames. The training iterations are 200 rounds. The model identifies construction activities such as pipe jacking, prefabricated cabin hoisting, pipeline welding, and pressure testing. Combined with gas sensor data, it judges the safety and compliance of gas chamber construction (such as whether ventilation is turned on during welding operations).

[0202] - Volume measurement task: Process the images of the inspection robot and the fixed camera through multi-view stereo vision algorithm to generate dense point cloud inside the pipe gallery compartment. After registration with the Civil 3D format pipe gallery BIM model, calculate the prefabricated compartment assembly gap, pipeline laying space and remaining space in the compartment, to provide data support for subsequent equipment installation. The relative error of the calculation is ≤4.8%.

[0203] 4. Data Post-processing and Fusion

[0204] A Bayesian fusion method was employed to integrate multi-source identification results from ground cameras, underground inspection robots, and drones, significantly improving the accuracy of identifying pipeline interface status and prefabricated compartment sealing performance. Kalman filtering was used to smooth time-series data such as pipe jacking progress and compartment concrete pouring volume, reducing underground vibration interference. A three-level association system based on the BIM model—compartment-pipeline-component—precisely integrated the fusion results with the utility tunnel's BIM model, dynamically updating the construction progress of each compartment, pipeline installation location, and quality inspection data, achieving full lifecycle data traceability for the utility tunnel.

[0205] 5. Progress visualization and reporting

[0206] Daily integrated utility tunnel progress reports are generated: Segmented Gantt charts are used to compare the planned and actual progress of each tunnel section and compartment; color-coded displays of compartment structure and pipeline installation status are shown on the 3D utility tunnel BIM model, supporting filtering by compartment type; and full-process reports are generated for prefabricated module acceptance, pipeline welding quality, and pressure test results. Automatic tiered warnings are triggered when prefabricated module joints exceed specifications (≥3mm), pipeline pressure tests fail, or progress deviations exceed 12%: Level 1 warnings are pushed to the construction team, Level 2 warnings to the project manager and supervisor, and Level 3 warnings are simultaneously pushed to the construction unit and municipal management department.

[0207] Example 6

[0208] Automatic recognition of the progress of large industrial plant construction projects

[0209] 1. Data Acquisition Deployment

[0210] For a certain auto parts manufacturing plant project (building area of ​​80,000 m²) 2The facility includes the main steel structure, equipment foundations, and cleanroom. 22 industrial-grade fixed high-definition cameras (2560×1440 resolution, supporting wide dynamic range, suitable for high-intensity welding scenarios within the factory) are deployed at the column grid nodes (1 camera per 40m), the steel structure hoisting area (6 cameras), the equipment foundation construction area (8 cameras), and the cleanroom entrance (4 cameras). Five wheeled inspection robots (equipped with high-definition cameras and infrared thermal imagers) are configured to inspect the facility every 50 minutes along a pre-set route. Multi-rotor drones (equipped with zoom cameras) are used to conduct aerial photography of the steel structure roof and facade four times a week, focusing on collecting data on steel structure installation, roof waterproofing, and exterior wall cladding progress.

[0211] Data acquisition intervals: once every 15 minutes during the steel structure hoisting stage; once every 20 minutes during the equipment foundation pouring stage; once every hour during the cleanroom decoration stage; real-time acquisition mode is activated during welding operations (synchronously capturing welding temperature field data).

[0212] 2. Data Preprocessing

[0213] A bilateral filtering algorithm is used to remove image noise caused by welding fumes and metal dust in the factory. For images with strong welding light interference, an adaptive exposure correction algorithm is used to reduce highlight clipping. Perspective transformation is used to correct the tilted images of steel structure components taken by the aerial camera. Combined with camera calibration, the measurement error of steel component dimensions is ensured to be ≤2mm.

[0214] The annotation adopts the "AI pre-annotation + welder verification" mode: LabelStudio is used to automatically annotate targets such as steel structure components, welding nodes, production equipment, and templates, and the annotation information is associated with the component number in the BIM model (such as "steel column-Z36" and "equipment foundation-JC12"). Data augmentation focuses on simulating scenarios such as welding sparks, shadows in the factory, and metal reflections, and a training set of 8,500 annotated images is constructed. By flipping, scaling (0.65-1.35 times), and random cropping, 25,500 enhanced images are generated to improve the model's adaptability to industrial scenarios.

[0215] 3. Model Training and Inference

[0216] - Object counting task: Using the RetinaNet object detection model, with an input resolution of 800×800, a batch size of 24, 400 training iterations, and a confidence threshold of 0.55, the task counts the number of steel structure components (steel columns, steel beams, purlins), equipment foundation reinforcement, and formwork. The task outputs a daily report on the amount of steel structure materials arriving and the installation efficiency, with a counting accuracy of ≥95.2%.

[0217] - Status analysis task: The Mask R-CNN model (backbone network ResNet50) was used for 320 training iterations to classify the status of components as "not yet on site, hoisting in progress, connection and fixation, welding completed, anti-corrosion treatment, and acceptance". The temperature distribution of welding nodes was judged by combining infrared thermal imager data (to avoid cold welding). The status classification accuracy was ≥94.5%.

[0218] - Behavior understanding task: Using a temporal Transformer model, the input video clip is 22 frames long with a frame interval of 3 frames. The training iterations are 240 rounds. The task is to identify construction activities such as steel structure hoisting, welding, bolt tightening, and equipment installation. Welding operation quality is evaluated by combining welding temperature data, and illegal welding behaviors (such as failure to preheat according to specifications) are identified.

[0219] - Volume measurement task: Process drone aerial photos and ground camera images using SfM technology to generate dense point clouds (point cloud density 0.03m / point) of steel structure roof and equipment foundation. After registration with the Revit format factory BIM model, calculate the verticality deviation of steel structure installation, the volume of equipment foundation pouring and the flatness of cleanroom floor, and calculate the relative error ≤3.6%.

[0220] 4. Data Post-processing and Fusion

[0221] A fusion method combining Kalman filtering and DS evidence theory is employed to integrate multi-source data, including steel structure hoisting progress, welding quality inspection, and equipment foundation dimension measurement, thereby improving the robustness of the results. For critical components such as welded joints, a comprehensive confidence score is calculated by combining infrared data and visual recognition results (a confidence score ≥ 0.9 is considered acceptable). Based on the BIM model's "steel structure-equipment foundation-production area" association system, the fusion results are integrated with the factory's BIM model to dynamically update construction progress, component quality, and safety compliance data for each area.

[0222] 5. Progress visualization and reporting

[0223] Daily industrial plant progress reports are generated: Gantt charts are used to compare the planned and actual progress of steel structures, equipment foundations, and cleanrooms, with filtering by construction area; the installation status and welding quality level of steel structure components are displayed in color-coded format on the 3D plant BIM model, and clicking on welding nodes allows viewing temperature curves and inspection reports; charts are generated for key indicators such as steel structure welding pass rate, equipment foundation dimensional deviation, and cleanroom renovation progress. When the verticality deviation of the steel structure installation exceeds the specification (≥10mm), welding quality is substandard, or the progress deviation exceeds 8%, an automatic warning is sent to construction management personnel, supervisors, and the steel structure manufacturer to ensure timely rectification of problems and guarantee the accuracy of subsequent equipment installation in the plant.

[0224] The examples provided in this invention are not intended to limit the implementation. Those skilled in the art will recognize that various variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations, and any obvious variations or modifications derived therefrom are still within the scope of this invention.

Claims

1. A multi-modal computer vision based method for automatic progress recognition in engineering projects, characterized in that, The method comprises the following steps: Data acquisition step: collecting multi-source time-series image and video data of the construction site at preset time intervals through various image acquisition devices deployed on the construction site; Data preprocessing step: performing preprocessing operations including denoising, correction, labeling, and data enhancement on the collected multi-source time-series image and video data to form standardized training set and inference data set; Model inference step: using a pre-trained deep learning model to perform at least three analysis tasks in parallel on the preprocessed data: object counting task, state analysis task, behavior understanding task, and volume measurement task; Data post-processing and fusion step: post-processing the multi-dimensional recognition results output by the model inference step, including deduplication, correlation, and confidence calibration, and spatiotemporally aligning and fusing the processed results with the building information model and the planned progress data to generate unified and structured current progress information; Progress visualization and reporting step: automatically generating visual progress reports, three-dimensional progress comparison models, and progress deviation warnings based on the fused current progress information.

2. The method of claim 1, wherein, In the object counting task, YOLO, Faster R-CNN, or RetinaNet target detection models are used to identify and count precast components, reinforcement bundles, templates, and construction machinery, and the statistical results are used for material arrival and installation efficiency analysis.

3. The method of claim 1, wherein, In the state analysis task, DeepLab and Mask R-CNN segmentation models are used to classify component states, including not started, in installation, completed, and ready for acceptance, and the pouring and curing states of beams, plates, and columns are determined through pixel-level analysis.

4. The method of claim 1, wherein, The volume measurement task specifically includes: Generating dense point clouds from image sequences through motion recovery structure technology and multi-view stereo vision algorithms; Registering the point clouds with the design model; Automatically calculating earth excavation / backfilling volume and concrete pouring volume by calculating the volume change of point clouds in a specific area or the volume difference with the design model.

5. The method of claim 1, wherein, In the behavior understanding task, 3D CNN or time-series Transformer models are used to analyze video clips, identify construction activity types, and evaluate the intensity and efficiency of construction activities based on object counting results.

6. The method of claim 1, wherein, In the data post-processing and fusion step, Kalman filtering or Bayesian fusion methods are used to fuse recognition results from different models and different time points to improve data accuracy and robustness; and the fusion results are associated with components in the BIM model one by one to realize synchronous updating of the physical site and the digital model.

7. The method of claim 1, wherein, In the progress visualization and reporting step, the generated report includes at least: a planned progress and actual progress comparison chart in the form of a Gantt chart, a component completion status displayed in color coding on a three-dimensional BIM model, resource consumption and efficiency analysis charts generated based on volume measurement and behavior understanding results, and a warning notification automatically triggered when the identified progress deviation exceeds a threshold.

8. The method of claim 1, wherein, In the data preprocessing step, the denoising operation adopts Gaussian filtering or median filtering algorithm, the correction operation adopts perspective transformation to realize image distortion correction, and the data enhancement operation includes image flipping, scaling, color gamut transformation and random cropping.

9. The method of claim 1, wherein, In the model inference step, the parallel execution of multi-class analysis tasks is realized through a distributed computing framework, and the computing resources are dynamically allocated according to the computing complexity of different tasks to improve the inference efficiency.

Images