A system and method for automatically tracking construction problems across image sources

Abstract

The present invention discloses a system and method for automatically tracking construction problems across image sources, the method comprising: step 1: multi-source construction engineering real-scene image acquisition and transmission; step 2: construction engineering real-scene image preprocessing; step 3: intelligent identification of construction problems in real-scene images; step 4: three-dimensional spatial mapping and information aggregation of construction problems; step 5: automatic tracking of construction problems. The present invention can automatically identify and track problems at the construction site, and realizes information aggregation and unified management of different image sources, which not only improves the efficiency of discovering and handling construction problems, but also facilitates project managers to view, analyze and handle construction problems in a macroscopic way, reduces the work pressure of managers, and improves the level of intelligent supervision of project construction.

Description

Technical Field

[0001] The present invention relates to the technical field of building construction supervision, and in particular to a system and method for automatically tracking building construction problems across image sources. Background Art

[0002] At construction sites, video surveillance and real-time inspections have gradually become conventional means of supervision at construction sites, providing image data reflecting the real-time situation at the construction site, allowing managers to remotely monitor project progress and safety, and assist in management decision-making. However, there is a lack of a unified coordinate system between different image data sources, acquisition times, and acquisition points, resulting in image data that can only reflect the local situation of the project at a certain moment in history, and cannot reflect the distribution and rectification of construction problems in the project as a whole, and does not maximize the value of existing supervision methods. In addition, the construction problems contained in multi-source images are not effectively extracted and aggregated, making it difficult for project managers to track the occurrence and rectification of the same problem in multiple image data, which seriously limits the progress of problem handling and the efficiency of project management. Summary of the invention

[0003] The purpose of the present invention is to provide a system and method for automatically tracking construction problems across image sources, so as to solve the problem that it is difficult to track the development and solution process of the same problem at a construction site through different image data sources.

[0004] In order to solve the above technical problems, the present invention provides an automatic tracking system for construction problems across image sources, comprising:

[0005] Data storage module: used to store multi-source real-scene image data of the construction site collected by various image acquisition devices;

[0006] Data preprocessing module: used to preprocess the multi-source real-scene image data in the data storage module;

[0007] Problem identification module: used to automatically identify construction problems in multi-source real-scene image data pre-processed by the data pre-processing module using image processing and artificial intelligence technology;

[0008] Coordinate mapping and information aggregation module: used to map the construction problems intelligently identified by the problem identification module from the two-dimensional image space to the three-dimensional space of the project BIM model or real-scene model, and merge the same construction problems according to the three-dimensional coordinates and problem types, and aggregate and display the construction problems in all image sources;

[0009] Problem tracking module: Search and track the same construction problem aggregated by the coordinate mapping and information aggregation module across image sources, record the occurrence and handling process of the construction problem, and issue a problem handling work order based on project requirements;

[0010] Data display module: The user interface intuitively displays the project BIM model or real-life model, multi-source real-life image data of the construction site, and the location and tracking results of each construction problem by the problem tracking module, making it easier for project managers to view and handle problems in real time.

[0011] In order to solve the above technical problems, the present invention further provides a method for automatically tracking construction problems across image sources, using the above automatic tracking system for construction problems across image sources, comprising:

[0012] Step 1: Multi-source construction engineering real-scene image acquisition and transmission;

[0013] Step 2: Preprocessing of real-scene images of construction projects;

[0014] Step 3: Intelligent identification of construction problems in real-life images;

[0015] Step 4: 3D spatial mapping and information aggregation of construction issues;

[0016] Step 5: Automatically track construction issues.

[0017] Furthermore, in the method for automatic tracking of construction problems across image sources provided by the present invention, in step 1, the method for multi-source construction project real-scene image acquisition and transmission includes: acquiring multi-source real-scene image data of indoor and outdoor construction projects through a variety of image acquisition devices; and transmitting the multi-source real-scene image data to the data storage module of the automatic tracking system for construction problems across image sources.

[0018] Furthermore, the present invention provides a method for automatically tracking construction problems across image sources, and the multiple image acquisition devices include a pan-tilt video surveillance camera and an inspection video shooting device, and the inspection video shooting device is one or more of a drone, a panoramic camera, a smart phone, and a tablet computer.

[0019] Furthermore, in the method for automatically tracking construction problems across image sources provided by the present invention, in step 2, the method for preprocessing the real-scene image of the construction project includes:

[0020] Step 2.1: Filter out real-life images with low content richness; including:

[0021] Step 2.1.1: When installing the PTZ video surveillance camera, manually determine the optimal shooting angle range of the probe at the current deployment position, and record the limit range of the posture parameters of the PTZ video surveillance camera;

[0022] Step 2.1.2: Monitor the posture of the PTZ video surveillance camera. When there is a change, read the posture parameters of the current PTZ video surveillance camera and compare them with the posture parameter limit range of the best shooting angle interval described in step 2.1.1. If it exceeds the limit, the current monitoring angle is judged to be poor and abandoned, and the PTZ video surveillance camera is remotely controlled to rotate to the best shooting angle interval;

[0023] Step 2.1.3: For the real-scene images within the optimal shooting angle range and the real-scene images collected by the inspection video shooting equipment, the content richness of the images is determined by comprehensively using texture analysis and edge detection technology. If there are only a few similar texture types and only a few edges or no obvious edges in the image, the image is considered to have low content richness and is filtered out;

[0024] Step 2.2: Filter out overexposed, dark, blurred, or jittery real-life images;

[0025] Step 2.2.1: Calculate the brightness histogram of the real scene image. If the percentage of pixels in the extremely high or extremely low brightness range in the histogram exceeds the preset threshold p1, the image is judged to be overexposed or too dark and is filtered out;

[0026] Step 2.2.2: Use the Laplace operator or Sobel operator to calculate the variance of the real scene image. If the variance is less than the preset threshold p2, the image is determined to be blurred and filtered out;

[0027] Step 2.2.3: Perform Fourier transform on the real scene image and count the proportion of high-frequency components in the image spectrum. If the proportion is higher than the preset threshold p3, the image is determined to be jittery and filtered out.

[0028] Step 2.3: Filter out duplicate and redundant real-life images;

[0029] Perform pixel-level difference calculation on the previous and next frame images under the same shooting angle, and analyze the proportion of the changed area in the entire image. If the proportion is less than a given threshold p4, the previous and next frame images are determined to be similar or repeated images, and the next frame image is filtered out;

[0030] Step 2.4: Calculate the shooting resolution of the real scene image and classify the image scene type;

[0031] Step 2.4.1: Train a construction site real-life worker detection model using a public image dataset, where the model’s precision, recall, and detection accuracy are no less than 90%, 85%, and 90%, respectively;

[0032] Step 2.4.2: Use the worker detection model described in step 2.4.1 to identify the construction workers in the real scene image, and output the vertical pixel size of the worker detection frame;

[0033] Step 2.4.3: Compare the vertical pixel size of the worker detection frame with the average height of the construction workers to calculate the shooting resolution of the real scene image;

[0034] Step 2.4.4: Determine the scene type of the real scene image according to the preset threshold of the shooting resolution, where the scene types include distant view, medium view, close view and close-up images; the distant view image is used to identify the relatively macroscopic site layout and the overall progress of the project; the medium view image is used to identify large-scale engineering machinery and equipment and the progress of sub-projects; the close view image is used to identify small and medium-sized machinery and equipment and construction personnel; the close-up image is used to identify the wearing of safety helmets and safety ropes by construction personnel;

[0035] Furthermore, in the method for automatically tracking construction problems across image sources provided by the present invention, in step 3, the method for intelligently identifying construction problems in the real-scene image includes:

[0036] Step 3.1: Sort out common construction problems according to construction site management regulations, clarify the subject, object, geometric scale, spatial relationship and threshold involved in each construction problem, and build a construction problem database;

[0037] Step 3.2: Collect and manually annotate real-life images containing any subject and object elements in the construction problem database, and build an intelligent recognition model for production factors at the construction site, where the precision, recall rate, and detection accuracy of the intelligent recognition model for production factors at the construction site are not less than 90%, 85%, and 90%, respectively;

[0038] Step 3.3: Use the intelligent identification model of production factors on the construction site described in step 3.2 to process the real-scene image of the construction site input into the automatic tracking system of construction problems, and output the subject and object elements in the construction problem database and their two-dimensional image coordinates;

[0039] Step 3.4: Based on the entries in the construction problem database, determine whether there are any anomalies between the corresponding subject elements or subject and object elements. If there are any anomalies, mark the corresponding elements as construction problems.

[0040] Furthermore, in the method for automatically tracking construction problems across image sources provided by the present invention, in step 4, the method for three-dimensional spatial mapping and information aggregation of construction problems includes:

[0041] Step 4.1: 3D spatial mapping of construction problems;

[0042] For PTZ video surveillance cameras and drones:

[0043] Step 4.1.1: Use the checkerboard calibration method or the picture-model joint forward calibration method to obtain the pose external parameters of the PTZ video surveillance camera, or read the real-scene image metadata to directly obtain the pose external parameters of the drone;

[0044] Step 4.1.2: Create a virtual camera in the 3D space of the BIM model or real scene model of the construction site according to the pose external parameters of step 4.1.1;

[0045] Step 4.1.3: According to the two-dimensional image coordinates of the element corresponding to the construction problem in step 3.3, determine the light ray from the center of the virtual camera to the center of the target element;

[0046] Step 4.1.4: Calculate the coordinates of the collision point between the light ray and the BIM model or the solid model. The coordinates are the three-dimensional spatial coordinates of the construction problem.

[0047] For panoramic cameras:

[0048] Step 4.1.5: Bind the starting point and end point of the panoramic camera inspection route in the construction drawing;

[0049] Step 4.1.6: Calculate the panoramic inspection route based on the orb-slam algorithm and the start and end point constraints on the construction drawings;

[0050] Step 4.1.7: According to the construction drawings and inspection routes, determine the coordinates of the panoramic photo points with construction problems in the three-dimensional space of the BIM model or the real scene model as the three-dimensional space coordinates of the construction problems;

[0051] For live images captured with a smartphone or tablet:

[0052] Step 4.1.8: Directly read the metadata of the real-life image in the smartphone and tablet computer to obtain the coordinates of the points where the image was taken as the three-dimensional coordinates of the construction problem;

[0053] Step 4.2: Aggregation and deduplication of construction issues;

[0054] The construction problems in the real-life images from all sources are uniformly mapped to the project's three-dimensional space according to the shooting time. All construction problems within the preset time window t1 are traversed and the spatial distance between problems of the same type is calculated. If the distance is less than the specified distance threshold d1, it is regarded as the same problem in the multi-source images and merged.

[0055] Furthermore, in the method for automatically tracking construction problems across image sources provided by the present invention, in step 5, the method for automatically tracking construction problems includes:

[0056] Map a construction problem to the three-dimensional space of the project, and search whether there are unclosed problems of the same type within the specified radius d1 of the problem point;

[0057] If not, it indicates that the problem has occurred. The system marks the image shooting time as the time when the problem occurred, sends a problem ticket to the project manager and sets a time limit for problem handling. After the problem is handled, the manager manually closes the ticket and monitors whether the same type of problem occurs in the range within the specified time period t2. If so, it indicates that the historical problem ticket has not been actually handled. The system sends the ticket and warning to the manager again. If not, the problem ticket is closed at the system level.

[0058] If yes, it determines whether the problem has exceeded the aforementioned set processing time limit. If yes, it indicates that the historical problem work order has not been processed in time, and the system sends the work order and warning to the management personnel again. If no, it only records the construction problem and does not take any other action.

[0059] Compared with the prior art, the present invention has the following beneficial effects:

[0060] The system and method for automatic tracking of construction problems across image sources provided by the present invention can call on multiple image acquisition devices to obtain real-life images of the construction site, extract key features from real-life images from different sources and map them into a unified project BIM model or real-life model three-dimensional coordinate system, automatically track and analyze the same problem at the construction site, improve the efficiency of discovering and handling construction problems, and strengthen the level of intelligent supervision on the construction front line.

[0061] The automatic tracking system and method for construction problems across image sources provided by the present invention can automatically identify and track problems on the construction site, and realize information aggregation and unified management of different image sources, which not only improves the efficiency of discovering and handling construction problems, but also facilitates project managers to view, analyze and handle construction problems at a macro level, reduces the work pressure of managers, and improves the level of intelligent supervision of project construction. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 It is a system architecture diagram of the automatic tracking system of building construction problems across image sources;

[0063] Figure 2 is a flow chart of a method for automatically tracking building construction problems across image sources;

[0064] Figure 3 It is a flowchart of the process of the automatic tracking method of building construction problems across image sources;

[0065] Figure 4 It is a schematic diagram of the structural composition relationship of various image acquisition devices;

[0066] Figure 5 It is a schematic diagram of the structural composition of data transmission equipment. DETAILED DESCRIPTION

[0067] The present invention is described in detail below in conjunction with the accompanying drawings: The advantages and features of the present invention will become more apparent from the following description. It should be noted that the accompanying drawings are in very simplified form and in non-precise proportions, and are only used to conveniently and clearly assist in explaining the purpose of the embodiments of the present invention.

[0068] Please refer to Figure 1 , Figures 3 to 5 The embodiment of the present invention provides a cross-image source construction problem automatic tracking system 300, including a data storage module 310, a data preprocessing module 320, a problem identification module 330, a coordinate mapping and information aggregation module 340, a problem tracking module 350 and a data display module 360, wherein:

[0069] Data storage module 310: used to store multi-source real-scene image data of the construction site collected by various image acquisition devices 100. The multi-source real-scene image data is collected and formed according to a variety of different image acquisition devices 100. The various image acquisition devices 100 include but are not limited to pan-tilt video surveillance cameras 110 and patrol video shooting devices, and the patrol video shooting devices include but are not limited to drones 120, panoramic cameras 130, smart phones, tablet computers and other electronic devices with camera functions. Various image acquisition devices 100 can be transmitted to the data storage module 310 through data transmission devices 200. The data transmission devices 120 include but are not limited to mobile communication network modules such as 5G network cards 210, terminal computers 220, etc. For example, the real-scene image data collected by various image acquisition devices 100 can be imported into the terminal computer 220, and uploaded to the data storage module 310 by the terminal computer 220. The data transmission device 120 is not limited to an independent device, but can also be integrated on a variety of image acquisition devices 100, such as a mobile communication network module, a WIFI module and other wireless communication modules integrated on a tablet computer.

[0070] The acquisition and transmission methods of real-scene images include but are not limited to: setting up pan-tilt video surveillance cameras 110 such as video surveillance dome cameras with switchable viewing angles at the commanding heights of the project, such as the top of the tower crane arm or the light stand, and transmitting the monitoring images in real time to the data storage module 310 of the cross-image source automatic tracking system 300 for construction problems through the enterprise 5G private network; regularly operating a small drone 120 to cruise at low altitude to shoot real-scene images or videos of the project site, connecting the drone 120 to the terminal computer 220, and uploading the images containing metadata to the data storage module 310 of the cross-image source automatic tracking system 300 for construction problems through the terminal computer 220; when users inspect the front line of the project, they can use smart phones and tablets to shoot the real scene on the site and directly upload the image data containing metadata through the system 300; the panoramic camera 130 can be connected to the top of the selfie stick or combined with the safety helmet to walk at a constant speed indoors and outdoors along the project site to obtain a panoramic image of the construction site, and upload the images to the system 300 through the terminal computer 220. The system is the abbreviation of the cross-image source automatic tracking system 300 for construction problems.

[0071] Data preprocessing module 320: used to preprocess the multi-source real scene image data in the data storage module 310. The preprocessing includes but is not limited to filtering out low content richness images and repeated redundant images, exposure correction, noise removal, scene classification, etc.

[0072] Specifically, the data preprocessing module 320 is mainly used to preprocess the original multi-source real-scene image data input into the system 300 by the data storage module 310, so as to filter out meaningless or redundant images, improve the overall quality of the image, ensure the accuracy of the system 300 in automatically tracking construction problems, and reduce the processing load of the system 300. Specific real-scene image preprocessing methods include but are not limited to: comprehensively using texture analysis and edge detection technology to judge and filter out low-content richness images, and for video surveillance, posture parameters can be used to assist in judgment; respectively calculating the image brightness histogram, variance and spectrum distribution, comparing the main parameter indicators with the preset thresholds to judge and filter out overexposed, dark, blurred and jittery real-scene images; calculating and judging and filtering out similar and repeated images based on the pixel level difference between the previous and next frame images; classifying the image scenes with reference to obvious elements such as construction personnel in the image, and determining the construction element recognition tasks of each image, etc.

[0073] Problem identification module 330: used to automatically identify construction problems in the multi-source real-scene image data preprocessed by the data preprocessing module 320 using image processing and artificial intelligence technology. Construction problems include but are not limited to workers not wearing safety helmets, personnel illegally intruding into the working range of construction machinery, and inadequate protection of edges and openings.

[0074] Specifically, the problem identification module 330 is mainly used to automatically identify construction problems in multi-source real-life images as source data for automatic tracking of construction problems. The module has a built-in construction problem library, a construction element intelligent identification engine, and a construction problem rule engine. The construction problem library records common construction problems on the construction site, such as workers not wearing safety helmets, personnel illegally invading the operating range of construction machinery, etc., and is expressed in structured fields such as subject, object, geometric scale, spatial relationship, and threshold, which can be directly used by the construction problem rule engine to read and understand the judgment rules of various construction problems; the construction element intelligent identification engine is developed based on image processing and artificial intelligence algorithms, and can autonomously identify various construction elements in the image, such as construction workers, construction machinery, protective equipment, edge openings, etc., which can be used as subject and object objects for the construction problem rule engine to call; the construction problem rule engine reads the construction problem library, clarifies the problem object and judgment logic, and then calls the construction element object identified by the construction element intelligent identification engine for logical rule matching to determine whether a construction problem has occurred.

[0075] Coordinate mapping and information aggregation module 340: used to map the construction problems intelligently identified by the problem identification module 330 from the two-dimensional image space to the three-dimensional space of the project BIM model or the real-life model, and merge the same construction problems according to the three-dimensional coordinates and problem types, and aggregate and display the construction problems in all image sources.

[0076] Specifically, the coordinate mapping and information aggregation module 340 is mainly used to map the identified construction problems from the scattered two-dimensional image space to the unified three-dimensional space of the project, and merge the same construction problems in different data sources based on the three-dimensional coordinates and problem categories, and aggregate and display all construction problems in all image sources. The core of this module is the coordinate mapping function, and the core functions are: for the PTZ video surveillance camera 110, the camera pose external parameters are obtained through the chessboard calibration method or the monitoring screen-model joint forward calibration method, and a virtual camera with the same pose is constructed in the model three-dimensional space to simulate and calculate the three-dimensional coordinates of the construction problem; for the drone 120, smart phones and other inspection video shooting equipment, the real-life image metadata is directly read to obtain the camera pose external parameters, and then a virtual camera is created to calculate the coordinates of the construction problem; for the panoramic inspection camera, the inspection route is calculated according to the sequence panoramic image and the start and end point coordinates and matched with the drawings or models, the image shooting position containing the construction problem is determined, and the construction problem is mapped to the three-dimensional model.

[0077] The problem tracking module 350 searches and tracks the same construction problem of the coordinate mapping and information aggregation module 340 across image sources, records the occurrence and handling process of the construction problem, and issues a problem handling work order in accordance with project requirements, that is, tracks the construction problem across image sources.

[0078] Specifically, the problem tracking module 350 is mainly used to monitor construction problems in three-dimensional space, search and track the same problem across image sources and timelines, and dispatch construction problem processing work orders in combination with project requirements. The specific tracking logic is that when a construction problem is mapped and aggregated into three-dimensional space, it is immediately searched for unprocessed problems of the same type within the specified radius of the problem point to determine whether the problem is in an occurrence state; for construction problems in an occurrence state, problem work orders are pushed to management personnel and a processing time limit is set, and the problem processing status and work order completion status are regularly checked; for construction problems that are being rectified, it is checked whether the predetermined processing time limit has been exceeded, and a detailed process record is formed.

[0079] Data display module 360: The user interface intuitively displays the project BIM model or real-life model, the real-life image data of the construction site, and the location and tracking results of each construction problem by the problem tracking module 350, so that project managers can view and handle problems in real time.

[0080] Specifically, the data display module 360 ​​is used to intuitively display the project 3D model and construction problem distribution on the user interface, and also supports attaching real-life image data of the project to enhance the visual performance. Users click on a specific construction problem to view the historical details of the problem, such as problem description, occurrence time, dispatch and processing status, and related images, which is convenient for project managers to view and handle problems in real time, and also provides data support for the preventive management of the project.

[0081] Please refer to Figures 2 to 5 This embodiment also provides a method for automatically tracking construction problems across image sources, which is based on the above system 300 and includes:

[0082] Step 1: Multi-source construction engineering real-scene image acquisition and transmission.

[0083] A variety of image acquisition devices 100 such as a pan-tilt video surveillance camera 110, a drone 120, a panoramic camera 130, and a smart phone are used together to obtain real-life image data of indoor and outdoor construction projects, and the data can be remotely transmitted to the cross-image source construction problem automatic tracking system 300 as basic material input through a 5G network card 210, a terminal computer 220, etc. Among them, the pan-tilt video surveillance camera 110 and the drone 120 and other inspection video shooting devices can be collectively referred to as cameras.

[0084] For example, a video surveillance camera is installed on the tower crane arm of a certain project, and the real-time monitoring video data is transmitted back to the data storage server of the automatic tracking system 300 for construction problem through the enterprise 5G private network for analysis and processing; at the same time, a panoramic inspection single-soldier equipment for the construction site is designed based on the panoramic camera 130, and safety specialists are regularly arranged to carry the equipment and walk at a constant speed indoors and outdoors on site to collect panoramic images, and upload the data to the system 300 through the terminal computer 220.

[0085] Step 2: Preprocessing of real-scene images of construction projects.

[0086] Step 2.1: Filter out real-life images with low content richness. Specifically include:

[0087] Step 2.1.1: When installing the PTZ video surveillance camera 110, manually determine the optimal shooting angle range of the probe at the current deployment position through experiments, and record the limit ranges of the posture parameters such as the heading angle, pitch angle and focal length of the PTZ video surveillance camera 110.

[0088] For example, a pan-tilt video surveillance camera 110 such as a video surveillance dome camera is installed on the tower crane arm of the project, and the pan-tilt video surveillance camera 110 is manually controlled to rotate and adjust the focal length, and the optimal shooting angle range of the current camera is determined, and the limit range of the camera's heading angle Pan, pitch angle Tilt, focal length Zoom and other posture parameters is recorded [Pan1, Pan2, Tilt1, Tilt2, Zoom1, Zoom2] = [24, 292, 0, 85, 1, 9]. Among them, the optimal shooting angle range is defined as the monitoring screen containing a clearly visible construction site as a whole or in part, without any extreme screens such as the camera installation background, the sky or the ground, and the details of a certain production factor on site.

[0089] Step 2.1.2: Monitor the posture of the PTZ video surveillance camera 110. When a change occurs, read the posture parameters of the current PTZ video surveillance camera 110 and compare them with the posture parameter limit range of the best shooting angle interval described in step 2.1.1. If the limit is exceeded, the current monitoring angle is determined to be poor and discarded, and the PTZ video surveillance camera 110 (i.e., the camera) is remotely controlled to rotate to the best shooting angle interval.

[0090] For example, monitor the posture of the pan-tilt video surveillance camera 110. When any posture changes and finally stops, read the posture parameters of the current pan-tilt (i.e., heading angle, pitch angle, and focal length), and compare them with the posture parameter limits described in step 2.1.1. If any one or more parameters exceed the corresponding limits, it is determined that the current monitoring viewing angle is not good and the real-scene image data under this viewing angle is discarded. At the same time, remotely control the camera to rotate to the best shooting viewing angle range or any preset patrol point. That is, if the posture parameters after the pan-tilt adjustment stops are Pan=105, Tilt=43, Zoom=12, and the Zoom parameter exceeds the limit [1, 9], it is determined that the current viewing angle is not good and the real-scene image under this viewing angle is discarded.

[0091] Step 2.1.3: For the real-scene images within the optimal shooting angle range and the real-scene images collected by the inspection video shooting equipment, the content richness of the picture is determined by a combination of texture analysis and edge detection technology. If there are only a few similar texture types in the picture and only a few edges or no obvious edges, the image is considered to have low content richness and is filtered out. For images collected by equipment such as drones 120 and panoramic cameras 130, the content richness of the image can be determined by this method, and images with low content richness can be filtered out.

[0092] For example, for real-life images within the optimal shooting angle range, texture analysis and edge detection techniques are used to determine the content richness of the image. If less than 3 texture types and less than 5 straight edges can be extracted from the image, the image is considered to have low content richness and is filtered out to avoid the image focusing on certain production factors and causing the image to have no practical significance. For images acquired by acquisition devices such as drones 120 and panoramic cameras 130 that cannot transmit data back to system 300 in real time, system 300 can only use the method in this step to determine the content richness of the image and filter out images with low content richness.

[0093] Step 2.2: Filter out overexposed, dark, blurred, or jittery real-life images. Specifically include:

[0094] Step 2.2.1: Calculate the brightness histogram of the real scene image. If the proportion of pixels in the extremely high or extremely low brightness range in the histogram exceeds the preset threshold p1, the image is judged to be overexposed or too dark and is filtered out.

[0095] For example, the brightness histogram of a real-life image is calculated. If the percentage of pixels with brightness less than 50 or greater than 230 in the histogram exceeds the preset threshold p1=30%, the image is judged to be too dark or overexposed and is filtered out. That is, if the percentage of pixels with brightness less than 50 and greater than 230 in the brightness histogram of an image is 70% and 8% respectively, the image is judged to be too dark and is filtered out.

[0096] Step 2.2.2: Use the Laplace operator or the Sobel operator to calculate the variance of the real scene image. If the variance is less than the preset threshold p2, the image is determined to be blurred and filtered out.

[0097] For example, the variance of a real-life image is calculated using the Laplace operator. If the variance is less than the preset threshold p2=100, the image is considered blurry and is filtered out. That is, if the image variance of an image calculated based on the Laplace operator is 83.17, which is less than the threshold 100, the image is considered blurry and is filtered out.

[0098] Step 2.2.3: Perform Fourier transform on the real scene image and count the proportion of high-frequency components in the image spectrum. If the proportion is higher than the preset threshold p3, the image is determined to be jittery and filtered out.

[0099] For example, a real-life image is subjected to a two-dimensional Fourier transform, the image grayscale image is converted into a spectrum image, and the proportion of high-frequency components in the spectrum is counted. If the proportion is higher than the preset threshold value p3, the image is determined to be jittery and filtered out. That is, if the proportion of high-frequency components with a frequency higher than 300Hz in the spectrum image is 82%, which is greater than the given threshold value of 70%, the image is determined to be jittery and filtered out.

[0100] Step 2.3: Filter out duplicate and redundant real-life images.

[0101] The pixel-level difference calculation is performed on the previous and next frame images under the same shooting angle, and the proportion of the changed area in the entire image is analyzed. If the proportion is less than the given threshold p4, the previous and next frame images are judged to be similar or repeated images, and the next frame image is filtered out.

[0102] For example, the pixel-level difference calculation is performed on the two frames of images captured by the PTZ video surveillance camera 110 at the same viewing angle, and the area with a difference greater than the difference threshold q1 is recorded as the changed area; the total number of pixels in the changed area is counted, and its proportion to the total number of pixels in the entire image is calculated. If the proportion is less than a given threshold p4, the previous and next frames are determined to be similar or repeated images, and the next frame is filtered out, otherwise it is retained. That is, if the difference between the previous and next frames of images under the same monitoring probe is greater than 30, the proportion of the changed area in the whole image is 3.7%, which is less than the given threshold 5%, then the previous and next frames are determined to be similar and the next frame is filtered out.

[0103] Step 2.4: Calculate the shooting resolution of the real scene image and classify the image scene type.

[0104] Step 2.4.1: Use the public image dataset to train a construction site real-life worker detection model, where the model's precision, recall, and detection accuracy are no less than 90%, 85%, and 90%, respectively.

[0105] For example, by using large public image datasets such as Pascal VOC, MS COCO, MOCS, SODA, etc. that contain images of people as datasets, and using the deep learning target detection framework YOLO v8 as the basic model, we iteratively trained a real-life image worker detection model for construction sites, until the model's precision, recall rate, and detection accuracy reached 90%, 85%, and 90%, respectively.

[0106] Step 2.4.2: Use the worker detection model described in step 2.4.1 to identify the construction workers in the real-life image and output the vertical pixel size of the worker detection frame.

[0107] For example, use the worker detection model described in step 2.4.1 to identify construction workers in real-life images and output the vertical pixel size of the worker detection frame. That is, if a real-life image contains a worker working on a construction site, the model should correctly identify the worker and output the vertical pixel size of the worker target detection frame as 140 pixels.

[0108] Step 2.4.3: Compare the vertical pixel size of the worker detection frame with the average height of the construction workers to calculate the shooting resolution of the real scene image.

[0109] For example, based on the average height of adult males in my country and the average thickness of construction safety shoes, assuming that the actual height of construction workers is 175 cm, the vertical pixel size of the worker detection frame is compared with the average height of construction workers to calculate the shooting resolution of the real scene image. That is, for the real scene image described in step 2.4.2, its shooting resolution can be calculated to be 0.8 pixel / cm.

[0110] Step 2.4.4: Determine the scene type of the real scene image according to the preset threshold of the shooting resolution, where the scene types include distant view, medium view, close view and close-up images; the specific classification thresholds are shown in Table 1, where distant view images can be used to identify relatively macro site layout and overall project progress; medium view images can be used to identify large-scale engineering machinery and equipment, and the progress of sub-projects; close view images can be used to identify small and medium-sized machinery and equipment, construction personnel, etc., and close-up images can be used to identify detailed features such as the wearing of construction workers' safety helmets and safety ropes.

[0111] For example, for the real-scene image described in step 2.4.2, its shooting resolution is 0.87 pixel / cm, so referring to Table 1, this image is a close-up, suitable for the detection of construction elements such as small and medium-sized machinery and equipment, construction personnel, etc. During implementation, construction elements with known actual sizes can be selected as reference objects for calculating the image shooting resolution according to the specific conditions of the project, such as concrete mixer trucks, excavators, etc.

[0112]

[0113] Step 3: Intelligent identification of construction problems in real-life images. Specifically includes:

[0114] Step 3.1: Sort out common construction problems according to the construction site management regulations, clarify the subjects, objects (if any), geometric scales, spatial relationships and thresholds involved in each construction problem, and build a construction problem database.

[0115] For example, according to the construction site management regulations, common construction problems are sorted out, such as "it is prohibited to park and operate automobile cranes above the underground roof of the construction site", "no up and down passages are set up for trenches longer than 2m, and no red sign lights are set up at dangerous places", "construction workers illegally enter the operation area of ​​automobile cranes", etc., and the subjects, objects (if any), geometric scales, spatial relationships and thresholds involved in each construction problem are clarified, and a construction problem database is constructed, as shown in Table 2.

[0116]

[0117] Step 3.2: Collect and manually annotate real-life images containing any subject and object elements in the construction problem database, and build an intelligent recognition model for construction site production factors. The precision, recall rate, and detection accuracy of the intelligent recognition model for construction site production factors are not less than 90%, 85%, and 90%, respectively.

[0118] For example, real-life images containing any subject and object elements (such as truck cranes, trenches, etc.) in the construction problem library are collected and manually annotated to form the construction site production factor dataset ConSite. The deep learning target detection framework YOLO v8 is used as the basic model for iterative training to build an intelligent recognition model for construction site production factors. The model's precision, recall rate and detection accuracy are not less than 90%, 85% and 90% respectively.

[0119] Step 3.3: Use the intelligent identification model of production factors on the construction site described in step 3.2 to process the real-scene image of the construction site input into the automatic construction problem tracking system 300, and output the subject and object elements in the construction problem library and their two-dimensional image coordinates.

[0120] For example, the intelligent recognition model described in step 3.2 is used to process the real-life image of the construction site input into the automatic construction problem tracking system 300, and output the subject and object elements in the construction problem library and their two-dimensional image coordinates; that is, if an image captured by the pan-tilt video surveillance camera 110 is a hoisting operation process, in which two truck cranes are performing hoisting operations, and there are several construction workers assisting or supervising the operation, and two of the workers illegally enter the hoisting operation range; the intelligent recognition model processes the image and outputs the two-dimensional image coordinates of the truck crane and the construction workers respectively:

[0121]

[0122] Among them, the first column of the output result is the type of production factor (0 refers to a truck crane, 1 refers to a construction worker), the second and third columns are the ratios of the center coordinates of the detection frame to the length and width of the image, and the fourth and fifth columns are the ratios of the length and width of the detection frame to the length and width of the image.

[0123] Step 3.4: Based on the entries in the construction problem database, determine whether there are any anomalies between the corresponding subject elements or subject and object elements. If there are any anomalies, mark the corresponding elements as construction problems.

[0124] For example: according to the entries in the construction problem library, determine whether there is an anomaly between the corresponding subject element or the subject and object elements. If there is an anomaly, mark the corresponding element as a construction problem. That is, for the image described in step 3.3, two workers illegally entered the lifting operation range of a truck crane, which meets the subject and object objects and spatial relationships described in the construction problem library. Therefore, the subject and object elements corresponding to the combined mark are construction problems, and the center coordinates of the combined mark rectangular box (325, 463) are defined as the two-dimensional image coordinates of the construction problem.

[0125] Step 4: Three-dimensional spatial mapping and information aggregation of construction issues.

[0126] Step 4.1: 3D spatial mapping of the construction problem.

[0127] For the PTZ video surveillance camera 110 or the drone 120, the specific processing method is as follows:

[0128] Step 4.1.1: Use a checkerboard calibration method or a picture-model joint forward calibration method to obtain the camera's pose external parameters, or read the real-scene image metadata to directly obtain the drone's 120 pose external parameters.

[0129] For example, since the selected PTZ video surveillance camera 110 is attached to the tower crane arm, it is extremely inconvenient to directly use Zhang Zhengyou's chessboard calibration method to determine the camera's posture external parameters, so the screen-model joint forward calibration method is used: first, the X and Y coordinates of the camera are determined on the orthogonal top view of the three-dimensional site layout model of the construction site; then switch to the monitoring screen perspective, adjust the horizon angle when the camera PTZ heading angle is 0, 90°, 180° and 270° respectively, and determine the camera's A X and A Y Parameters; then link the comparison view and overlay view of the video monitoring screen and the model screen, adjust A Z The X and Y parameters are used to make the real scene and the model scene completely consistent, thus completing the camera external parameter calibration. X , A Y Used to correct the torsion caused by the non-rigid connection between the camera and the crane, A Z Indicates the angle between the camera shooting direction and the east direction when the camera yaw angle is 0.

[0130] Step 4.1.2: Create a virtual camera in the three-dimensional space of the BIM model or real scene model of the construction site based on the pose external parameters of step 4.1.1.

[0131] For example, according to the camera pose external parameters described in step 4.1.1, a virtual camera is created in the three-dimensional space of the BIM model of the construction site, and the pose external parameters of the virtual camera are consistent with those of the actual camera.

[0132] Step 4.1.3: According to the two-dimensional image coordinates of the element corresponding to the construction problem in step 3.3, determine the light ray from the center of the virtual camera to the center of the target element.

[0133] For example, according to the two-dimensional image coordinates of the construction problem in step 3.3, determine the light ray from the center of the virtual camera (ie, the virtual imaging center) to the center of the construction problem.

[0134] Step 4.1.4: Calculate the coordinates of the collision point between the light ray and the BIM model or the solid model. The coordinates are the three-dimensional space coordinates of the construction problem.

[0135] For example, calculate the coordinates of the collision point between the light ray described in step 4.1.4 and the collision block in the BIM model. The coordinates are the three-dimensional space coordinates of the construction problem. For the construction problem described in step 3.4, the three-dimensional space coordinates obtained after collision calculation are (-2225.873m, 19096.392m, 3.984m).

[0136] For the panoramic camera 130, the specific processing method includes:

[0137] Step 4.1.5: Bind the starting point and the end point of the inspection route of the panoramic camera 130 in the construction drawing;

[0138] For example, when a project manager uses a panoramic camera 130 for inspection, the starting and ending points of the inspection route are marked in the construction drawings to constrain the subsequent automatically generated panoramic route; that is, if a project manager starts inspection at Gate 7 West of Hall A and stops at Gate 2 South of Hall A, the starting and ending points must be marked at the corresponding positions on the drawings.

[0139] Step 4.1.6: Calculate the panoramic inspection route based on the orb-slam algorithm and the start and end point constraints on the construction drawings;

[0140] For example, after the inspection is completed, the inspection sequence photos are uploaded to the system 300, and the orb-slam algorithm is called to calculate the panoramic inspection route based on the sequence panoramic photos and the corresponding posture information. The reasonable route is matched on the construction drawing according to the start and end point constraints described in step 4.1.1, and finally the actual route is generated and bound to the construction drawing and BIM model.

[0141] Step 4.1.7: According to the construction drawings and inspection routes, determine the coordinates of the panoramic photo points with construction problems in the three-dimensional space of the BIM model or real-life model as the three-dimensional space coordinates of the construction problems.

[0142] For example, the intelligent recognition model detects a construction problem in the 84th panoramic image of this inspection. The system 300 can automatically determine the shooting point coordinates of the 84th frame of the image based on the panoramic route and the BIM model, and record the point as the three-dimensional coordinates of the construction problem.

[0143] For real-life images captured by smartphones or tablets, specific processing methods include:

[0144] Step 4.1.8: Directly read the metadata of the real-life image in the smartphone and tablet to obtain the image

[0145] The coordinates of the photographed points are used as the three-dimensional coordinates of the construction problem.

[0146] Step 4.2: Aggregation and deduplication of construction issues.

[0147] The construction problems in the real-life images from all sources are uniformly mapped to the project's three-dimensional space according to the shooting time. All construction problems within the preset time window t1 are traversed and the spatial distance between problems of the same type is calculated. If the distance is less than the specified distance threshold d1, it is regarded as the same problem in the multi-source images and merged.

[0148] For example, the construction problems in the real-life images from all sources are mapped to the project three-dimensional space according to the shooting time, all construction problems are traversed according to the preset time window of 10 minutes, and the spatial distance between all the same type of problems in each time window is calculated. If it is less than the specified distance threshold of 5m, it is regarded as the same problem in the multi-source image, and the construction problem information detected earlier is retained and the rest of the same construction problem information in the time window is discarded. For the PTZ video surveillance camera 110 described in step 4.1, it has captured the problem of construction personnel illegally invading the operating range of the mobile crane several times before and after the inspection, and another PTZ video surveillance camera 110 has also captured the problem several times. When the system 300 maps and aggregates the construction problems in the multi-source data to the project three-dimensional space, it calculates whether there are the same type of construction problems in the 10-minute window after each construction problem in chronological order. If there are, it calculates whether the spatial distance between the construction problems is less than 5m. If so, the same construction problems in the time window are merged, and the time window is switched backward to merge the same construction problems in sequence until any construction problem of the same type in the area is unique and the recording time interval is longer than 10 minutes.

[0149] Step 5: Automatically track construction issues.

[0150] A construction problem is mapped to the three-dimensional space of the project, and a search is performed to see if there are any unclosed problems of the same type within the specified radius d1 of the problem point.

[0151] If not, it indicates that the problem has occurred. The system 300 marks the image shooting time as the time when the problem occurred, sends a problem ticket to the project manager and sets a time limit for problem processing. After the problem is processed, the manager manually closes the ticket and monitors whether the same type of problem occurs in the range within the specified time period t2. If so, it indicates that the historical problem ticket has not been actually processed. The system 300 sends the ticket and warning to the manager again. If not, the system 300 closes the problem ticket at the level of the system 300.

[0152] If yes, it is determined whether the problem has exceeded the aforementioned set processing time limit. If yes, it indicates that the historical problem work order has not been processed in time, and the system 300 sends the work order and warning to the management personnel again. If no, it only records the construction problem and does not take any other action.

[0153] For example, after mapping the problem of construction workers illegally invading the working range of the mobile crane described in step 4 to the three-dimensional space of the project, search whether there are unclosed problems of the same type within a specified radius of 5m of the problem point;

[0154] If not, it means that the problem has occurred for the first time. The automatic tracking system 300 for construction problems will mark the time of recording the image shooting of the problem as the time of the problem occurrence, and send a construction problem work order to the project manager. The work order includes the details of the construction problem, the screenshot of the problem, the time of occurrence, the problem point and the problem handling time limit. The project manager arranges a full-time staff to deal with the problem and then manually closes the work order. After the system 300 detects that the problem work order is manually closed, it will monitor whether there is a problem of construction personnel illegally invading the operating range of the mobile crane within the specified time period of 1 day. If not, the problem work order will be closed at the system 300 level, and the problem has been actually handled; if so, it means that the problem has not been actually handled. The system 300 will reopen the problem work order and push the warning message to the project manager again until the problem is actually solved.

[0155] If yes, it means that the problem has occurred and is being rectified. At this time, it is determined whether the current time point has exceeded the problem processing time limit set in the aforementioned work order. If yes, it means that the historical problem work order has not been processed in time. The system 300 will mark the work order as overdue and push the work order to the project manager again; if no, it means that the problem is still being rectified. Only a screenshot of the current problem is recorded in the work order, and no other message push or other processing is performed.

[0156] The system and method for automatic tracking of construction problems across image sources provided by the above-mentioned embodiments of the present invention can call on multiple image acquisition devices to obtain real-life images of the construction site, extract key features from real-life images from different sources and map them into a unified project BIM model or real-life model three-dimensional coordinate system, automatically track and analyze the same problem at the construction site, improve the efficiency of discovering and handling construction problems, and strengthen the level of intelligent supervision on the construction front line.

[0157] The system and method for automatic tracking of construction problems across image sources provided by the above-mentioned embodiments of the present invention can automatically identify and track problems at the construction site, and realize information aggregation and unified management of different image sources, which not only improves the efficiency of discovering and handling construction problems, but also facilitates project managers to view, analyze and handle construction problems at a macro level, reduces the work pressure of managers, and improves the level of intelligent supervision of project construction.

[0158] The present invention is not limited to the above-mentioned specific implementation modes. Obviously, the above-mentioned embodiments are only some embodiments of the embodiments of the present invention, but not all embodiments. Based on the described embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present invention. Those skilled in the art can make other levels of modifications and changes to the present invention. In this way, if these modifications and changes of the present invention fall within the scope of the claims of the present invention, the present invention is also intended to include these changes and changes.

Claims

1. An automatic tracking system for construction problems across image sources, characterized in that: include: Data storage module: used to store multi-source real-scene image data of the construction site collected by various image acquisition devices; Data preprocessing module: used to preprocess the multi-source real-scene image data in the data storage module; Problem identification module: used to automatically identify construction problems in multi-source real-scene image data pre-processed by the data pre-processing module using image processing and artificial intelligence technology; Coordinate mapping and information aggregation module: used to map the construction problems intelligently identified by the problem identification module from the two-dimensional image space to the three-dimensional space of the project BIM model or real-scene model, and merge the same construction problems according to the three-dimensional coordinates and problem types, and aggregate and display the construction problems in all image sources; Problem tracking module: Search and track the same construction problem aggregated by the coordinate mapping and information aggregation module across image sources, record the occurrence and handling process of the construction problem, and issue a problem handling work order based on project requirements; Data display module: The user interface intuitively displays the project BIM model or real-life model, multi-source real-life image data of the construction site, and the location and tracking results of each construction problem by the problem tracking module, making it easier for project managers to view and handle problems in real time.

2. A method for automatically tracking construction problems across image sources, characterized in that: The automatic tracking system for construction problems across image sources according to claim 1 comprises: Step 1: Multi-source construction engineering real-scene image acquisition and transmission; Step 2: Preprocessing of real-scene images of construction projects; Step 3: Intelligent identification of construction problems in real-life images; Step 4: 3D spatial mapping and information aggregation of construction issues; Step 5: Automatically track construction issues.

3. The method for automatically tracking construction problems across image sources according to claim 2, characterized in that: In step 1, the method for collecting and transmitting multi-source real-scene images of construction projects includes: obtaining multi-source real-scene image data of indoor and outdoor construction projects through multiple image acquisition devices; and transmitting the multi-source real-scene image data to the data storage module of the automatic tracking system for construction problems across image sources.

4. The method for automatically tracking construction problems across image sources according to claim 3 is characterized in that: The various image acquisition devices include pan-tilt video surveillance cameras and patrol video shooting devices, and the patrol video shooting device is one or more of a drone, a panoramic camera, a smart phone, and a tablet computer.

5. The method for automatically tracking construction problems across image sources according to claim 4 is characterized in that: In step 2, the method for preprocessing the real-scene image of the construction project includes: Step 2.1: Filter out real-life images with low content richness; including: Step 2.1.1: When installing the PTZ video surveillance camera, manually determine the optimal shooting angle range of the probe at the current deployment position, and record the limit range of the posture parameters of the PTZ video surveillance camera; Step 2.1.2: Monitor the posture of the PTZ video surveillance camera. When there is a change, read the posture parameters of the current PTZ video surveillance camera and compare them with the posture parameter limit range of the best shooting angle interval described in step 2.1.

1. If it exceeds the limit, the current monitoring angle is judged to be poor and abandoned, and the PTZ video surveillance camera is remotely controlled to rotate to the best shooting angle interval; Step 2.1.3: For the real-scene images within the optimal shooting angle range and the real-scene images collected by the inspection video shooting equipment, the content richness of the images is determined by comprehensively using texture analysis and edge detection technology. If there are only a few similar texture types and only a few edges or no obvious edges in the image, the image is considered to have low content richness and is filtered out; Step 2.2: Filter out overexposed, dark, blurred, or jittery real-life images; Step 2.2.1: Calculate the brightness histogram of the real scene image. If the proportion of pixels in the extremely high or extremely low brightness range in the histogram exceeds the preset threshold p1, the image is judged to be overexposed or too dark and is filtered out; Step 2.2.2: Use the Laplace operator or Sobel operator to calculate the variance of the real scene image. If the variance is less than the preset threshold p2, the image is determined to be blurred and filtered out; Step 2.2.3: Perform Fourier transform on the real scene image, and count the proportion of high-frequency components in the image spectrum. If the proportion is higher than the preset threshold p3, the image is determined to be jittery and filtered out; Step 2.3: Filter out duplicate and redundant real-life images; Perform pixel-level difference calculation on the previous and next frames of the same shooting angle, and analyze the proportion of the changed area in the entire image. If the proportion is less than a given threshold p4, the previous and next frames are determined to be similar or repeated images, and the next frame is filtered out; Step 2.4: Calculate the shooting resolution of the real scene image and classify the image scene type; Step 2.4.1: Train a construction site real-life worker detection model using a public image dataset, where the model’s precision, recall, and detection accuracy are no less than 90%, 85%, and 90%, respectively; Step 2.4.2: Use the worker detection model described in step 2.4.1 to identify the construction workers in the real scene image, and output the vertical pixel size of the worker detection frame; Step 2.4.3: Compare the vertical pixel size of the worker detection frame with the average height of the construction workers to calculate the shooting resolution of the real scene image; Step 2.4.4: Determine the scene type of the real scene image based on the preset threshold of the shooting resolution, where the scene types include long shot, medium shot, close shot and close-up images; the long shot image is used to identify the relatively macro site layout and the overall progress of the project; the medium shot image is used to identify large-scale engineering machinery and equipment, and the progress of sub-projects; the close shot image is used to identify small and medium-sized machinery and equipment, and construction personnel; the close-up image is used to identify the construction personnel's helmets and safety ropes.

6. The method for automatically tracking construction problems across image sources according to claim 5, characterized in that: In step 3, the method for intelligently identifying construction problems in the real-scene image includes: Step 3.1: Sort out common construction problems according to construction site management regulations, clarify the subject, object, geometric scale, spatial relationship and threshold involved in each construction problem, and build a construction problem database; Step 3.2: Collect and manually annotate real-life images containing any subject and object elements in the construction problem database, and build an intelligent recognition model for production factors at the construction site, where the precision, recall rate, and detection accuracy of the intelligent recognition model for production factors at the construction site are not less than 90%, 85%, and 90%, respectively; Step 3.3: Use the intelligent identification model of production factors on the construction site described in step 3.2 to process the real-scene image of the construction site input into the automatic tracking system of construction problems, and output the subject and object elements in the construction problem database and their two-dimensional image coordinates; Step 3.4: Based on the entries in the construction problem database, determine whether there are any anomalies between the corresponding subject elements or subject and object elements. If there are any anomalies, mark the corresponding elements as construction problems.

7. The method for automatically tracking construction problems across image sources according to claim 6, characterized in that: In step 4, the method of three-dimensional spatial mapping and information aggregation of construction problems includes: Step 4.1: 3D spatial mapping of construction problems; For PTZ video surveillance cameras and drones: Step 4.1.1: Use the checkerboard calibration method or the picture-model joint forward calibration method to obtain the pose external parameters of the PTZ video surveillance camera, or read the real-scene image metadata to directly obtain the pose external parameters of the drone; Step 4.1.2: Create a virtual camera in the 3D space of the BIM model or real scene model of the construction site according to the pose external parameters of step 4.1.1; Step 4.1.3: According to the two-dimensional image coordinates of the element corresponding to the construction problem in step 3.3, determine the light ray from the center of the virtual camera to the center of the target element; Step 4.1.4: Calculate the coordinates of the collision point between the light ray and the BIM model or the solid model. The coordinates are the three-dimensional spatial coordinates of the construction problem. For panoramic cameras: Step 4.1.5: Bind the starting point and end point of the panoramic camera inspection route in the construction drawing; Step 4.1.6: Calculate the panoramic inspection route based on the orb-slam algorithm and the start and end point constraints on the construction drawings; Step 4.1.7: According to the construction drawings and inspection routes, determine the coordinates of the panoramic photo points with construction problems in the three-dimensional space of the BIM model or the real scene model as the three-dimensional space coordinates of the construction problems; For live images captured with a smartphone or tablet: Step 4.1.8: Directly read the metadata of the real-life image in the smartphone and tablet computer to obtain the coordinates of the points where the image was taken as the three-dimensional coordinates of the construction problem; Step 4.2: Aggregation and deduplication of construction issues; The construction problems in the real-life images from all sources are uniformly mapped to the project's three-dimensional space according to the shooting time. All construction problems within the preset time window t1 are traversed and the spatial distance between problems of the same type is calculated. If the distance is less than the specified distance threshold d1, it is regarded as the same problem in the multi-source images and merged.

8. The method for automatically tracking construction problems across image sources according to claim 7, characterized in that: In step 5, the method for automatically tracking construction problems includes: Map a construction problem to the three-dimensional space of the project, and search whether there are unclosed problems of the same type within the specified radius d1 of the problem point; If not, it indicates that the problem has occurred. The system marks the image shooting time as the time when the problem occurred, sends a problem ticket to the project manager and sets a time limit for problem handling. After the problem is handled, the manager manually closes the ticket and monitors whether the same type of problem occurs in the range within the specified time period t2. If so, it indicates that the historical problem ticket has not been actually handled. The system sends the ticket and warning to the manager again. If not, the problem ticket is closed at the system level. If yes, it determines whether the problem has exceeded the problem handling time limit set above. If yes, it means that the historical problem work order has not been handled in time, and the system sends the work order and warning to the management personnel again. If no, it only records the construction problem and does not take any other action.

Images