Image recognition-based device installation progress monitoring method and system

Abstract

The application provides an image recognition-based device installation progress monitoring method and system, wherein the method comprises the following steps: respectively constructing a BIM structure model and a BIM device model, and adding the BIM device model to a required position in the BIM structure model to obtain a BIM model; scanning a construction environment to obtain a construction environment image, superimposing the BIM model on the construction environment image to obtain a virtual-real combined scene image; acquiring the scene image and the construction environment image at a set rate, respectively, identifying the position relationship between the BIM device model in the scene image and the real device in the construction environment image, and automatically counting the device installation situation. The application does not depend on specific devices, can acquire real-time data for engineering projects, and can automatically complete the device installation progress counting process by the system, thereby improving the work efficiency, realizing the visualization and information interaction of the engineering construction progress, and further realizing the real-time monitoring of the device installation progress of the engineering site.

Description

Technical Field

[0001] This invention relates to the field of engineering progress management technology, specifically to a method and system for monitoring equipment installation progress based on image recognition, and also provides a corresponding device and computer-readable storage medium. Background Technology

[0002] Managing the progress of infrastructure projects to ensure timely completion is a crucial management indicator for both construction and contracting units. To guarantee project progress and achieve schedule targets, each unit tracks and manages the progress during construction. Our conventional method uses pre-prepared Gantt charts and network diagrams as a baseline plan. As the project progresses, progress-related data is collected, and the original plan is updated. The pace of progress is assessed by comparing the new plan with the baseline plan, and adjustments are made through corrective measures. This traditional method of project progress management certainly has many advantages. However, its biggest drawback is the lack of visualization and information interaction.

[0003] Currently, no descriptions or reports of technologies similar to this invention have been found, and no similar information has been collected domestically or internationally. Summary of the Invention

[0004] To address the aforementioned shortcomings in the prior art, this invention provides a method and system for monitoring device installation progress based on image recognition, along with a corresponding device and computer-readable storage medium.

[0005] This invention is achieved through the following technical solution.

[0006] According to one aspect of the present invention, a method for monitoring device installation progress based on image recognition is provided, comprising:

[0007] Construct a BIM structural model and a BIM equipment model separately, and add the BIM equipment model to the desired position in the BIM structural model to obtain the BIM model;

[0008] Scan the construction environment to obtain an image of the construction environment, and overlay the BIM model onto the construction environment image to obtain a scene image that combines virtual and real elements;

[0009] The scene image and the construction environment image are acquired at a set rate, the positional relationship between the BIM equipment model in the scene image and the real equipment in the construction environment image is identified, and the equipment installation status is automatically calculated.

[0010] Optionally, the step of separately constructing a BIM structural model and a BIM equipment model, and adding the BIM equipment model to the desired location in the BIM structural model to obtain a BIM model, includes:

[0011] Based on the CAD drawings, BIM structural models and BIM equipment models are created at a 1:1 scale with the construction site. The BIM equipment models are then added to the BIM structural models according to their actual placement locations to obtain a preliminary BIM model.

[0012] The preliminary BIM model is converted into a 3D model file, and the materials of the BIM structural model and the BIM equipment model are configured respectively to obtain the final BIM model; wherein:

[0013] The material of the BIM structural model is a fixed material, while the material of the BIM equipment model is set differently according to three dimensions: equipment type, equipment model and equipment name.

[0014] Optionally, the step of scanning the construction environment to obtain a construction environment image, and then overlaying the BIM model onto the construction environment image to obtain a scene image combining virtual and real elements, includes:

[0015] The construction site was scanned to obtain images of the construction environment;

[0016] Markers are created on the BIM model to obtain virtual spatial coordinates, and points corresponding to the marks in the construction environment image are obtained to obtain the construction environment spatial coordinates.

[0017] The virtual space coordinates are registered with the construction environment coordinates, so that the BIM model and the construction environment image are overlapped to obtain a scene image that combines virtual and real elements.

[0018] Optionally, the step of acquiring the scene image and the construction environment image at a set rate, identifying the positional relationship between the BIM equipment model in the scene image and the actual equipment in the construction environment image, and automatically calculating the equipment installation status includes:

[0019] The scene images and construction site images are acquired at a set rate, and the scene images and construction site images are matched one-to-one according to the timestamps to identify the BIM equipment models in the scene images and the real equipment in the construction site images.

[0020] Using the materials of the BIM model, determine whether the BIM equipment model overlaps with the actual equipment:

[0021] If there is an overlap, it is further determined whether the BIM equipment model and the real equipment are of the same equipment type, equipment model and equipment name. If they are, the real equipment is marked as an installed equipment. If not, it is searched within the specified pixel range of the material of the BIM structure model to see if there is a real equipment with the same equipment type, equipment model and equipment name. If so, the real equipment is marked as an installed equipment.

[0022] Conversely, the actual device is marked as an uninstalled device;

[0023] Complete the automatic statistics of equipment installation status.

[0024] Optionally, determining whether the BIM equipment model overlaps with the actual equipment using the materials of the BIM model includes:

[0025] The BIM structural model is set to a semi-transparent material; the BIM equipment model is set to different monochrome materials based on three dimensions: equipment type, equipment model, and equipment name.

[0026] All non-transparent materials in the current image are obtained, and the two-dimensional coordinates of the BIM device model in the current image are obtained according to the preset correspondence between monochrome materials and BIM device models. At the same time, the real device in the current image is identified and the two-dimensional coordinates of the real device are obtained. Based on these two sets of coordinates, it is determined whether the BIM device model and the real device overlap.

[0027] Optionally, the step of searching within a specified pixel range of the material of the BIM structural model to determine whether there is a real device with the same device type, model, and name includes:

[0028] Obtain the length and width pixel values ​​of the BIM structure model from the scene image, draw a circle or rectangle with a diameter of x times the pixel value, and obtain a specified range of pixel points, where x can be an adjustable parameter;

[0029] If a real device with the same device type, model, and name as the BIM device model exists within the specified pixel range, then the real device is considered to have been installed.

[0030] According to another aspect of the present invention, an image recognition-based device installation progress monitoring system is provided, comprising:

[0031] The BIM model processing module is used to construct BIM structural models and BIM equipment models respectively, and add the BIM equipment models to the desired positions in the BIM structural models to obtain BIM models.

[0032] The positioning module is used to scan the construction environment, obtain the construction environment image, and overlay the BIM model onto the construction environment image to obtain a scene image that combines virtual and real elements.

[0033] An image processing module is used to acquire the scene image and the construction environment image at a set rate, identify the BIM equipment model in the scene image and the real equipment in the construction environment image, and associate them.

[0034] The progress recognition module is used to determine the positional relationship between the BIM equipment model in the scene image and the actual equipment in the construction environment image, to recognize the installation progress, and to realize automatic statistics on the equipment installation status.

[0035] The network module is responsible for communication between the various modules.

[0036] Optionally, the BIM model processing module includes:

[0037] The model building unit is used to create BIM structural models and BIM equipment models according to CAD drawings and at a 1:1 scale with the construction site. The BIM equipment models are then added to the BIM structural models according to their actual placement locations to obtain a preliminary BIM model.

[0038] A format conversion unit is used to convert the preliminary BIM model into a 3D model file.

[0039] The attribute export unit is used to export the model attributes of the preliminary BIM model as a text document by component. The model attributes include model type, system, model, name, and dimension data, which can be used for subsequent material settings of the preliminary BIM model.

[0040] The material unit is used to assign different materials to the preliminary BIM model according to three dimensions: equipment type, equipment model and equipment name, to obtain the final BIM model.

[0041] Optionally, the positioning module includes:

[0042] The visual inertial SLAM unit is used to create markers on the BIM model, obtain virtual spatial coordinates, and acquire the points corresponding to the markers in the construction environment image to obtain the construction environment spatial coordinates. The virtual spatial coordinates are registered with the construction environment coordinates to keep the equipment stable when it moves in the construction environment, resulting in a scene image that combines virtual and real elements.

[0043] The coordinate input unit is used to output initial coordinates, providing an initial pose for coordinate matching between virtual and real spaces, which is used to improve the accuracy of coordinate matching.

[0044] Optionally, the image processing module includes:

[0045] A scene recording unit, which is used to acquire the scene images and the construction site images at a set rate respectively;

[0046] A sensor data recording unit, which is used to record each frame of the acquired scene image and the construction site image;

[0047] The data processing unit is used to match the scene images and the construction site images one-to-one according to the timestamps, and to associate the BIM equipment models in the scene images with the real equipment in the construction environment images.

[0048] Optionally, the progress recognition module includes:

[0049] An image recognition unit is used to identify the type and location of the BIM equipment model appearing in each frame of an image, as well as the category and location of the actual equipment appearing in each frame.

[0050] An image matching unit is used to determine whether the BIM equipment model overlaps with the actual equipment. If they overlap, it further determines whether the BIM equipment model and the actual equipment are of the same equipment type, model, and name. If they are, the actual equipment is marked as an installed device. If not, it searches within a specified pixel range of the material of the BIM structural model to see if there is an actual device with the same equipment type, model, and name. If so, the actual device is marked as an installed device; otherwise, it is marked as an uninstalled device.

[0051] Optionally, the system may also include any one or more of the following modules:

[0052] - Recording module, which stores data generated during system operation;

[0053] - Progress model viewing module, which is used to display the equipment installation progress.

[0054] According to a third aspect of the present invention, an apparatus is provided, comprising: a computer and a field scanning device connected to the computer; wherein:

[0055] The on-site scanning device includes a camera and an inertial sensor, used to scan the construction environment, acquire images of the construction environment, and send them to the computer;

[0056] The computer includes: a display, a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it can be used to perform the method described in any of the preceding claims, or to run the system described in any of the preceding claims.

[0057] Optionally, the on-site scanning device can be a mobile smart device, such as a mobile phone or tablet computer.

[0058] According to four aspects of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, can be used to perform the methods described in any of the preceding claims, or to run the system described in any of the preceding claims.

[0059] By adopting the above technical solution, the present invention has at least one of the following beneficial effects compared with the prior art:

[0060] This invention does not rely on specific equipment and can achieve on-site data scanning using the most common mobile smart devices on the market (such as mobile phones, tablets, etc.). During operation, all calculations can be performed on a server (computer). The smart device can provide the required on-site image data and inertial sensor data, reducing the requirements for equipment.

[0061] The equipment installation progress statistics of this invention can be automatically completed by the system, which improves work efficiency.

[0062] This invention is specifically designed for the equipment installation phase in engineering construction. It enables visual archiving and automatic statistics of installation progress during the equipment installation process, thereby achieving visualization and information interaction of the engineering construction progress.

[0063] This invention enables on-site data collection through mobile smart devices equipped with high-definition cameras and other network-enabled devices. It allows for real-time data acquisition of engineering projects, obtaining the installation progress of equipment on-site, and thus achieving real-time monitoring of the equipment installation progress. Attached Figure Description

[0064] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0065] Figure 1 This is an overall flowchart of a device installation progress monitoring method based on image recognition in one embodiment of the present invention;

[0066] Figure 2 This is a functional structure diagram of an image recognition-based equipment installation progress monitoring system according to an embodiment of the present invention;

[0067] Figure 3This is an overall flowchart of a preferred embodiment of the device installation progress monitoring method based on image recognition according to the present invention;

[0068] Figure 4 This is a functional structure diagram of an image recognition-based equipment installation progress monitoring system according to a preferred embodiment of the present invention;

[0069] Figure 5 This is a schematic diagram of the initial interface in a specific application example of the present invention;

[0070] Figure 6 This is a schematic diagram of the scene data acquisition interface in a specific application example of the present invention; wherein, (a) is the BIM model interface and (b) is the construction environment image interface;

[0071] Figure 7 This is a schematic diagram of the device identification process in a specific application example of the present invention. Detailed Implementation

[0072] The embodiments of the present invention are described in detail below: These embodiments are implemented based on the technical solution of the present invention, and provide detailed implementation methods and specific operation processes. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention.

[0073] One embodiment of the present invention provides a method for monitoring the installation progress of equipment based on image recognition.

[0074] like Figure 1 As shown, the image recognition-based device installation progress monitoring method provided in this embodiment may include the following steps:

[0075] S1. Construct the BIM structural model and the BIM equipment model respectively, and add the BIM equipment model to the desired position in the BIM structural model to obtain the BIM model;

[0076] S2, scan the construction environment to obtain the construction environment image, and overlay the BIM model onto the construction environment image to obtain a scene image that combines virtual and real elements;

[0077] S3 acquires scene images and construction environment images at a set rate, identifies the positional relationship between the BIM equipment model in the scene image and the actual equipment in the construction environment image, and automatically calculates the equipment installation status.

[0078] In a preferred embodiment of S1, a BIM structural model and a BIM equipment model are constructed respectively, and the BIM equipment model is added to the desired position in the BIM structural model to obtain a BIM model. This may include the following steps:

[0079] S101. Based on the CAD drawings, create BIM structural models and BIM equipment models at a 1:1 scale with the construction site. Add the BIM equipment models to the BIM structural models according to their actual placement locations to obtain the preliminary BIM model.

[0080] S102, the preliminary BIM model is converted into a 3D model file, and the materials of the BIM structural model and BIM equipment model are configured separately to obtain the final BIM model; in a specific application example, the 3D model file can be an FBX or OBJ file; where:

[0081] The materials for BIM structural models are fixed, while the materials for BIM equipment models are differentiated based on three dimensions: equipment type, equipment model, and equipment name. In a specific application example, a semi-transparent material can be added to the BIM structural model; the materials for the BIM equipment model are differentiated according to equipment type, and different equipment of the same type also need to be distinguished, so a single-color material can be added.

[0082] In a preferred embodiment of S2, scanning the construction environment to obtain an image of the construction environment, and overlaying the BIM model onto the construction environment image to obtain a scene image combining virtual and real elements, may include the following steps:

[0083] S201, Scan the construction site to obtain images of the construction environment;

[0084] S202: Create punctuation marks on the BIM model, obtain virtual space coordinates, and acquire the points corresponding to the punctuation marks in the construction environment image to obtain the construction environment space coordinates.

[0085] S203, register the virtual space coordinates with the construction environment coordinates, so that the BIM model and the construction environment image are overlaid to obtain a scene image that combines virtual and real elements.

[0086] In a preferred embodiment of S3, scene images and construction environment images are acquired at a set rate, the positional relationship between the BIM equipment model in the scene image and the actual equipment in the construction environment image is identified, and the equipment installation status is automatically calculated. This may include the following steps:

[0087] S301 acquires scene images and construction site images at a set rate, and matches the scene images and construction site images one by one according to the timestamp, and identifies the BIM equipment model in the scene image and the real equipment in the construction site image.

[0088] S302, using the materials of the BIM model, determine whether the BIM equipment model overlaps with the actual equipment:

[0089] If there is an overlap, it is further determined whether the BIM equipment model and the real equipment are the same equipment type, equipment model and equipment name. If they are, the real equipment is marked as installed equipment; if not, the BIM structural model is searched within the specified pixel range of the material to see if there is a real equipment with the same equipment type, equipment model and equipment name. If so, the real equipment is marked as installed equipment.

[0090] Conversely, if the actual device is not installed, it will be marked as an uninstalled device.

[0091] S303 automatically compiles statistics on equipment installation status.

[0092] In a preferred embodiment of S302, determining whether the BIM equipment model overlaps with the actual equipment using the materials of the BIM model includes:

[0093] Set the BIM structural model to a semi-transparent material; set the BIM equipment model to a different monochrome material based on three dimensions: equipment type, equipment model, and equipment name.

[0094] The system acquires all non-transparent materials in the current frame and obtains the two-dimensional coordinates of the BIM equipment model in the current frame based on the pre-set correspondence between monochrome materials and BIM equipment models. At the same time, it identifies the real equipment in the current frame and obtains the two-dimensional coordinates of the real equipment. Based on these two sets of coordinates, it determines whether the BIM equipment model and the real equipment overlap.

[0095] In a preferred embodiment of S302, searching within a specified pixel range of the material of the BIM structural model to determine whether there is a real device with the same device type, device model, and device name as it can include the following steps:

[0096] S3021: Obtain the length and width pixel values ​​of the BIM structural model from the scene image, draw a circle or rectangle with a diameter of x times the pixel value, and obtain the specified range of pixel points, where x can be an adjustable parameter;

[0097] S3022: If a real device with the same device type, model and name as the BIM device model exists within the specified pixel range, then the real device is considered to have been installed.

[0098] In some embodiments of the present invention:

[0099] The construction environment is scanned using equipment equipped with a camera and inertial sensors. The BIM structural model and BIM equipment model are then overlaid and displayed on the scanned image of the construction environment. This overlay display can be shown on a monitor.

[0100] Creating a BIM model requires the following processing:

[0101] Based on the BIM structural and equipment models created from CAD drawings, the BIM models are converted into 3D model files (e.g., FBX, OBJ formats). The materials of the BIM structural and equipment models are then replaced with different types of materials. After format conversion and material replacement, the models are imported into development tools (e.g., Unity software). Alternatively, material replacement can be performed within the development software itself.

[0102] The materials of BIM structural models are fixed, such as the addition of semi-transparent materials. The materials of BIM equipment models are differentiated according to equipment type, and different equipment of the same type (which can be distinguished by equipment name) also need to be differentiated, and monochrome materials can be added.

[0103] Using equipment equipped with cameras and inertial sensors, the construction environment space is scanned, and the BIM structural model and BIM equipment model are overlaid and displayed together in the scanned image of the construction environment space. Specifically, this can be done as follows:

[0104] By creating markers on the BIM model and registering them with the coordinates of points in the construction environment, the virtual space coordinates overlap with the spatial coordinates of the construction environment.

[0105] When the virtual space coordinates overlap with the construction environment space coordinates, scanned images of the construction environment and screen images (i.e., scene images) are acquired and stored at a set rate (e.g., 30 frames per second) for equipment installation progress recognition. Further, video (for obtaining scanned images of the construction environment) and screen recording (for obtaining scene images) are captured at the same frame rate. Through image recognition, equipment data from the video is obtained, including equipment type, model, name, recognition accuracy, time, location, and camera transform attribute. Equipment data from the BIM equipment model in the recorded screen is also obtained, and equipment location information is derived from material color values ​​and ranges.

[0106] The parameters obtained through image recognition are: device type and device model are used to label the device in the results; recognition accuracy is used to determine whether the image recognition is correct (an accuracy greater than a specified value is considered accurate); and the three parameters of time, position, and camera Transform attribute are used to confirm whether the virtual and real data are synchronized.

[0107] The image data obtained by recording the screen contains monochrome color blocks representing different devices. By querying the GPU rendering process data, the specific RGB values ​​of the monochrome color blocks and their two-dimensional coordinates in the current screen can be obtained.

[0108] The device installation status is automatically calculated by reading each frame of the image. The calculation method is as follows:

[0109] The system identifies whether the monochrome material of the BIM equipment model in the scene image overlaps with the actual equipment in the identified construction environment image. If they overlap, it further determines whether the two types of equipment are the same. If so, the actual equipment is marked as installed. If not, it searches within a specified pixel range of the semi-transparent material for the same type and the same equipment (i.e., the equipment type, model, and name are all the same). If so, the actual equipment is marked as installed; otherwise, it is marked as not installed. By comparing the equipment data obtained from the video and the equipment data obtained from the recorded screen, the installation progress of the equipment is finally determined.

[0110] Searching for a specified range of pixels within a semi-transparent material can be specifically:

[0111] Obtain the length and width pixel values ​​of the BIM structural model from the scene image, and draw a circle or rectangle with a diameter of x times the pixel value; x can be an adjustable parameter.

[0112] If a real device with the same device type, model, and name as the BIM device model exists within a specified pixel range, then the real device is considered to have been installed.

[0113] The process provides a tolerance value; when the equipment position difference is less than the tolerance value, the equipment can be considered to have been installed.

[0114] One embodiment of the present invention provides a device installation progress monitoring system based on image recognition.

[0115] like Figure 2 As shown, the image recognition-based device installation progress monitoring system provided in this embodiment may include the following modules:

[0116] The BIM model processing module is used to build BIM structural models and BIM equipment models separately, and add the BIM equipment models to the desired locations in the BIM structural models to obtain the BIM models.

[0117] The positioning module is used to scan the construction environment, obtain images of the construction environment, and overlay the BIM model onto the construction environment images to obtain a scene image that combines virtual and real elements.

[0118] The image processing module is used to acquire scene images and construction environment images at a set rate, identify the BIM equipment models in the scene images and the real equipment in the construction environment images, and associate them.

[0119] The progress recognition module is used to determine the positional relationship between the BIM equipment model in the scene image and the actual equipment in the construction environment image, to identify the installation progress, and to realize the automatic statistics of equipment installation status.

[0120] The network module is responsible for communication between modules, ensuring smooth data exchange.

[0121] In a preferred embodiment, the BIM model processing module may include the following units:

[0122] The model building unit is used to create BIM structural models and BIM equipment models based on CAD drawings and at a 1:1 scale with the construction site. The BIM equipment models are then added to the BIM structural models according to their actual placement locations to obtain a preliminary BIM model. In a specific application example, this model building unit can be existing 3D modeling software or a plugin for modeling software.

[0123] The format conversion unit is used to convert the initial BIM model into 3D model files in formats such as FBX and OBJ.

[0124] The Attribute Export Unit is used to export model attributes as a text document by component. In a specific application example, the model attributes include model type (equipment type), system, model model (equipment model), model name (equipment name), and dimension data, which can be used for subsequent settings of the initial BIM model materials.

[0125] Material unit: This unit is used to assign different materials to the initial BIM model according to three dimensions: equipment type, equipment model and equipment name, to obtain the final BIM model.

[0126] Furthermore, there are multiple shaders in the material unit, and each shader can control one or more materials.

[0127] In a preferred embodiment, the positioning module may include the following units:

[0128] The visual inertial SLAM unit is used to create markers on the BIM model, obtain virtual spatial coordinates, and acquire the points corresponding to the markers in the construction environment image to obtain the construction environment spatial coordinates. The virtual spatial coordinates are registered with the construction environment coordinates to keep the equipment stable when it moves in the construction environment, resulting in a scene image that combines virtual and real elements.

[0129] The coordinate input unit is used to output initial coordinates, providing an initial pose for matching virtual and real spaces. This initial pose can be used to improve the accuracy of coordinate matching.

[0130] In a preferred embodiment, the image processing module may include the following units:

[0131] Scene recording unit, which is used to acquire scene images and construction site images at a set rate respectively;

[0132] The sensor data recording unit is used to record each frame of scene image and construction site image acquired. In a specific application example, when each frame of image is recorded, the data of the real equipment in the construction site image at that time is recorded synchronously, including acceleration, angular velocity, etc.

[0133] The data processing unit is used to map scene images and construction site images one-to-one according to timestamps, and to associate the BIM equipment models in the scene images with the real equipment in the construction environment images. In a specific application example, the pose of the real equipment can be calculated when each frame of the image is acquired based on the data (acceleration, angular velocity) of the real equipment in the construction site images.

[0134] Furthermore, by calculating the actual pose in real time, the movement trajectory of the equipment can be obtained. By synchronizing this trajectory with the camera in the BIM model, a stable integration of the BIM model and the construction environment can be achieved.

[0135] In a preferred embodiment, the progress recognition module may include the following units:

[0136] Image recognition unit, which is used to identify the type and location of the BIM equipment model appearing in each frame of image, as well as the category and location of the real equipment appearing;

[0137] The image matching unit determines whether the BIM equipment model overlaps with the actual equipment. If they overlap, it further determines whether the BIM equipment model and the actual equipment are of the same equipment type, model, and name. If they are, the actual equipment is marked as installed. If not, it searches within a specified pixel range of the material of the BIM structural model to see if there is an actual equipment with the same equipment type, model, and name. If so, the actual equipment is marked as installed; otherwise, it is marked as not installed.

[0138] In a preferred embodiment, the system further includes any one or more of the following modules:

[0139] - Recording module, which stores a series of data generated during system operation;

[0140] - Progress model viewing module, which is used to display the equipment installation progress.

[0141] It should be noted that the steps in the method provided by the present invention can be implemented using corresponding modules, devices, units, etc. in the system. Those skilled in the art can refer to the technical solution of the method to realize the composition of the system. That is, the embodiments in the method can be understood as preferred examples for building the system, and will not be elaborated here.

[0142] One embodiment of the present invention provides a device that may include: a computer and a field scanning device connected to the computer; wherein:

[0143] The on-site scanning device includes a camera and an inertial sensor, used to scan the construction environment, acquire images of the construction environment, and send them to a computer;

[0144] A computer includes: a display, a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, may perform any of the methods described above, or run any of the systems described above.

[0145] In a specific application example, the field scanning device can be a mobile smart device, such as a mobile phone or tablet computer that includes a camera and an inertial sensor, or a portable device such as a laptop computer with an external camera and an inertial sensor.

[0146] An embodiment of the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can be used to perform any of the above-described methods, or to run any of the above-described systems.

[0147] Optionally, the memory is used to store programs; the memory may include volatile memory, such as random-access memory (RAM), such as static random-access memory (SRAM), double data rate synchronous dynamic random-access memory (DDR SDRAM), etc.; the memory may also include non-volatile memory, such as flash memory. The memory is used to store computer programs (such as application programs, functional modules, etc. that implement the above methods), computer instructions, etc., and the aforementioned computer programs, computer instructions, etc., can be partitioned and stored in one or more memories. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by the processor.

[0148] The aforementioned computer programs, computer instructions, etc., can be stored in partitions within one or more memory locations. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by a processor.

[0149] A processor is used to execute a computer program stored in memory to implement the various steps of the methods involved in the above embodiments. For details, please refer to the relevant descriptions in the preceding method embodiments.

[0150] The processor and memory can be separate structures or integrated structures. When the processor and memory are separate structures, they can be coupled together via a bus.

[0151] To make the objectives, technical solutions, and advantages of the above embodiments of the present invention clearer, the technical solutions in the above embodiments of the present invention will be further described clearly and completely below in conjunction with a specific application example and accompanying drawings.

[0152] In the following specific application examples, the devices used, equipped with displays, cameras, and inertial sensors, can be smartphones or tablets that include this hardware, or tablets or laptops with external cameras and inertial sensors. These devices provide the hardware foundation for realizing the function of overlaying virtual models onto real-world models in the following specific application examples.

[0153] This specific application example illustrates a device installation progress monitoring method based on image recognition. As shown in Figure 3, this specific application example provides a device installation progress monitoring method based on image recognition, which includes the following steps:

[0154] Step 100: Using modeling software, create BIM structural models and BIM equipment models separately based on the CAD drawings. Import the models into development tools such as U3D or UE4, and add different materials to different types of models.

[0155] Step 200: Using the positioning function, match the virtual space coordinates with the real space, and then display the BIM model on the screen, thus achieving virtual-real integration.

[0156] Step 300: Acquire scene images, synchronously read inertial sensor data, and match the images with the sensor data one by one according to the timestamp.

[0157] Step 400: Frame by frame, acquire the BIM equipment model in the image and the equipment information at the construction site to obtain the equipment installation information.

[0158] Specifically, step 100 corresponds to the BIM model processing module, step 200 corresponds to the positioning module, step 300 corresponds to the image processing module, and step 400 corresponds to the progress recognition module. When different modules are installed on different devices, the network module is responsible for information transmission and interaction.

[0159] In step 100, Revit is used for modeling, Unity3D is used for development, and C# is the programming language. The model import tool is partly a secondary development based on Revit, and its main function is to export information for each component as a text document. These text documents are named according to the component family type (Family) and the component number (Element-ID). The model import tool is also developed based on Unity3D, and its main function is to associate the attribute text document file with the model's FBX file.

[0160] Furthermore, different materials are added to different types of models. Specifically, all structural column models are given the same material M_Column, exterior walls are given the material M_ExteriorWall, and so on. Fire sprinklers are given the material M_Sprinkler, 50KV transformers are given the material M_PT50KV, smoke exhaust ducts are given the material M_PYDuct, and so on. All materials are solid colors, and different materials are distinguished by different RGB values ​​to facilitate the subsequent image recognition process. All materials are shaded using two passes: the first pass renders only the back side with Cull set to Front; the second pass renders only the front side with Cull set to Back.

[0161] In step 200, the virtual space coordinates are matched with the real space. Specifically, two points are selected in the top view of the BIM model, and their coordinates in the model space are temporarily recorded as planar coordinates (vertical value is 0). Then, the camera is activated to identify the ground plane of the construction site, and two anchor points are added at the corresponding locations. The two coordinates in the model space are then overlapped with the two virtual anchor points on the ground plane of the construction site, thus achieving the matching of the virtual space coordinates with the real space.

[0162] Furthermore, the method for identifying the ground plane of the construction site, for example, uses the `UnityEngine.XR.ARFoundation.ARPlanesChangedEventArgs` function in the built-in ARFoundation plugin of Unity3D. The operation of adding anchor points after identifying the construction site plane is performed using `UnityEngine.Object.Instantiate`. As for the matching method between anchor points and virtual space plane anchor points, the specific process is as follows:

[0163] Assume the coordinates of two points in virtual space are X1 and X2; the coordinates of anchor points are A1 and A2; the virtual space coordinate system is e; the real space coordinate system is E; the translation vector is T; and the rotation vector is R.

[0164] Calculate the values ​​of T and R such that E = Re + T;

[0165] in

[0166] Furthermore, once the coordinates of the virtual and real spaces are matched, the BIM model needs to be stably superimposed on the real space as the device moves. This process is achieved using SLAM technology. In the field of robotics, this technology is defined as Simultaneous Localization and Mapping (SLAM). SLAM technology can be described as follows: a robot starts moving from an unknown location in an unknown environment, performs self-localization based on position estimation and a map during the movement, and simultaneously builds an incremental map based on its self-localization to achieve autonomous localization and navigation. For a SLAM technology, depending on the sensor used, there are three main solutions: SLAM based on RGB-D cameras, SLAM based on laser point clouds, and vision-based SLAM. Vision-based SLAM, when combined with Visual-Inertial Odometry (VIO), is called VIO-SLAM, which can be further divided into three types: based on monocular cameras, binocular cameras, and RGB-D cameras. Each of the three implementation schemes has its advantages and disadvantages.

[0167] VIO-SLAM solutions based on monocular cameras are low in cost, but have uncertain spatial scales and initialization issues.

[0168] The VIO-SLAM solution using binocular cameras can acquire spatial scale, but it is complex to configure and has a high cost.

[0169] The VIO-SLAM solution for RGB-D cameras can also acquire spatial scale data at a low cost, but it is susceptible to lighting interference.

[0170] For this specific application example, both monocular camera solutions and RGB-D solutions can be used depending on the different site conditions. In this specific application example, the monocular solution is adopted.

[0171] In step 300, a screen image is acquired, inertial sensor data is read synchronously, and the image and sensor data are matched one-to-one according to the timestamp. Specifically, this includes:

[0172] The screen recording function is implemented using UnityEngine.Recorder, with a recording frame rate of 30. The screen recording dataset is denoted as RecorderSet30. Camera permissions are obtained using Unity.Application.RequestUserAuthorization(), and the camera dataset is denoted as WebCamTexturSet30. Accelerometer data is obtained using Sensor.TYPE_ACCELEROMETER, orientation sensor data using Sensor.TYPE_ORIENTATION, gyroscope data using Sensor.TYPE_GYROSCOPE, linear accelerometer data using Sensor.TYPE_LINEAR_ACCELERATION, and gravity sensor data using Sensor.TYPE_GRAVITY, totaling VIOSET. Timestamps are obtained using System.DateTime.Now and denoted as TimeSET. All data is packaged into a dataset DATASET at 30 sets per second and then sent to the progress recognition module via the network module.

[0173] Furthermore, the inertial sensor dataset can be used for camera transformation verification, but it is not mandatory.

[0174] Furthermore, when the screen recording function is running, the transparency of the structural model materials in the BIM model can be set to 0.

[0175] In step 400, the BIM equipment model in the image and the equipment information of the construction site are acquired frame by frame to obtain the equipment installation information. Specifically, this includes: acquiring a set of DATASET, first identifying the material color blocks of the BIM equipment model in RecorderSet30 to obtain the equipment type, quantity, and relative pixel coordinates C in the image. BIM Then, it identifies the device type and relative pixel coordinates C present in WebCamTexturSet30. R And count their quantity.

[0176] Furthermore, in this specific application example, the PaddlePaddle image recognition database was used to implement the image recognition function. The dataset was collected from equipment actually used in the construction project.

[0177] Furthermore, the image recognition results are recorded as serialized information, extracting multiple device data points from each image, including device type, recognition accuracy, time, location (relative coordinates in the image), and the camera's Transform attribute at that moment, m(Family.Name, λ, Time, C). R,Transform); extract the 20 data points before and after the maximum value of λ as the progress monitoring dataset, denoted as mSet[Family.Name]. If there are fewer than 20 data points, take the maximum value.

[0178] Furthermore, obtain screen recording data, specifically the image set corresponding to the time period mSet[Family.Name], and calculate whether the device position is consistent in these images.

[0179] Furthermore, the device type and location are obtained from the screen recording data. The specific process is as follows:

[0180] Search for the target device material in the image, specifically the pre-set color parameters, obtain all pixels with that RGB value, get the boundary values, and then calculate the midpoint based on the boundary values. The coordinates of the midpoint are the device position C. BIM .

[0181] Furthermore, considering that the equipment installation location often deviates slightly from the drawings due to various reasons during actual construction, an allowable deviation value D is given when identifying the equipment installation progress, as long as the following conditions are met:

[0182] D≥|C BIM -C R |

[0183] The device can be considered to be installed at location C. R .

[0184] Where D = F(X,Y,L), X and Y are the dimensions of the device in the image, and L is the distance between the BIM device model and the camera, which can be calculated by m.Transform and mSet[Family.name].Transform.

[0185] Through this cyclical process, all equipment will be tested, ultimately yielding the equipment installation progress information for the construction project. All calculation results generated in the above steps will be stored in the recording module.

[0186] The installation status of the equipment can be viewed through the progress model module.

[0187] Specifically, in this application example, seven functional modules are defined: BIM model processing module, positioning module, image processing module, progress recognition module, network module, recording module, and progress model viewing module. The BIM model processing module builds the model on the Revit platform and exports model information as FBX, model attribute information as text, loads the FBX on the Unity platform, and sets the model materials. The positioning module handles the matching of virtual space coordinates with real space, enabling the BIM model to be overlaid and displayed in the construction scene. The image processing module is divided into three functional groups: a scene recording unit (acquiring camera and screen images), a sensor data recording unit, and a data processing unit, packaging multiple datasets per frame. The progress recognition module identifies equipment information in the camera dataset and screen recording dataset respectively, comparing them to obtain equipment installation information. The network module is responsible for communication between the various modules. The recording module is responsible for storing all data generated during system operation. The progress model viewing module is responsible for displaying the monitoring results to the user.

[0188] In some instances, communication between the modules can be achieved using a mobile phone as the front-end acquisition device and a cloud server as the back-end processing device: except for the positioning module and the image processing module, the other three modules are all located on the back-end server.

[0189] In some instances, when using a standalone monocular camera and a tablet computer as hardware, everything except the scene recording unit of the image processing module is located on the backend server.

[0190] The following describes the composition of the equipment installation monitoring system involved in this specific application example.

[0191] like Figure 4 As shown, the specific application provides a device installation monitoring system that can be used to execute the image recognition-based device installation progress monitoring method described above, including the following modules:

[0192] BIM model processing module 1 is used to convert model formats, modify model materials, and compile model information.

[0193] Positioning module 2 is used for coordinate matching between virtual and real spaces to achieve a virtual-real fusion effect.

[0194] Image processing module 3 is used to acquire screen images and simultaneously obtain inertial sensor data from the device, and associate it with the images.

[0195] Progress recognition module 4 identifies whether the equipment model has been installed in the construction scenario.

[0196] Network module 5 is responsible for communication between the modules.

[0197] Module 6 is responsible for all data generated during system operation.

[0198] The progress model viewing module 7 is responsible for displaying the equipment installation progress.

[0199] Furthermore, the BIM model processing module 1 includes:

[0200] Model building unit 10 creates BIM structural models and BIM equipment models, and adds the BIM equipment models to the BIM structural models according to their actual placement locations.

[0201] The format conversion unit 11 is used to process the BIM model into formats such as FBX and OBJ.

[0202] Attribute Export Unit 12: Exports model attributes as a text document by component.

[0203] Material unit 13 is used to assign different materials to the model according to its category.

[0204] Furthermore, the positioning module 2 is characterized by comprising:

[0205] The visual-inertial SLAM unit 21 acquires the device's position in space, matching the coordinates of the virtual space and the real space, and keeping the device stable when it moves.

[0206] The coordinate input unit 22 is used to output the initial coordinates, providing the initial pose for matching virtual space and real space.

[0207] Furthermore, the image processing module 3 includes:

[0208] Scene recording unit 31 records the image of the BIM model superimposed on the real scene after matching the coordinates of the virtual space and the real space. At the same time, the camera records the scene image without the BIM model.

[0209] The sensor data recording unit 32 synchronously records the inertial sensor data, including acceleration and angular velocity, when each frame of image is recorded.

[0210] The data processing unit 33 calculates the device's pose when each frame of image is acquired based on the inertial sensor data.

[0211] Furthermore, the progress recognition module 4 includes:

[0212] Image recognition unit 41 is used to identify the type and location of BIM equipment models appearing in each frame of image, as well as the type and location of real equipment models appearing.

[0213] Image matching unit 42 is used to calculate whether similar models from different sources appear within a specified pixel range.

[0214] Furthermore, similar models from different sources refer to both BIM equipment models and actual equipment models existing on the construction site.

[0215] The following section further explains the device installation progress monitoring method and the user interface of the system in actual operation, as provided in this specific application example.

[0216] like Figure 5 As shown, the specific example of the device installation progress monitoring method and the corresponding user interface during system operation in this application instance is explained below:

[0217] Figure 5 The initial interface for execution is also the scene selection interface. In this specific application example, the software interface has been simplified to the greatest extent, retaining only the functions related to the technical solution in this specific application example. Figure 6 This is the interface for scene data collection. Figure 7 This is an example of device identification.

[0218] The application method is as follows:

[0219] (1) When entering a new construction environment, first select the BIM structural model corresponding to the construction environment in the "SELECT PROJECT" interface, and then create two positioning points on the top view of the model. It is best to select the four corner points of the column as the positioning points to improve the subsequent registration accuracy. Finally, click the button on the left to enter the scene data acquisition interface.

[0220] (2) After entering the scene acquisition interface, move the mobile device to acquire as much ground information as possible to establish a working plane. Then click the button to create two anchor points, the anchor points being the corresponding locations of the BIM model's positioning points on the construction site. Click the button "Register and Scale" or "Register Without Scale," and the BIM model will be overlaid onto the construction site. Furthermore, the two buttons do not affect the implementation of the invention's functions; the identification of BIM devices is based on color blocks.

[0221] (3) After the model is overlaid, click the "Acquire" button. The user, carrying the equipment, will slowly move through the construction scene to comprehensively collect data from the site. Take a fire sprinkler head as an example: Figure 7 (a) is the image captured by the camera. Figure 7 (b) is the screen display. Figure 7 (c) is an example of device identification.

[0222] It should be noted that the specific interface interaction can be adjusted according to user preferences and actual engineering conditions. This embodiment is only an example of method demonstration and is not intended to limit the present invention.

[0223] In this specific application example, both the device installation progress monitoring method and system can be implemented using a computer-readable storage medium. This medium stores computer code, and when the computer code is executed, the methods described above are performed. Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. This program can be stored in a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0224] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media, such as modulated data signals and carrier waves.

[0225] It should be noted that the software program can be executed by a processor to implement the steps or functions described above. Similarly, the software program of this invention (including related data structures) can be stored in a computer-readable recording medium, such as RAM memory, magnetic or optical drives, floppy disks, and similar devices. Furthermore, some steps or functions of this invention can be implemented in hardware, for example, as circuitry that works with a processor to perform the various functions or steps. The methods disclosed in the embodiments shown in this specification can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In implementation, the steps of the above methods can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor described above can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The various methods, steps, and logic block diagrams disclosed in the embodiments of this specification can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this specification can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above methods.

[0226] The systems, devices, modules, or units described in the above embodiments of the present invention can be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices. Computer-readable media includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEP ROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves. It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. In the absence of further restrictions, an element defined by the phrase "including one..." does not exclude the presence of other identical elements in the process, method, goods or equipment that include the element.

[0227] Furthermore, a portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. The program instructions invoking the methods of the invention may be stored in a fixed or removable recording medium, and / or transmitted via a data stream in a broadcast or other signal-carrying medium, and / or stored in the working memory of a computer device operating according to the program instructions. Here, one embodiment of the invention includes an apparatus comprising a memory for storing computer program instructions and a processor for executing the program instructions, wherein, when the computer program instructions are executed by the processor, the apparatus is triggered to operate the methods and / or technical solutions based on the foregoing embodiments of the invention.

[0228] The image recognition-based equipment installation progress monitoring method and system provided in the above embodiments of the present invention uses a device equipped with a camera and an inertial sensor to scan the construction environment space and overlay the BIM model onto the construction environment. Then, it synchronously collects camera and screen image data, and automatically determines the equipment installation status through an image recognition system. It does not rely on specific equipment and can achieve on-site data scanning using the most common commercially available mobile smart devices (such as mobile phones and tablets). All calculations during operation can be performed on a server (computer). Smart devices can provide the required on-site image data and inertial sensor data, reducing equipment requirements. The statistical process of equipment installation progress can be automatically completed by the system, improving work efficiency. Specifically targeting the equipment installation stage in engineering construction, it performs visual archiving and automatic statistics of installation progress during the equipment installation process, realizing visualization and information interaction of engineering construction progress. On-site data collection is achieved through mobile smart devices with high-definition cameras and other network-enabled devices, enabling real-time data acquisition of engineering projects and obtaining the on-site equipment installation progress, thereby achieving real-time monitoring of the on-site equipment installation progress.

[0229] Any matters not covered in the above embodiments of the present invention are well-known in the art.

[0230] The specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A method for monitoring equipment installation progress based on image recognition, characterized in that, include: Construct a BIM structural model and a BIM equipment model separately, and add the BIM equipment model to the desired position in the BIM structural model to obtain the BIM model; Scan the construction environment to obtain an image of the construction environment, and overlay the BIM model onto the construction environment image to obtain a scene image that combines virtual and real elements; The scene image and the construction environment image are acquired at a set rate, and the scene image and the construction environment image are matched one by one according to the timestamp to identify the BIM equipment model in the scene image and the real equipment in the construction environment image. The BIM structural model is set to a semi-transparent material; based on the three dimensions of equipment type, equipment model, and equipment name, the BIM equipment model is set to different monochrome materials; all non-semi-transparent materials in the current image are obtained, and the two-dimensional coordinates of the BIM equipment model in the current image are obtained according to the preset correspondence between monochrome materials and BIM equipment models; at the same time, the real equipment in the current image is identified, and the two-dimensional coordinates of the real equipment are obtained, and it is determined whether the BIM equipment model and the real equipment overlap based on these two sets of coordinates; If there is overlap, it is further determined whether the BIM equipment model and the real equipment are of the same equipment type, model, and name. If they are, the real equipment is marked as installed equipment. If not, the length and width pixel values ​​of the BIM structure model are obtained from the scene image. A circle is drawn with x times the length or width pixel value as the diameter, or a rectangle is drawn with x times the length and width pixel value as the new length and width, to obtain a specified range of pixels, where x is an adjustable parameter. When there is a real equipment of the same equipment type, model, and name as the BIM equipment model within the specified range of pixels, the real equipment is considered to be installed. Conversely, the actual device is marked as an uninstalled device; Complete the automatic statistics of equipment installation status.

2. The device installation progress monitoring method based on image recognition according to claim 1, characterized in that, The process of separately constructing a BIM structural model and a BIM equipment model, and adding the BIM equipment model to the desired location in the BIM structural model to obtain a BIM model, includes: Based on the CAD drawings, BIM structural models and BIM equipment models are created at a 1:1 scale with the construction site. The BIM equipment models are then added to the BIM structural models according to their actual placement locations to obtain a preliminary BIM model. The preliminary BIM model is converted into a 3D model file, and the materials of the BIM structural model and the BIM equipment model are configured respectively to obtain the final BIM model; wherein: The material of the BIM structural model is a fixed material, while the material of the BIM equipment model is set differently according to three dimensions: equipment type, equipment model and equipment name.

3. The device installation progress monitoring method based on image recognition according to claim 1, characterized in that, The process involves scanning the construction environment to obtain an image of the construction environment, then overlaying the BIM model onto the construction environment image to obtain a scene image that combines virtual and real elements, including: The construction site was scanned to obtain images of the construction environment; Markers are created on the BIM model to obtain virtual spatial coordinates, and points corresponding to the marks in the construction environment image are obtained to obtain the construction environment spatial coordinates. The virtual space coordinates are registered with the construction environment coordinates, so that the BIM model and the construction environment image are overlapped to obtain a scene image that combines virtual and real elements.

4. A device installation progress monitoring system based on image recognition, characterized in that, include: The BIM model processing module is used to construct BIM structural models and BIM equipment models respectively. The BIM structural model is set to a semi-transparent material. Based on three dimensions—equipment type, equipment model, and equipment name—the BIM equipment model is set to different monochrome materials. The BIM equipment model is then added to the desired position in the BIM structural model to obtain the BIM model. The positioning module is used to scan the construction environment, obtain the construction environment image, and overlay the BIM model onto the construction environment image to obtain a scene image that combines virtual and real elements. An image processing module is used to acquire the scene image and the construction environment image at a set rate, and to match the scene image and the construction environment image one by one according to the timestamp, identify the BIM equipment model in the scene image and the real equipment in the construction environment image, and associate them. A progress recognition module is used to determine the positional relationship between the BIM equipment model in the scene image and the actual equipment in the construction environment image, to recognize the installation progress, and to automatically count the installation status of the equipment; including: An image recognition unit is used to identify the type and location of the BIM equipment model appearing in each frame of an image, as well as the category and location of the actual equipment appearing in each frame. An image matching unit is used to determine whether the BIM equipment model overlaps with the actual equipment. If they overlap, it further determines whether the BIM equipment model and the actual equipment are of the same equipment type, model, and name. If they are, the actual equipment is marked as installed. If not, the length and width pixel values ​​of the BIM structure model are obtained from the scene image. A circle is drawn with a diameter equal to x times the length or width pixel value, or a rectangle is drawn with a new length and width equal to x times the length and width pixel value, to obtain a specified range of pixels, where x is an adjustable parameter. When an actual equipment of the same equipment type, model, and name as the BIM equipment model exists within the specified range of pixels, the actual equipment is considered to be installed. Otherwise, the actual equipment is marked as not installed. The network module is responsible for communication between the various modules.

5. The equipment installation progress monitoring system based on image recognition according to claim 4, characterized in that, It also includes any one or more of the following: - The BIM model processing module includes: The model building unit creates a BIM structural model and a BIM equipment model based on CAD drawings and at a 1:1 scale with the construction site. The BIM equipment model is then added to the BIM structural model according to its actual placement location to obtain a preliminary BIM model. A format conversion unit is used to convert the preliminary BIM model into a 3D model file. The attribute export unit is used to export the model attributes of the preliminary BIM model as a text document by component, which is used for the subsequent material settings of the preliminary BIM model. Material unit, which is used to assign different materials to the preliminary BIM model according to three dimensions: equipment type, equipment model and equipment name, to obtain the final BIM model; - The positioning module includes: The visual inertial SLAM unit is used to create markers on the BIM model, obtain virtual spatial coordinates, and acquire the points corresponding to the markers in the construction environment image to obtain the construction environment spatial coordinates. The virtual spatial coordinates are registered with the construction environment coordinates to keep the equipment stable when it moves in the construction environment, resulting in a scene image that combines virtual and real elements. The coordinate input unit is used to output the initial coordinates, providing the initial pose for coordinate matching between the virtual space and the real space; - The image processing module includes: A scene recording unit, which is used to acquire the scene images and the construction site images at a set rate respectively; A sensor data recording unit, which is used to record each frame of the acquired scene image and the construction site image; The data processing unit is used to match the scene images and the construction site images one-to-one according to the timestamps, and to associate the BIM equipment models in the scene images with the real equipment in the construction environment images.

6. The equipment installation progress monitoring system based on image recognition according to claim 4 or 5, characterized in that, It also includes any one or more of the following modules: - Recording module, which stores data generated during system operation; - Progress model viewing module, which is used to display the equipment installation progress.

7. A device, characterized in that, include: A computer and a field scanning device connected to the computer; wherein: The on-site scanning device includes a camera and an inertial sensor, used to scan the construction environment, acquire images of the construction environment, and send them to the computer; The computer includes: a display, a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it can be used to perform the method of any one of claims 1-3, or to run the system of any one of claims 4-6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program can be used to perform the method of any one of claims 1-3, or to run the system of any one of claims 4-6.

Images