Man-machine collaborative management method and system for electromechanical facility installation based on BIM and AR

Abstract

The invention relates to an electromechanical facility installation man-machine collaborative management method and system based on BIM and AR. The method comprises the steps that AR equipment displays AR scene content corresponding to a building scene; the AR scene content at least comprises a BIM model and a three-dimensional scene model of a building; wherein the three-dimensional scene model is generated by pre-collected point cloud data of a building scene target area and a texture map; an input operation of a user through the AR device is detected, the input operation is related to at least one electromechanical facility model, and the at least one electromechanical facility model is a component model corresponding to one or more electromechanical facilities in the BIM model; and executing a preset operation on a target electromechanical facility model in response to the input operation, wherein the target electromechanical facility model is any one of the at least one electromechanical facility model.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the technical field of electromechanical facility installation management, and in particular, to a method and system for human-machine collaborative management of electromechanical facility installation based on BIM and AR. Background Art

[0002] At present, the installation management of complex electromechanical facilities in construction sites faces challenges such as multidisciplinary collaboration, errors and collisions during installation, and inefficient progress management and problem feedback, resulting in low construction efficiency and management level of complex electromechanical facilities, making it difficult to provide strong support for intelligent management of construction sites. BIM (Building Information Modeling), building information modeling technology is the most advanced modeling method with the highest degree of digitization in the construction field. It can realize information integration and sharing throughout the life cycle of a building, and improve the collaborative efficiency of various stages such as design, construction and maintenance.

[0003] BIM is used in related technologies to realize the installation management of electromechanical facilities. However, there is a lack of installation guidance and human-machine collaborative management of electromechanical facilities on construction sites, such as complex electromechanical facilities. For example, there is a lack of effective construction guidance research on complex electromechanical facility systems with multiple disciplines, which makes it difficult to meet the needs of complex scenarios in actual projects. There is a lack of research on human-machine collaborative management of multiple terminal devices, and information between devices cannot be shared in real time, making it difficult to meet the needs of multi-user collaborative operation for installation management. This results in low construction management efficiency and collaboration level for complex electromechanical facilities. Summary of the invention

[0004] In order to solve the above technical problems or at least partially solve the above technical problems, the embodiments of the present disclosure provide a method and system for human-machine collaborative management of electromechanical facility installation based on BIM and AR.

[0005] In a first aspect, an embodiment of the present disclosure provides a method for human-machine collaborative management of electromechanical facility installation based on BIM and AR, including: The AR device displays AR scene content corresponding to the building scene; the AR scene content includes at least a BIM model of the building and a three-dimensional scene model; wherein the three-dimensional scene model is generated by pre-collected point cloud data and texture mapping of the target area of ​​the building scene; Detecting an input operation of a user through the AR device, wherein the input operation is related to at least one electromechanical facility model, and the at least one electromechanical facility model is a component model corresponding to one or more electromechanical facilities in the BIM model; A preset operation on a target electromechanical facility model is performed in response to the input operation, the target electromechanical facility model being any one of the at least one electromechanical facility model.

[0006] In one embodiment, the three-dimensional scene model is generated based on point cloud data and texture maps generated by a panoramic camera collecting data at multiple sites. The panoramic camera includes a depth sensor and at least one camera. The layout positions and collection order of the multiple sites are obtained by performing path planning processing on the spatial information of the building based on a path planning algorithm. The spatial information of the building is obtained based on the BIM model of the building.

[0007] In one embodiment, the method further comprises: Capturing a plurality of key geometric feature points in the three-dimensional scene model, and storing the plurality of key geometric feature points and the texture map in a first database; Acquire multiple key geometric feature points of the BIM model, acquire multiple key geometric feature points of the three-dimensional scene model from the first database, and calculate a transformation matrix based on the multiple key geometric feature points of the BIM model and the multiple key geometric feature points of the three-dimensional scene model, wherein the transformation matrix includes translation, rotation, and scaling information of the BIM model; The position, size ratio and direction of the BIM model in the three-dimensional scene model are adjusted based on the transformation matrix to achieve accurate alignment with the scene.

[0008] In one embodiment, the method further comprises: Acquire semantic information of each component in the BIM model, store the semantic information of each component in a target database in the form of multiple fields, and introduce a timestamp field to record the latest update time of each piece of data; During the construction management process, when it is detected that a user changes the component information in the BIM model through the AR device, a pre-written script is used to automatically update the data to the target database, wherein each time the data is updated, the target database automatically refreshes the timestamp field of the corresponding component to mark the latest status; After the BIM model is loaded in the AR scene content, the semantic information of the corresponding component is queried from the target database through the unique ID of the component, and the semantic information of the component obtained by the query is bound to the corresponding component model.

[0009] In one embodiment, the AR device includes at least a head-mounted AR device and a handheld AR device; the handheld AR device provides a 2D AR interaction interface and supports a touch screen interaction mode; the head-mounted AR device provides a 3D AR interaction interface and supports one or more of a hand ray interaction mode, a press interaction mode, and a voice command interaction mode; the virtual scene space sharing between the head-mounted AR device and the handheld AR device is achieved through the Azure Spatial Anchors (ASA) technology; the real-time communication between the head-mounted AR device and the handheld AR device is achieved through the PUN (PhotonUnity Networking) technology.

[0010] In one embodiment, the user operation results on the head-mounted AR device and the handheld AR device are updated to the target database in real time, and the user operation results are synchronized with each other and displayed in the respective AR scene contents in a dynamic rendering manner.

[0011] In one embodiment, the preset operations include at least one or more of visual installation guidance and component review, component information display, component visual screening, construction progress simulation and comparison, and inspection record and management corresponding to the target electromechanical facility model.

[0012] In a second aspect, the embodiment of the present disclosure provides a BIM and AR-based human-machine collaborative management system for electromechanical facility installation, including: A display module is used for an AR device to display AR scene content corresponding to a building scene; the AR scene content includes at least a BIM model of a building and a three-dimensional scene model; wherein the three-dimensional scene model is generated by pre-collected point cloud data and texture mapping of a target area of ​​the building scene; a detection module, configured to detect an input operation of a user through the AR device, wherein the input operation is related to at least one electromechanical facility model, and the at least one electromechanical facility model is a component model corresponding to one or more electromechanical facilities in the BIM model; An execution module is used to execute a preset operation on a target electromechanical facility model in response to the input operation, wherein the target electromechanical facility model is any one of the at least one electromechanical facility model.

[0013] In a third aspect, an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for human-machine collaborative management of electromechanical facility installation based on BIM and AR as described in any of the above embodiments is implemented.

[0014] In a fourth aspect, an embodiment of the present disclosure provides an electronic device, including: Processor; and Memory for storing computer programs; Wherein, the processor is configured to execute the BIM and AR-based human-machine collaborative management method for electromechanical facility installation described in any of the above embodiments by executing the computer program.

[0015] Compared with the prior art, the technical solution provided by the embodiments of the present disclosure has the following advantages: The BIM and AR-based human-machine collaborative management method and system for electromechanical facility installation provided by the embodiments of the present disclosure, wherein an AR device displays AR scene content corresponding to a building scene; the AR scene content includes at least a BIM model of the building and a three-dimensional scene model; wherein the three-dimensional scene model is generated by pre-collected point cloud data and texture mapping of a target area of ​​the building scene; an input operation of a user through the AR device is detected, wherein the input operation is related to at least one electromechanical facility model, and the at least one electromechanical facility model is a component model corresponding to one or more electromechanical facilities in the BIM model; and a preset operation is performed on a target electromechanical facility model in response to the input operation, wherein the target electromechanical facility model is any one of the at least one electromechanical facility model. In this embodiment, a fine and accurate three-dimensional scene model can be generated through the pre-collected point cloud data and texture map of the building scene. Combined with the BIM model of the building, fine and accurate AR scene content can be generated. Combined with the AR device display, the electromechanical facility model in the interactive operation AR scene content can support high-precision and dynamic construction management of electromechanical facilities, which is suitable for complex building scenes. For example, it can realize accurate installation guidance and human-machine collaborative management of electromechanical facilities at the construction site, such as complex electromechanical facilities, such as avoiding errors and collisions during the installation process, thereby improving the construction management efficiency of electromechanical facilities, such as complex electromechanical facilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0017] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0018] Figure 1 This is a flow chart of a method for human-machine collaborative management of electromechanical facility installation based on BIM and AR in an embodiment of the present disclosure; Figure 2 This is a flow chart of a method for human-machine collaborative management of electromechanical facility installation based on BIM and AR according to another embodiment of the present disclosure; Figure 3A schematic diagram of a collaborative management system between multiple AR devices according to an embodiment of the present disclosure; Figure 4 A schematic diagram of a human-machine collaborative management system for installing electromechanical facilities based on BIM and AR according to an embodiment of the present disclosure; Figure 5 Schematic diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0019] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0020] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0021] It should be understood that, in the following, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0022] Figure 1 The flowchart of a method for man-machine collaborative management of electromechanical facility installation based on BIM and AR in an embodiment of the present disclosure is as follows. The method can be executed by an AR device. The method for man-machine collaborative management of electromechanical facility installation based on BIM and AR can specifically include the following steps: Step S101: The AR device displays AR scene content corresponding to the building scene; the AR scene content includes at least a BIM model and a three-dimensional scene model of the building; wherein the three-dimensional scene model is generated by pre-collected point cloud data and texture mapping of the target area of ​​the building scene.

[0023] Exemplarily, the AR device may be a head-mounted AR device or a handheld AR device. The acquisition of architectural scenes is a prerequisite for achieving AR effects. In this embodiment, a Matterport Pro 2 panoramic camera can be used to intelligently acquire three-dimensional scenes inside the building, generate high-resolution three-dimensional point cloud data and texture maps, and accurately generate three-dimensional scene models based on this, providing data support for AR applications such as AR scene content generation. The three-dimensional scene model not only contains the overall geometric information of the building, but also retains fine texture details, laying a solid foundation for subsequent virtual-reality fusion and augmented reality applications. The AR scene content can be implemented in, for example, a Unity environment, and the AR scene content may include at least a BIM model and a three-dimensional scene model of the building. The BIM model can be placed in a position in the three-dimensional scene model that matches the position in the actual physical scene.

[0024] Step S102: Detecting an input operation of a user through the AR device, where the input operation is related to at least one electromechanical facility model, and the at least one electromechanical facility model is a component model corresponding to one or more electromechanical facilities in the BIM model.

[0025] Exemplarily, AR devices can provide a user interface (UI) for human-computer interaction. For example, for handheld AR devices, i.e. mobile AR devices (such as iPad), Unity 2020.3.35 version and ARKit technology can be used to develop a 2D interactive interface based on touch screen operation, but it is not limited to this. For head-mounted AR devices (such as Microsoft HoloLens 2), Unity 2021.3.30 version and MRTK 2.8.3 toolkit can be used to develop a 3D interactive interface that supports hand rays, press interaction, and voice command operations. Regardless of the device, interactive operations can be performed, such as dynamically adjusting the position, size ratio, and direction of the electromechanical facility model through Unity's Transform API to ensure the superposition effect of the virtual component model in the AR scene.

[0026] Step S103: executing a preset operation on a target electromechanical facility model in response to the input operation, wherein the target electromechanical facility model is any one of the at least one electromechanical facility model.

[0027] For example, when a user such as a construction worker or manager performs an input operation through the user interface of an AR device, such as selecting or viewing a certain electromechanical facility model, a preset operation is performed on the corresponding target electromechanical facility model, such as displaying the model's attribute information, such as category, construction period, and floor, etc., which can be dynamically displayed through Unity's UI interface. Mobile AR devices can use a floating 2D panel display, while head-mounted AR devices can be implemented through a 3D floating window. This function can simplify the process of construction workers obtaining component model information of electromechanical facilities and improve on-site operation efficiency.

[0028] In this embodiment, a fine and accurate three-dimensional scene model can be generated through the pre-collected point cloud data and texture map of the building scene. Combined with the BIM model of the building, fine and accurate AR scene content can be generated. Combined with the AR device display, the electromechanical facility model in the interactive operation AR scene content can support high-precision and dynamic construction management of electromechanical facilities, which is suitable for complex building scenes. For example, it can realize accurate installation guidance and human-machine collaborative management of electromechanical facilities at the construction site, such as complex electromechanical facilities, such as avoiding errors and collisions during the installation process, thereby improving the construction management efficiency of electromechanical facilities, such as complex electromechanical facilities.

[0029] In one embodiment, the three-dimensional scene model is generated based on point cloud data and texture maps generated by a panoramic camera collecting data at multiple sites. The panoramic camera includes a depth sensor and at least one camera. The layout positions and collection order of the multiple sites are obtained by performing path planning processing on the spatial information of the building based on a path planning algorithm. The spatial information of the building is obtained based on the BIM model of the building.

[0030] In this embodiment, before the AR scene is generated, a set of intelligent site planning solutions based on path planning is designed based on the geometric information of the BIM model and the acquisition requirements of the panoramic camera equipment. First, the spatial information of the building, including the room layout, floor distribution, and obstacle location, is extracted from the BIM model to determine the boundary and internal structure of the area to be collected. Based on the effective acquisition range of the Matterport Pro 2 panoramic camera (usually 3 to 5 meters) and the acquisition accuracy requirements, the site spacing is set to 1.5 to 2.5 meters in this embodiment to balance data coverage and acquisition efficiency. In order to realize the intelligence of the acquisition path, the path planning algorithm in this embodiment uses the A* (A-star) path planning algorithm to plan the path of the building space, automatically generates the layout position and acquisition order of the site according to the complexity of the building, and provides guidance for the acquisition operation. Afterwards, in each acquisition operation, the Matterport Pro 2 panoramic camera performs a 360-degree full-range scan at the site location, and generates high-resolution point clouds and texture maps through synchronous shooting of multiple cameras and data acquisition of depth sensors. In some other embodiments, after the acquisition is completed, the operator can mark the key feature positions (such as windows, openings and column corners) in the building. These key geometric feature points can be used as important references in the subsequent BIM model alignment process to improve the accuracy of model alignment.

[0031] When data collection for all sites is completed, the system can automatically stitch the point cloud data of each site into a complete 3D scene model based on Matterport's own stitching algorithm. The stitched 3D scene model not only contains the overall geometric information of the building, but also retains fine texture details, providing a reliable basis for the geometric alignment of the BIM model.

[0032] The above-mentioned building scene collection scheme in this embodiment realizes comprehensive coverage and efficient collection, which not only optimizes the collection efficiency, but also significantly improves the accuracy and consistency of the data, making the subsequently generated AR scene content more detailed and accurate, and providing key technical support for the management and construction of complex electromechanical facilities. The scheme of this embodiment can support higher-precision and dynamic construction management of electromechanical facilities, which is suitable for complex building scenes. For example, it can realize accurate installation guidance and human-machine collaborative management of electromechanical facilities at the construction site, such as complex electromechanical facilities, such as avoiding errors and collisions during the installation process, thereby improving the construction management efficiency of electromechanical facilities, such as complex electromechanical facilities.

[0033] In one embodiment, in conjunction with reference Figure 2 As shown in , the method may also include the following steps: Step S201: capturing a plurality of key geometric feature points in the three-dimensional scene model, and storing the plurality of key geometric feature points and the texture map in a first database; Step S202: obtaining a plurality of key geometric feature points of the BIM model, obtaining a plurality of key geometric feature points of the three-dimensional scene model from the first database, and calculating a transformation matrix based on the plurality of key geometric feature points of the BIM model and the plurality of key geometric feature points of the three-dimensional scene model, wherein the transformation matrix includes translation, rotation and scaling information of the BIM model; Step S203: adjusting the position, size ratio and direction of the BIM model in the three-dimensional scene model based on the transformation matrix to achieve accurate alignment with the scene.

[0034] For example, in a complex building environment, the matching of geometric and semantic information of the electromechanical facility model (part of the component model in the BIM model) is a key link in the application of augmented reality. The matching of geometric information can ensure that the virtual component model is highly consistent with the actual scene in terms of spatial position and shape, while the matching of semantic information gives each component detailed attribute data to support dynamic interaction and construction management.

[0035] After the above-mentioned building scene acquisition is completed, the generated 3D point cloud data and texture map can be imported into Vuforia's Area Target tool for processing. By capturing significant geometric features in the 3D scene model, such as key geometric feature points, a scene map database containing feature information, namely the first database, is generated. This database not only saves the building texture map, but also records the significant geometric features in the scene, which becomes the basis for subsequent matching algorithms and scene recognition. In the Unity environment, the generated scene feature map database can be loaded using Vuforia's SDK, and then the physical environment scene of the building site can be scanned in real time through the AR device to capture the geometric features in the scene, and then compared with the geometric features in the feature map database, and the position of the 3D scene model in the physical environment scene can be adjusted based on the comparison results. This process ensures that the virtual model can achieve high-precision positioning with the physical scene, providing conditions for geometric alignment.

[0036] The intelligent alignment of geometric information relies on key reference points, namely key geometric feature points, in the BIM model and feature map database. First, three significant key geometric feature points (such as columns or wall corners) are selected from the BIM model and feature map database respectively, and their three-dimensional coordinates are recorded. Subsequently, based on the coordinate relationship of the selected key geometric feature points, the transformation matrix is ​​calculated by the matching algorithm, which contains the translation, rotation and scaling information of the BIM model. By applying the transformation matrix to the BIM model, the position, size ratio and direction of the BIM model are automatically adjusted to achieve precise alignment with the scene. This intelligent alignment method significantly improves the efficiency and accuracy of aligning the model with the real environment through automatic recognition of scene features. It is more efficient than traditional manual adjustment and can adapt to changes in architectural geometry in complex environments. In this way, more detailed and accurate AR scene content related to electromechanical facilities can be generated, providing key technical support for the management and construction of complex electromechanical facilities, so that the solution of this embodiment can support higher-precision and dynamic construction management of electromechanical facilities, which is suitable for complex building scenes. For example, it can achieve accurate installation guidance and human-machine collaborative management of electromechanical facilities such as complex electromechanical facilities at the construction site, such as avoiding errors and collisions during the installation process, thereby improving the construction management efficiency of electromechanical facilities such as complex electromechanical facilities.

[0037] In one embodiment, the method may further include the following steps: Acquire semantic information of each component in the BIM model, store the semantic information of each component in a target database in the form of multiple fields, and introduce a timestamp field to record the latest update time of each piece of data; During the construction management process, when it is detected that a user changes the component information in the BIM model through the AR device, a pre-written script is used to automatically update the data to the target database, wherein each time the data is updated, the target database automatically refreshes the timestamp field of the corresponding component to mark the latest status; After the BIM model is loaded in the AR scene content, the semantic information of the corresponding component is queried from the target database through the unique ID of the component, and the semantic information of the component obtained by the query is bound to the corresponding component model.

[0038] Exemplarily, after the alignment of the above-mentioned model based on geometric information and the scene is completed, the matching of semantic information provides rich attribute data for the virtual model in augmented reality. This process can first extract the semantic information of the component from the BIM model through Dynamo. The semantic information includes key attribute information such as the unique ID, category, construction phase and floor of each component. The extracted data is exported in a structured manner and stored in the target database such as MySQL database. When designing the database, hierarchical logic is used to divide the semantic information into multiple fields, such as ComponentID, Category, ConstructionPhase and Floor, and a timestamp field is introduced to record the latest update time of each data. This structure not only supports the storage of initial semantic information, but also provides a basis for the subsequent dynamic update of data.

[0039] During the construction process, when the component information in the BIM model changes (such as construction phase or floor adjustment), Dynamo is used to write scripts to implement the automated process of updating data to the MySQL database. Each time the data is updated, the database automatically refreshes the timestamp field of the corresponding component to mark the latest status. In this way, the semantic information in the database is always kept consistent with the actual construction site, providing a reliable guarantee for the real-time data in augmented reality.

[0040] After loading the BIM model in Unity, the geometric information is bound to the semantic information through the unique ID of the component. In the Unity scene, each component's name field contains its ComponentID. The script will traverse all components and parse the name field to extract the corresponding ID. Then, the script queries the database for matching semantic information based on the component ID through the integrated MySQL database interface. For example, for a component with a component ID of ID123, the script will retrieve its attribute data such as category, floor, and construction stage.

[0041] After querying the semantic information, the system dynamically generates a semantic information component for each component in Unity and binds the retrieved attribute data to the component fields. For example, the semantic information component of a pipe component will contain its ID, category, construction stage, and floor information. These components are embedded in the corresponding geometric model, i.e., the component model, so that each virtual component model can carry comprehensive attribute data.

[0042] By binding the semantic information component, the user can dynamically view its detailed properties by clicking or selecting the component in the augmented reality scene. Unity's UI system will call the bound semantic information component and present the attribute data in a visual form. This binding method ensures the integration of semantic information and geometric models, and at the same time, through the real-time update of the MySQL database, enhances the dynamic adaptability of the virtual component model to the actual construction scene. The dual registration of geometric and semantic information makes the model in the augmented reality scene not only accurate in spatial position, but also supports real-time interaction and information management during the construction process, providing comprehensive technical support for the construction guidance and on-site management of complex electromechanical facilities, so that the solution of this embodiment can support higher-precision and dynamic construction management of electromechanical facilities, suitable for complex building scenes, for example, it can realize accurate installation guidance and human-machine collaborative management of electromechanical facilities such as complex electromechanical facilities at the construction site, such as avoiding errors and collisions during the installation process, thereby improving the construction management efficiency of electromechanical facilities such as complex electromechanical facilities.

[0043] In one embodiment, the AR device includes at least a head-mounted AR device and a handheld AR device; the handheld AR device provides a 2D AR interaction interface and supports a touch screen interaction mode; the head-mounted AR device provides a 3D AR interaction interface and supports one or more of a hand ray interaction mode, a press interaction mode, and a voice command interaction mode; the virtual scene space sharing between the head-mounted AR device and the handheld AR device is achieved through the Azure Spatial Anchors (ASA) technology; the real-time communication between the head-mounted AR device and the handheld AR device is achieved through the PUN (PhotonUnity Networking) technology.

[0044] This embodiment proposes a multi-terminal human-computer interaction solution based on augmented reality (AR) technology, which realizes efficient management of complex electromechanical facilities through AR devices (such as iPad and HoloLens 2). The solution covers functions such as visual installation guidance and component review, component information display, component visual screening, construction progress simulation and comparison, inspection record and management. For different terminal devices, adaptive interactive interfaces and operation logic are developed respectively, and combined with the fusion technology of BIM model geometry and semantic information, high-precision and dynamic construction management is supported.

[0045] In order to achieve efficient construction management of complex electromechanical facilities, this embodiment introduces head-mounted AR devices (such as HoloLens 2) and handheld AR devices (such as iPad) to build a multi-terminal device human-machine collaborative management system. By integrating AR perception interaction, spatial sharing and real-time communication collaboration of multiple terminal devices, and real-time synchronization and management of inspection data, the problems of isolated information and poor communication at the construction site are solved, and management efficiency is significantly improved.

[0046] Specifically, refer to Figure 3 As shown in the figure, different devices provide adaptive interaction modes in terms of interaction methods. HoloLens 2 supports a variety of immersive operations, including hand ray interaction, press interaction and voice command interaction. The hand ray function uses ray projection to select model components, which is suitable for long-distance operation; the press interaction improves the intuitiveness and accuracy of the operation through the depth feedback of the virtual button; the voice command interaction allows users to complete tasks such as information query and status marking through voice, which is particularly suitable for scenarios where both hands are occupied. The iPad adopts a touch-screen-based interaction method. Users can realize the human-computer interaction function of the model through touch, and provide a simple 2D UI interface for quickly filtering and viewing component information. The different designs of the two devices in terms of interaction methods meet the needs of different users such as managers and construction workers and scenarios.

[0047] Spatial sharing of multiple terminal devices relies on Azure Spatial Anchors (ASA) technology. HoloLens 2 can create and upload spatial anchors through ASA, which record key feature points in the physical environment, such as the three-dimensional coordinates of pillars or corners. iPad synchronizes with ASA cloud through ARKit, downloads anchor data and aligns local scenes to achieve the same virtual model position as HoloLens 2. In this way, different devices can load consistent virtual scenes in their respective physical environments to ensure the uniformity of spatial understanding. In addition, ASA's cross-platform capabilities make the sharing of spatial anchors between HoloLens 2 and iPad efficient and stable, providing a solid foundation for collaborative operations.

[0048] Communication collaboration is achieved through PUN (Photon Unity Networking). PUN is responsible for real-time communication between multiple devices, synchronizing the user's operating status on any device (such as model position adjustment, inspection record update) to other devices. The communication collaboration mechanism supports status broadcasting and event synchronization. For example, when a user completes the status marking of a component on HoloLens 2, the iPad will instantly update the display status of the component to ensure that all devices have a consistent understanding of the on-site status. In addition, the combination of PUN and ASA further enhances the collaboration of multiple terminal devices in space sharing and operation feedback. Therefore, the above scheme in this embodiment can realize space sharing and data synchronization between multiple AR devices such as HoloLens 2 and iPad, improve the efficiency of multi-user collaboration, and provide strong support for complex construction scenes, thereby improving the construction management efficiency and collaboration level of electromechanical facilities such as complex electromechanical facilities.

[0049] In one embodiment, the user operation results on the head-mounted AR device and the handheld AR device are updated to the target database in real time, and the user operation results are synchronized with each other and displayed in the respective AR scene contents in a dynamic rendering manner.

[0050] Exemplarily, the target database is the above-mentioned MySQL database. The management of inspection data is implemented based on the above-mentioned MySQL database, which supports real-time updating and synchronization of data among multiple devices. The operation results completed by the user on any terminal device, such as inspection results, including component status, problem description and time records, will be uploaded to the database immediately. Other devices obtain the latest data through scheduled queries or event triggers to ensure that all inspection information remains consistent. This database not only supports real-time entry and query, but also stores historical inspection records for construction progress tracking and problem analysis. The operation results on HoloLens 2 and iPad will be reflected in the database in real time, and displayed in the AR scene through dynamic rendering, forming a two-way connection between data and scenes.

[0051] Through this multi-terminal device collaborative management, on-site construction management has achieved a closed loop of data sharing, operation synchronization, and real-time feedback. HoloLens 2 provides an immersive three-dimensional experience, suitable for scenarios that require precise positioning and complex operations; iPad, with its portability and flexibility, meets the needs of mobile operations. This collaborative application of multi-terminal AR devices not only improves the communication efficiency of on-site construction, but also ensures the efficiency, accuracy, and reliability of complex electromechanical facility management.

[0052] In one embodiment, the preset operation includes at least one or more of the following: visual installation guidance and component review, component information display, component visual screening, construction progress simulation and comparison, and inspection record and management corresponding to the target electromechanical facility model. The following describes the above specific preset operations in detail: Visual installation guidance and component review: This function provides intuitive installation guidance and hidden component inspection support for construction personnel through virtual overlay and transparent display technology. In Unity, based on the geometric alignment results of the BIM model, the system dynamically loads the component model and marks its key points (such as pipe interfaces and hole locations). For hidden components, the system uses transparent materials or cutaway views to display their internal structure, and combines the X-ray detection function to highlight the components selected by the user to assist construction personnel in confirming their location and installation status.

[0053] Component information display: This function is realized through the interaction between Unity's ray detection function and the database. When the user selects a component through the touch screen, hand ray or voice, the system will extract its unique ID and send a query request to the MySQL database to retrieve the detailed attribute data of the component (such as category, construction period, and floor to which it belongs). The search results are dynamically displayed through the Unity UI interface. This function simplifies the process of construction personnel obtaining component information and improves on-site operation efficiency.

[0054] Component visual screening: This function helps construction workers quickly locate specific types of components. The system supports screening operations based on attributes such as category, floor, and room. After the user selects the screening conditions through the menu, the Unity script sends a screening request to the database and dynamically adjusts the display status of the components based on the returned results. The screened components are displayed in a highlighted or unselected manner to reduce visual interference and help construction workers focus on key areas.

[0055] Construction progress simulation: This function is implemented based on the construction period field of the BIM model. The user sets the target date in the AR scene, and the system loads the component model that matches the target date by querying the construction period field in the database. Through material coloring or transparency processing, components that are not completed as planned are highlighted to help managers intuitively compare the plan with the actual progress. Unity's MaterialPropertyBlock function can be used to dynamically change the color of components to distinguish between completed and unfinished states.

[0056] Inspection records and management: This function is realized through the real-time update and synchronization function of the MySQL database. In the AR scene, after the user completes the inspection operation, he can directly enter the inspection results (including problem description, component status, etc.). The system stores the entered data in MySQL through the database interface and generates an update timestamp. The inspection records are shared synchronously through multiple terminal devices, and construction personnel and management personnel can access the latest data at any time to ensure that problems are fed back and resolved in a timely manner.

[0057] The above-mentioned scheme of this embodiment can provide AR human-computer interaction applications such as component installation guidance, hidden component review, component display and visual screening, and construction progress simulation, which can meet the diverse needs of complex electromechanical facility installation, thereby significantly improving the management efficiency of electromechanical facility installation. It should be noted that, although the steps of the method in the present disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step for execution, and / or one step may be decomposed into multiple steps for execution, etc. In addition, it is also easy to understand that these steps may be, for example, executed synchronously or asynchronously in multiple modules / processes / threads.

[0058] like Figure 4 As shown, the embodiment of the present disclosure provides a BIM and AR-based human-machine collaborative management system for electromechanical facility installation, which can be applied to AR equipment, including: Display module 401, used for the AR device to display AR scene content corresponding to the building scene; the AR scene content at least includes a BIM model of the building and a three-dimensional scene model; wherein the three-dimensional scene model is generated by pre-collected point cloud data and texture mapping of the target area of ​​the building scene; A detection module 402 is used to detect an input operation of a user through the AR device, where the input operation is related to at least one electromechanical facility model, and the at least one electromechanical facility model is a component model corresponding to one or more electromechanical facilities in the BIM model; The execution module 403 is used to execute a preset operation on a target electromechanical facility model in response to the input operation, and the target electromechanical facility model is any one of the at least one electromechanical facility model.

[0059] In one embodiment, the three-dimensional scene model is generated based on point cloud data and texture maps generated by a panoramic camera collecting data at multiple sites. The panoramic camera includes a depth sensor and at least one camera. The layout positions and collection order of the multiple sites are obtained by performing path planning processing on the spatial information of the building based on a path planning algorithm. The spatial information of the building is obtained based on the BIM model of the building.

[0060] In one embodiment, the system may further include a geometric information alignment module, for: Capturing a plurality of key geometric feature points in the three-dimensional scene model, and storing the plurality of key geometric feature points and the texture map in a first database; Acquire multiple key geometric feature points of the BIM model, acquire multiple key geometric feature points of the three-dimensional scene model from the first database, and calculate a transformation matrix based on the multiple key geometric feature points of the BIM model and the multiple key geometric feature points of the three-dimensional scene model, wherein the transformation matrix includes translation, rotation, and scaling information of the BIM model; The position, size ratio and direction of the BIM model in the three-dimensional scene model are adjusted based on the transformation matrix to achieve accurate alignment with the scene.

[0061] In one embodiment, the system may further include a semantic information binding module for: Acquire semantic information of each component in the BIM model, store the semantic information of each component in a target database in the form of multiple fields, and introduce a timestamp field to record the latest update time of each piece of data; During the construction management process, when it is detected that a user changes the component information in the BIM model through the AR device, a pre-written script is used to automatically update the data to the target database, wherein each time the data is updated, the target database automatically refreshes the timestamp field of the corresponding component to mark the latest status; After the BIM model is loaded in the AR scene content, the semantic information of the corresponding component is queried from the target database through the unique ID of the component, and the semantic information of the component obtained by the query is bound to the corresponding component model.

[0062] In one embodiment, the AR device includes at least a head-mounted AR device and a handheld AR device; the handheld AR device provides a 2D AR interaction interface and supports a touch screen interaction mode; the head-mounted AR device provides a 3D AR interaction interface and supports one or more of a hand ray interaction mode, a press interaction mode, and a voice command interaction mode; the virtual scene space sharing between the head-mounted AR device and the handheld AR device is achieved through the Azure Spatial Anchors (ASA) technology; the real-time communication between the head-mounted AR device and the handheld AR device is achieved through the PUN (PhotonUnity Networking) technology.

[0063] In one embodiment, the user operation results on the head-mounted AR device and the handheld AR device are updated to the target database in real time, and the user operation results are synchronized with each other and displayed in the respective AR scene contents in a dynamic rendering manner.

[0064] In one embodiment, the preset operations include at least one or more of visual installation guidance and component review, component information display, component visual screening, construction progress simulation and comparison, and inspection record and management corresponding to the target electromechanical facility model.

[0065] Regarding the system in the above embodiment, the specific manner in which each module performs operations and the corresponding technical effects brought about have been described in detail in the embodiment of the method, and will not be elaborated here.

[0066] It should be noted that although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiment of the present disclosure, the features and functions of two or more modules or units described above can be concretized in one module or unit. Conversely, the features and functions of a module or unit described above can be further divided into multiple modules or units for concretization. The components displayed as modules or units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the disclosed solution. Those of ordinary skill in the art can understand and implement it without paying creative work.

[0067] The embodiments of the present disclosure also provide a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for human-machine collaborative management of electromechanical facility installation based on BIM and AR according to any of the above embodiments is implemented.

[0068] Exemplarily, the readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0069] The computer readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by an instruction execution system, an apparatus, or a device or used in combination with it. The program code contained on the readable storage medium may be transmitted with any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0070] The present disclosure also provides an electronic device, including a processor and a memory, wherein the memory is used to store a computer program, wherein the processor is configured to execute the BIM and AR-based electromechanical facility installation human-machine collaborative management method in any of the above embodiments by executing the computer program.

[0071] Refer to the following Figure 5 The electronic device 600 according to this embodiment of the present invention is described. Figure 5 The electronic device 600 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0072] like Figure 5 As shown, the electronic device 600 is in the form of a general computing device. The components of the electronic device 600 may include but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different system components (including the storage unit 620 and the processing unit 610), a display unit 640, etc.

[0073] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 performs the steps according to various exemplary embodiments of the present invention described in the above method embodiment section of this specification. For example, the processing unit 610 can perform the following steps: Figure 1 or Figure 2The steps of the method shown in .

[0074] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .

[0075] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each of which or some combination may include the implementation of a network environment.

[0076] Bus 630 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0077] The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 600, and / or communicate with any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface 650. In addition, the electronic device 600 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) through a network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 through a bus 630. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0078] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software, or by combining software with necessary hardware. Therefore, the technical solution according to the implementation methods of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes a number of instructions to enable a computing device (which can be a personal computer, a server, or a network device, etc.) to execute the method steps of the above-mentioned embodiments according to the implementation methods of the present disclosure.

[0079] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0080] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A human-machine collaborative management method for electromechanical facility installation based on BIM and AR, characterized in that: include: The AR device displays AR scene content corresponding to the building scene; the AR scene content includes at least a BIM model of the building and a three-dimensional scene model; wherein the three-dimensional scene model is generated by pre-collected point cloud data and texture mapping of the target area of ​​the building scene; Detecting an input operation of a user through the AR device, wherein the input operation is related to at least one electromechanical facility model, and the at least one electromechanical facility model is a component model corresponding to one or more electromechanical facilities in the BIM model; A preset operation on a target electromechanical facility model is performed in response to the input operation, the target electromechanical facility model being any one of the at least one electromechanical facility model.

2. The method according to claim 1, characterized in that: The three-dimensional scene model is generated based on point cloud data and texture maps generated by a panoramic camera collecting data at multiple sites. The panoramic camera includes a depth sensor and at least one camera. The layout positions and collection order of the multiple sites are obtained by performing path planning processing on the spatial information of the building based on a path planning algorithm. The spatial information of the building is obtained based on the BIM model of the building.

3. The method according to claim 1, characterized in that: The method further includes: Capturing a plurality of key geometric feature points in the three-dimensional scene model, and storing the plurality of key geometric feature points and the texture map in a first database; Acquire multiple key geometric feature points of the BIM model, acquire multiple key geometric feature points of the three-dimensional scene model from the first database, and calculate a transformation matrix based on the multiple key geometric feature points of the BIM model and the multiple key geometric feature points of the three-dimensional scene model, wherein the transformation matrix includes translation, rotation, and scaling information of the BIM model; The position, size ratio and direction of the BIM model in the three-dimensional scene model are adjusted based on the transformation matrix to achieve accurate alignment with the scene.

4. The method according to claim 1, characterized in that The method further includes: Acquire semantic information of each component in the BIM model, store the semantic information of each component in a target database in the form of multiple fields, and introduce a timestamp field to record the latest update time of each piece of data; During the construction management process, when it is detected that a user changes the component information in the BIM model through the AR device, a pre-written script is used to automatically update the data to the target database, wherein each time the data is updated, the target database automatically refreshes the timestamp field of the corresponding component to mark the latest status; After the BIM model is loaded in the AR scene content, the semantic information of the corresponding component is queried from the target database through the unique ID of the component, and the semantic information of the component obtained by the query is bound to the corresponding component model.

5. The method according to claim 4, characterized in that The AR device includes at least a head-mounted AR device and a handheld AR device; the handheld AR device provides a 2D AR interaction interface and supports touch screen interaction; the head-mounted AR device provides a 3D AR interaction interface and supports one or more of hand ray interaction, press interaction and voice command interaction; the virtual scene space sharing between the head-mounted AR device and the handheld AR device is achieved through the AzureSpatial Anchors (ASA) technology; the real-time communication between the head-mounted AR device and the handheld AR device is achieved through the PUN (Photon Unity Networking) technology.

6. The method according to claim 5, characterized in that The user operation results on the head-mounted AR device and the handheld AR device are updated to the target database in real time, and the user operation results are synchronized with each other and displayed in the respective AR scene contents in a dynamic rendering manner.

7. The method according to any one of claims 1 to 6, characterized in that: The preset operations include at least one or more of visual installation guidance and component review, component information display, component visual screening, construction progress simulation and comparison, and inspection record and management corresponding to the target electromechanical facility model.

8. A BIM and AR-based human-machine collaborative management system for electromechanical facility installation, characterized in that: include: A display module is used for an AR device to display AR scene content corresponding to a building scene; the AR scene content includes at least a BIM model of a building and a three-dimensional scene model; wherein the three-dimensional scene model is generated by pre-collected point cloud data and texture mapping of a target area of ​​the building scene; a detection module, configured to detect an input operation of a user through the AR device, wherein the input operation is related to at least one electromechanical facility model, and the at least one electromechanical facility model is a component model corresponding to one or more electromechanical facilities in the BIM model; An execution module is used to execute a preset operation on a target electromechanical facility model in response to the input operation, wherein the target electromechanical facility model is any one of the at least one electromechanical facility model.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the human-machine collaborative management method for electromechanical facility installation based on BIM and AR as described in any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: include: processor; as well as Memory for storing computer programs; Wherein, the processor is configured to execute the BIM and AR-based human-machine collaborative management method for electromechanical facility installation as described in any one of claims 1 to 7 by executing the computer program.

Images