A method for architectural in-situ information retrieval based on visual gaze

Abstract

The present invention discloses a method for in-situ building information retrieval based on visual gaze, including: extracting and optimizing building information models; acquiring gaze data; interacting gaze data with building models; compiling projects in Visual Studio and publishing them to HoloLens 2; aligning BIM (Building Information Model) models with physical structures of physical buildings through spatial anchor point technology; and performing on-site building information retrieval by wearing a HoloLens 2 device. The present invention proposes a method for visual gaze retrieval of building information based on mixed reality technology, which uses real building components as carriers. Mixed reality technology is used to link and bind digital information of building components with physical entities of building components; eye tracking technology is used to perform intuitive information retrieval of "what you see is what you get" in-situ on building components. The present invention enables users to quickly obtain relevant building information, such as material properties and design parameters, by gazing at real building components, thereby improving the efficiency of on-site work and bringing significant improvements and convenience to the construction industry.

Description

Technical Field

[0001] The present invention relates to the fields of civil engineering and intelligent construction, and in particular to a method for retrieving building in-situ information based on visual gaze. Background Art

[0002] Building Information Modeling (BIM) technology is at the core of the modern construction industry and enables efficient planning, design, construction and management of construction projects. BIM creates detailed digital models that enable construction professionals to simulate and analyze all aspects of a building throughout its life cycle. However, traditional information retrieval methods require consulting paper drawings or digital files on mobile devices to obtain building information. This process requires frequent searching and comparing drawings, which is often time-consuming and inefficient. In addition, information integration is not intuitive, and it is difficult to intuitively combine drawing information with the actual situation on site, which increases the difficulty of understanding and applying information. Although BIM technology can visualize two-dimensional drawings in three dimensions and enhance the understanding of drawing information, BIM technology faces several challenges in actual on-site applications: (1) Traditional BIM technology relies on desktop software, which is often limited by the performance and portability of mobile devices when applied on actual construction sites. (2) BIM technology requires users to have high professional skills, and for large or complex construction projects, it may take a lot of time to find specific information among many components based on the software interface alone. (3) There are still challenges in effectively combining BIM models with the physical environment of the actual construction site.

[0003] The introduction of mixed reality (MR) technology provides a new solution to overcome these challenges. As an emerging technology in the field of physical and digital interaction, MR technology provides a way to combine the real and virtual worlds to create a mixed reality space that combines a person, a computer, and an environment. This space enables people to enhance their understanding of the real environment with the help of computer digital information. In this space, engineers and construction workers can intuitively see the integration of digital models and the actual building environment.

[0004] The introduction of eye tracking technology has brought breakthrough improvements to MR technology in BIM applications. This technology allows users to interact with BIM models by moving their eyes, providing a more intuitive and efficient way to retrieve information. Through eye tracking, users can directly gaze at a specific building component, and the system will then display detailed information about the component, such as material properties, design parameters, etc., greatly improving the efficiency and accuracy of information acquisition. The application of this technology not only improves the practicality of MR in the fields of architectural design, construction, and operation and maintenance, but also brings users a more natural and immersive interactive experience. Summary of the invention

[0005] In view of the above problems, the present invention aims to provide a method for retrieval of architectural in-situ information based on visual gaze, which comprises the following steps:

[0006] Step 1: Extract and export the BIM model in Autodesk Revit to FBX format, optimize the model to reduce the number of triangle meshes, then configure the mixed reality development environment in Unity and import the optimized model. Based on the MRTK toolkit, realize the functions of grabbing, moving, rotating and scaling the BIM model.

[0007] Step 2: Write C# code to initialize the eye tracker in the MRTK toolkit, write code to determine the validity of eye movement data, obtain the user's gaze position, and access related timestamps;

[0008] Step 3: In Unity, create an information panel for the BIM model component and set it to an initially invisible state, which is activated only when the user gazes at it. Use an algorithm to monitor the user's gaze point, activate and position the panel based on the interaction threshold, and add a script to automatically adjust the panel orientation to optimize the interaction experience;

[0009] Step 4, Visual Studio compiles the project and publishes it to HoloLens 2;

[0010] Step 5: Align the BIM model with the physical entity of the building component through spatial anchor technology;

[0011] Step 6: Wearing the HoloLens 2 device, when the user gazes at a building component, the system automatically identifies the gaze target and displays detailed information of the component in the user's field of view, such as material properties, size specifications, etc., to achieve in-situ building information retrieval based on visual gaze.

[0012] Further, in step 1, the following steps may be performed in sequence:

[0013] Step 1.1: Use Autodesk Revit as the main BIM software to extract building design and structural information, including detailed components and material properties; export the BIM model in Revit to FBX format;

[0014] Step 1.2: Optimize the model exported from Revit to reduce the number of triangle meshes;

[0015] Step 1.3: Configure the mixed reality development environment in Unity and import the building model information into Unity. Based on the MRTK toolkit, mount "NearInteractionGrabbable.cs", "ManipulationHandler.cs", "BoundingBox.cs" and "Interactable.cs" on the BIM model in sequence to realize the functions of grabbing, moving, rotating and scaling the BIM model.

[0016] Further, in step 2, the following steps may be performed in sequence:

[0017] Step 2.1: Activate and configure the MRTK eye tracker;

[0018] Step 2.2: Write C# code in Visual Studio to determine the validity of eye movement data, obtain gaze point location data, and obtain timestamps to keep data synchronized.

[0019] Further, in step 3, the following steps may be performed in sequence:

[0020] Step 3.1: In Unity, create an information panel for each widget and write C# code to make the information panel active only when staring at it.

[0021] Step 3.2: Use Unity's "Update" method to monitor the user's gaze point in each frame, and use a self-developed algorithm to process the gaze data in real time to determine object selection and switching;

[0022] Step 3.3: In Unity, add a "BoxCollider" collision body to the model component, perform collision detection through the self-developed algorithm, and activate the corresponding information panel when a collision occurs;

[0023] Step 3.4: In Unity, use a self-developed algorithm to dynamically position the information panel next to the gaze point and adjust it to a suitable viewing size;

[0024] Step 3.5: Optimize the information panel display logic in Unity so that the panel always faces the user.

[0025] Further, in step 4, the project is packaged into a Visual Studio Solution file, the project is compiled into the ARM64 format using Visual Studio, and published into an APPX format installation package. The project is deployed to HoloLens 2 via a Wi-Fi connection.

[0026] Further, in step 5, the following steps may be performed in sequence:

[0027] Step 5.1: Use the grab, move, rotate, and scale functions added in step 1.3 to align the BIM model with the physical entity of the building component. Use the MRTK toolkit combined with the spatial mapping of Hololens 2 to create a spatial anchor for the BIM model and lock the BIM model in space to link and bind the digital information of the building component with the physical entity of the building component.

[0028] Step 5.2: Verify the alignment accuracy of the model with the physical entity of the building component through field testing, with a deviation of less than 1 cm.

[0029] Furthermore, in step 6, when the user wears the HoloLens 2 device at the construction site, the system can recognize and respond to the user's gaze point in real time. When the user gazes at the building entity, the system immediately displays the corresponding model information based on the gaze point, such as material properties and design parameters.

[0030] The advantages and positive effects of the present invention are:

[0031] (1) Innovative application of mixed reality in civil engineering. This paper combines the MRTK toolkit with an independently developed algorithm and uses eye tracking technology to retrieve building information in the field of civil engineering. This innovative application combines mixed reality technology with civil engineering, opens up a new field of information retrieval and building model interaction, and improves the efficiency and accuracy of information acquisition in the process of building design and construction.

[0032] (2) Real-time “what you see is what you get” information retrieval and in-situ display. This invention realizes the “what you see is what you get” interactive experience through eye tracking technology and the computing power of HoloLens 2. When the user gazes at a specific building component in real space, the system can instantly identify the user's gaze focus and display relevant detailed attribute information, such as material properties, design parameters, etc., in situ at the corresponding location. This intuitive information retrieval method not only greatly improves the speed and accuracy of data acquisition, but also enhances the user's cognitive efficiency and immersion, and optimizes the interactivity and efficiency of on-site work.

[0033] (3) Improve the interactivity and efficiency of on-site work. The interactive mode of the present invention greatly improves the interactivity and efficiency of on-site work. The present invention solves the problems of operational complexity and low efficiency of information acquisition in on-site applications of traditional BIM technology, reduces the reliance on professional skills, and simplifies the cumbersome drawing review process. It eliminates the time delay and complexity of traditional methods in information retrieval and display, allowing users to obtain the required data while visually exploring the building environment.

[0034] In summary, the present invention realizes a method for in-situ information retrieval of buildings based on visual gaze by innovatively integrating eye tracking technology and mixed reality technology and applying them to the retrieval and interaction of building information. This method directly uses the user's gaze point to obtain detailed information of building components and displays them in real time in real space, greatly improving the efficiency, accuracy and natural interaction of information retrieval. During the construction process, especially in the construction acceptance stage, the application of the present invention can enable the construction team to accurately and quickly check and confirm the details of the project. In the building operation and maintenance stage, the present invention provides an efficient asset management tool for operation and maintenance personnel. Through eye tracking, the relevant information of key building components, such as maintenance records, performance data, etc., can be quickly accessed, and operation and maintenance management becomes more efficient and accurate. This direct and real-time information access method greatly improves the maintenance efficiency and effectiveness of building facilities. The application of the present invention not only optimizes the user's workflow, improves the interactivity and immersion of on-site work, but also demonstrates the huge potential and value of digital transformation in the construction industry, and provides strong support for the sustainable development of the civil engineering field. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The above and / or other aspects and advantages of the present invention will become clearer and easier to understand through the detailed description made in conjunction with the following drawings, which are only exemplary and do not limit the present invention, wherein:

[0036] Figure 1 It is a flow chart of the architectural in-situ information retrieval method based on visual gaze.

[0037] Figure 2 This is an example diagram of information extraction based on BIM technology.

[0038] Figure 3 This is an example diagram of BIM model optimization based on lightweight technology.

[0039] Figure 4 This is an example diagram of initializing the eye tracker in the self-developed algorithm.

[0040] Figure 5 This is an example diagram of the method of obtaining gaze data in the independently developed algorithm.

[0041] Figure 6 This is an example diagram of information panel status management in the self-developed algorithm.

[0042] Figure 7 This is an example diagram of real-time gaze point detection and processing in the independently developed algorithm.

[0043] Figure 8 This is an example of a collision body in the Unity engine.

[0044] Fig. 9This is an example diagram of aligning the BIM model with the building entity based on the spatial anchor point technology.

[0045] Fig.10 This is an example diagram of the architectural in-situ information retrieval method based on visual gaze. DETAILED DESCRIPTION

[0046] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0047] like Figure 1 As shown, the present invention provides a method for retrieval of architectural in-situ information based on visual gaze, the method comprising the following steps:

[0048] Step 1: Extract and export the BIM model in Autodesk Revit to FBX format, optimize the model to reduce the number of triangle meshes, then configure the mixed reality development environment in Unity and import the optimized model. Based on the MRTK toolkit, realize the functions of grabbing, moving, rotating and scaling the BIM model.

[0049] Step 2: Write C# code to initialize the eye tracker in the MRTK toolkit, write code to determine the validity of eye movement data, obtain the user's gaze position, and access related timestamps;

[0050] Step 3: In Unity, create an information panel for the BIM model component and set it to an initially invisible state, which is activated only when the user gazes at it. Use an algorithm to monitor the user's gaze point, activate and position the panel based on the interaction threshold, and add a script to automatically adjust the panel orientation to optimize the interaction experience;

[0051] Step 4, Visual Studio compiles the project and publishes it to HoloLens 2;

[0052] Step 5: Align the BIM model with the physical entity of the building component through spatial anchor technology;

[0053] Step 6: Wearing the HoloLens 2 device, when the user gazes at a building component, the system automatically identifies the gaze target and displays detailed information of the component in the user's field of view, such as material properties, size specifications, etc., to achieve in-situ building information retrieval based on visual gaze.

[0054] Further, in step 1, the following steps may be performed in sequence:

[0055] Step 1.1: Export the required component information from Revit, including but not limited to material properties, size parameters, etc. Figure 2 As shown, export the BIM model in Revit to FBX format;

[0056] Step 1.2: Use PiXYZStudio software to optimize the FBX format model exported from Revit, such as Figure 3 As shown in the figure, perform the Optimize Mesh operation to reduce the number of polygons to improve the rendering efficiency and smoothness of the model on the HoloLens 2 device.

[0057] Step 1.3: Configure the mixed reality development environment in Unity and import the building model information into Unity. Based on the MRTK toolkit, mount "NearInteractionGrabbable.cs", "ManipulationHandler.cs", "BoundingBox.cs" and "Interactable.cs" on the BIM model in sequence to realize the functions of grabbing, moving, rotating and scaling the BIM model.

[0058] Further, in step 2, the following steps may be performed in sequence:

[0059] Step 2.1: Initialize the MRTK eye tracker. In Unity's "Start" method, write C# code to initialize the "IMixedRealityEyeGazeProvider" object to obtain and process eye tracking data, such as Figure 4 As shown;

[0060] Step 2.2: Write C# code in Visual Studio to determine whether the eye tracking data is valid and obtain the user's current gaze point position. Access "eyeGazeProvider.Timestamp" to obtain the timestamp associated with the current gaze data to keep the data synchronized, such as Figure 5 shown.

[0061] Further, in step 3, the following steps may be performed in sequence:

[0062] Step 3.1: Create an information panel for each component and store it in an array as Unity Prefabs; call the Unity engine "SetActive(false)" command to set the initial state of all information panels to invisible, so that the panels are only activated when staring, such as Figure 6 shown.

[0063] Step 3.2: Use Unity's "Update" method to monitor the user's gaze point every frame, and combine the self-developed algorithm to determine the intersection of the user's line of sight with objects in the scene. The algorithm analyzes the gaze time, completes the object selection and handles the switching of the gaze object. Set a threshold "gazeStayThreshold" to determine the minimum gaze duration required to trigger the interaction, which is 1 second. Use the "gazeTimer" variable to record the user's gaze duration on the current gaze object. When the user's gaze focus changes, reset the "gazeTimer" and handle the interaction of looking away. When the user's gaze on the same object exceeds the set threshold, the selection of the gaze object is completed, such as Figure 7 shown.

[0064] Step 3.3: In Unity, add a "BoxCollider" collision body to the imported BIM model component, such as Figure 8 When the gaze ray collides with the collision body, the BIM model component information of the collision body is returned and the corresponding information panel is activated.

[0065] Step 3.4: Record the user's gaze point position according to the self-developed algorithm, and calculate the optimal display position of the information panel in combination with the preset distance parameters. When the gaze ray collides with the collision body, move the activated information panel to the calculated optimal display position.

[0066] Step 3.5: Add the "billboard.cs" and "radialview.cs" scripts to the information panel to automatically adjust the orientation of the information panel and ensure that the panel always faces the user, thereby improving the readability and interactive experience of the information. The "billboard.cs" script makes the panel always face the user's line of sight, and the "radialview.cs" script is used to maintain the specific position and angle of the panel relative to the user. In this way, the information on the panel is always easy to see and interact with even if the user moves or turns his head.

[0067] Further, in step 4, the project is packaged into a Visual Studio Solution file, the project is compiled into the ARM64 format using Visual Studio, and published into an APPX format installation package. The project is deployed to HoloLens 2 via a Wi-Fi connection.

[0068] Further, in step 5, the following steps may be performed in sequence:

[0069] Step 5.1: Use the grab, move, rotate, and scale functions added in step 1.3 to align the BIM model with the physical entity of the building component. Use the MRTK toolkit combined with the spatial mapping of Hololens 2 to create a spatial anchor for the BIM model and lock the BIM model in space to link and bind the digital information of the building component with the physical entity of the building component. Fig. 9 shown.

[0070] Step 5.2: Verify the alignment accuracy between the model and the physical building through on-site testing, making the deviation less than 1 cm.

[0071] Further, in step 6, the user wears the HoloLens 2 device at the construction site, such as Fig.10 As shown in Figure 1, the system can recognize and respond to the user's gaze point in real time. When the user gazes at the building entity, the system immediately displays the corresponding model information based on the gaze point, such as material properties and design parameters.

[0072] The visual gaze-based building in-situ information retrieval method provided by the present invention integrates eye tracking technology, mixed reality technology and building information modeling technology into the whole life cycle of civil engineering and intelligent construction. This method directly uses the user's gaze point to obtain detailed information of building components and displays them in real time in real space, which greatly improves the efficiency, accuracy and interactive naturalness of information retrieval. Through the self-developed algorithm, the Unity engine and the HoloLens 2 device are used to avoid the use of desktop BIM software on the construction site. During the construction process, especially in the construction acceptance stage, the application of the present invention enables the construction team to accurately and quickly check and confirm the details of the project. In the building operation and maintenance stage, the present invention provides an efficient asset management tool for operation and maintenance personnel. Through eye tracking, the relevant information of key building components, such as maintenance records, performance data, etc., can be quickly accessed, and the operation and maintenance management becomes more efficient and accurate. This direct and real-time information access method greatly improves the maintenance efficiency and effectiveness of building facilities. The application of the present invention not only optimizes the user's workflow, improves the interactivity and immersion of on-site work, but also demonstrates the huge potential and value of digital transformation in the construction industry, and provides strong support for the sustainable development of the civil engineering field.

[0073] The present invention is not limited to the above-mentioned embodiments. Anyone can derive other various forms of products under the inspiration of the present invention. However, no matter what changes are made in the shape or structure, any technical solution that is the same or similar to that of the present application falls within the protection scope of the present invention.

Claims

1. A method for retrieval of architectural in-situ information based on visual gaze, characterized in that: The method comprises the following steps: Step 1: Extract and export the BIM model in Autodesk Revit to FBX format, optimize the model to reduce the number of triangle meshes, then configure the mixed reality development environment in Unity and import the optimized model. Based on MRTK (MixedReality Toolkit), realize the functions of grabbing, moving, rotating and scaling the BIM model. Step 2: Write C# code to initialize the eye tracker in the MRTK toolkit, write code to determine the validity of the eye movement data, obtain the user's gaze point position, and access the timestamp associated with the current gaze data to keep the data synchronized; Step 3. In Unity, create an information panel for the BIM model component and set it to an initially invisible state, which is activated only when the user gazes at it. By monitoring the user's gaze point, use a floating-point variable to record the user's gaze duration on the current gaze object, set a gaze duration threshold, determine the minimum gaze duration required to trigger the interaction, and add a "BoxCollider" collision body to the imported BIM model component in Unity. When the gaze ray collides with the collision body, the BIM model component information to which the collision body belongs is returned, and the corresponding information panel is activated. Position the information panel next to the gaze point and adjust it to an appropriate size. Add the UI optimization script in the MRTK toolkit to the information panel so that the information panel always faces the user. Step 4: Visual Studio compiles the project to ARM64 format and publishes it to Hololens 2; Step 5: Align the BIM model with the physical entity of the building component through the grab, move, rotate and scale functions added in step 1, create a spatial anchor point for the BIM model, and lock the BIM model in space to link and bind the digital information of the building component with the physical entity of the building component; Step 6: Wearing the Hololens 2 device in real space, when the user stares at a building component for longer than the set threshold, the stared component is automatically identified, the information of the building component is returned, and the detailed information of the component is displayed in situ in the user's field of vision, including material properties and size specifications, thereby realizing in-situ building information retrieval based on visual gaze in the building entity.

2. The method for retrieval of architectural in-situ information based on visual gaze according to claim 1, characterized in that: In step 1, perform the following steps in sequence: Step 1.1: Use Autodesk Revit as the main BIM software to extract building design and structural information, including detailed components and material properties; export the BIM model in Revit to FBX format; Step 1.2: Optimize the model exported from Revit to reduce the number of triangle meshes; Step 1.3: Configure the mixed reality development environment in Unity and import the building model information into Unity. Based on the MRTK toolkit, mount "NearInteractionGrabbable.cs", "ManipulationHandler.cs", "BoundingBox.cs", and "Interactable.cs" on the BIM model in sequence to realize the functions of grabbing, moving, rotating, and scaling the BIM model.

3. The method for retrieval of architectural in-situ information based on visual gaze according to claim 1, characterized in that: In step 2, perform the following steps in sequence: Step 2.1: Activate and configure the MRTK eye tracker; Step 2.2: Write C# code in Visual Studio to determine the validity of eye movement data, obtain gaze point location data, and obtain timestamps to keep data synchronized.

4. The method for retrieval of architectural in-situ information based on visual gaze according to claim 1, characterized in that: In step 3, the following steps may be performed in sequence: Step 3.1: In Unity, create an information panel for each widget and write C# code to make the information panel active only when staring at it. Step 3.2: Use Unity's "Update" method to monitor the user's gaze point in each frame, and process the gaze data in real time to determine object selection and switching; Step 3.3: In Unity, add a "BoxCollider" collision body to the model component, perform collision detection, and activate the corresponding information panel when a collision occurs; Step 3.4: In Unity, dynamically position the information panel next to the gaze point and adjust it to a suitable size for viewing; Step 3.5: Optimize the information panel display logic in Unity so that the panel always faces the user.

5. The method for retrieval of architectural in-situ information based on visual gaze according to claim 1, characterized in that: In step 4, the project is packaged into a Visual Studio Solution file, the project is compiled into ARM64 format using Visual Studio, and published into an APPX format installation package; Deploy the project to HoloLens 2 via Wi-Fi connection.

6. The method for retrieval of architectural in-situ information based on visual gaze according to claim 2, characterized in that: In step 5, the following steps may be performed in sequence: Step 5.1: Use the grab, move, rotate, and scale functions added in step 1.3 to align the BIM model with the physical entity of the building component. Use the MRTK toolkit combined with the spatial mapping of Hololens 2 to create a spatial anchor for the BIM model and lock the BIM model in space to link and bind the digital information of the building component with the physical entity of the building component. Step 5.2: Verify the alignment accuracy between the model and the physical building through on-site testing, making the deviation less than 1 cm.

7. The method for retrieval of architectural in-situ information based on visual gaze according to claim 1, characterized in that: In step 6, at the construction site, when the user wears the HoloLens 2 device, the system can recognize and respond to the user's gaze point in real time; when the user gazes at the building entity, the system displays the corresponding information based on the gaze point, including material properties and design parameters.

Images