Guniting robot in-situ operation simulation method and system based on mixed reality

Abstract

The invention discloses a guniting robot in-situ operation simulation method and system based on mixed reality. The method comprises the steps that a mixed reality environment is configured; spouting particle simulation; spraying a patch to be attached to the surface; managing a spraying and pasting object; carrying out persistence and visual analysis on the guniting effect; compiling deployment and on-site in-situ simulation are carried out. The invention provides a real environment driven guniting robot in-situ operation simulation method based on a mixed reality technology. Accurately linking a virtual guniting effect with a real environment surface by using a mixed reality technology; the simulation of the guniting track, the guniting thickness distribution and the coverage rate is realized by combining a physical engine and a dynamic particle collision detection algorithm. The digital construction rehearsal tool provided by the invention solves the problems of dependence on manual trial spraying, large off-line simulation deviation and lack of real-time feedback in the traditional process, and remarkably improves the construction quality and the intelligent level.

Description

Technical Field

[0001] The present invention relates to the fields of civil engineering and intelligent construction, and in particular to a mixed reality-based in-situ operation simulation method and system for a shotcrete robot. Background Art

[0002] Shotcrete construction is a core process in construction, tunneling and mining engineering. Its goal is to reinforce the target surface by spraying concrete or mortar materials. At present, the operation planning and effect verification of shotcrete robots mainly rely on traditional technical means. Before construction, the shotcrete equipment is manually operated to conduct local test spraying, and the coverage effect is judged based on experience. This method has the following main defects: (1) It is out of touch with the real environment: offline simulation relies on simplified models and cannot reflect the geometric complexity and dynamic changes of the real physical scene; (2) Construction personnel cannot directly observe the spraying effect and dynamically adjust the parameters at the real work site; (3) Physical test spraying and repeated corrections lead to a significant increase in material, time and labor costs. In order to break through the bottleneck of traditional technology, mixed reality (MR) technology provides an innovative direction for shotcrete simulation. MR technology deeply integrates the virtual world with the real world to create a mixed environment of virtual and real interaction. Users can directly control virtual objects through natural interaction methods such as gestures and voice. Utilizing the depth sensor and spatial mapping function of the mixed reality device, the geometric structure of the real environment is scanned and reconstructed in real time, generating a high-precision spatial grid model, providing a real physical basis for shotcrete simulation, and rendering the shotcrete particle trajectory and deposition effect in real time, mapping the shotcrete effect to the real environment surface in real time.

[0003] Therefore, the present invention proposes a mixed reality-based shotcrete robot in-situ operation simulation method and system, aiming to achieve high-precision dynamic fusion of shotcrete effects and real environment through mixed reality technology, combined with physical simulation algorithms, to improve construction efficiency and decision-making accuracy. Summary of the Invention

[0004] In response to the above problems, the present invention aims to provide a mixed reality-based in-situ operation simulation method and system for a shotcrete robot, the method comprising the following steps:

[0005] Step 1: Mixed Reality Environment Configuration: Use the Mixed Reality Toolkit (MRTK) to create a mixed reality scene in Unity for simulating the in-situ operation of the shotcrete robot in a real physical environment. Create a shotcrete robot model. Create a SpatialMeshObserver to obtain the spatial mesh data scanned by the mixed reality device and add a Mesh Collider to the scanned spatial mesh.

[0006] Step 2: Spray particle simulation: Configure the particle system in Unity to simulate the spray effect of spray, set the particle life cycle, speed, size and emission rate parameters, and enable the Collision particle system collision detection module;

[0007] Step 3: Spray patch generation and surface fitting: When the spray particles collide with the spatial grid, a spray patch object is generated based on the coordinates of the collision point. According to the surface normal direction of the collision point, the rotation matrix of the spray patch object is adjusted, and an offset in the normal direction is applied to make the spray patch fit the real physical environment.

[0008] Step 4: Spray patch object management: Use object pool technology to instantiate and recycle spray patch objects; set the life cycle of spray patch objects;

[0009] Step 5: Persistence and visual analysis of the spraying effect: Record the sprayed patch object data and generate a three-dimensional heat map of the spraying coverage area to achieve quantitative analysis and visual feedback of the spraying effect;

[0010] Step 6: Compile, deploy, and simulate on-site: Use Visual Studio to compile the project into ARM64 format and publish it as an APPX installation package, which is then deployed to the mixed reality device. Wear the mixed reality device in real space and simulate the in-situ operation of the shotcrete robot at the actual work site.

[0011] Furthermore, in step 1, the following steps may be performed in sequence:

[0012] Step 1.1: Create a new scene in Unity and use the Mixed Reality Toolkit (MRTK) to create a mixed reality scene.

[0013] Step 1.2: Create a shotcrete robot model and set the model hierarchy and initial position in advance;

[0014] Step 1.3: Enable the Spatial Awareness System, create a SpatialMeshObserver, obtain the spatial mesh data scanned by the mixed reality device, and add a Mesh Collider to the scanned mesh.

[0015] Furthermore, in step 2, the following steps may be performed in sequence:

[0016] Step 2.1: Set the particle emission shape to cone, set the cone angle range and the initial particle velocity;

[0017] Step 2.2: Enable the gravity effect module of the particle system and set the gravity coefficient to simulate the actual motion trajectory of the spraying particles under the influence of gravity;

[0018] Step 2.3: Bind the custom shader to the particle material and enable the collision detection module to receive particle collision event information.

[0019] Furthermore, in step 3, the following steps may be performed in sequence:

[0020] Step 3.1: Obtain the coordinates of the collision point and the surface normal direction through the particle collision event;

[0021] Step 3.2: Calculate the rotation matrix of the spray patch object based on the surface normal direction and apply an offset to fit the surface;

[0022] Step 3.3: Assign the spray patch object to a separate spray patch layer and exclude that layer from the particle system's collision detection.

[0023] Furthermore, in step 4, the following steps may be performed in sequence:

[0024] Step 4.1: Initialize the spray patch object pool and preset the object pool capacity;

[0025] Step 4.2: When the number of spray patch generation requests exceeds the object pool capacity, the earliest generated spray patch is recycled.

[0026] Furthermore, in step 5, the following steps may be performed in sequence:

[0027] Step 5.1: Record the location, generation time and texture density data of all spray patches;

[0028] Step 5.2: Generate a three-dimensional heat map of the spray coverage area based on the spray patch data, with red representing high-density areas and gray representing low-density areas.

[0029] Furthermore, in step 6, by using Visual Studio to compile the project into ARM64 format, generating an APPX installation package, and deploying it to the mixed reality device, the spraying robot operation simulation is performed on-site at the construction site, and key indicators such as the spraying thickness distribution and coverage are visualized on-site through the mixed reality device.

[0030] The advantages and positive effects of the present invention are:

[0031] (1) The present invention utilizes mixed reality space mapping and dynamic particle collision detection technology to achieve a precise fusion of virtual spraying effects and the real physical environment. By real-time scanning of the environment to generate a spatial grid and bind collision bodies, the motion trajectory of the spraying particles and the surface adhesion effect strictly follow the laws of real physics, making the user subjectively believe that the spraying deposition effect directly acts on the surface of the real environment. This method avoids the separation problem of virtual models and physical environments in traditional simulation software, ensures the authenticity of the spraying simulation and the accuracy of construction feedback, and improves the user's ability to predict construction quality.

[0032] (2) The present invention achieves efficient and stable simulation performance through dynamic spray patch generation and object pool management technology. Spray patches are generated in real time based on particle collision events, and a normal direction offset algorithm is used to ensure seamless adhesion between the patch and the surface. Combined with the object pool recycling mechanism and lifecycle control, the system maintains a stable frame rate when running on mixed reality devices. This efficient resource management method not only reduces hardware performance dependence but also significantly improves simulation fluency in complex scenarios. It is particularly suitable for in-situ operation verification in large-scale dynamic environments such as tunnels and construction sites.

[0033] (3) The three-dimensional thermal map analysis and in-situ deployment of the present invention enable quantitative evaluation of the spraying effect and real-time decision support. By recording the location, density, and time data of the sprayed patches, a thermal map of the coverage area is generated, allowing users to intuitively identify areas of uneven spraying. This approach transforms traditional post-construction inspection into a process-controllable digital preview, reducing material waste and shortening the iteration cycle of construction plans.

[0034] In summary, the present invention has constructed a high-precision in-situ operation simulation system for shotcrete robots through the deep integration of mixed reality technology, real-time physical simulation and intelligent resource management. The present invention overcomes the technical difficulties of dynamic adaptation of virtual shotcrete to the real environment, and significantly improves the scientific nature and operability of construction planning. In the fields of tunnel support, building reinforcement, etc., users can verify the shotcrete plan and optimize construction parameters in situ through mixed reality equipment without relying on physical test spraying or offline simulation software. This technology reduces the cost of engineering trial and error, improves construction safety in complex scenarios, and provides a reusable technical framework for intelligent construction and industrial automation, accelerating the transformation of the infrastructure field towards digitalization and intelligence. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The above and / or other aspects and advantages of the present invention will become more clear and easier to understand through the detailed description made in conjunction with the following drawings, which are only illustrative and do not limit the present invention, wherein:

[0036] Figure 1 It is a mixed reality-based simulation method and system flow chart for the in-situ operation of the shotcrete robot.

[0037] Figure 2 This is an example diagram of a mixed reality space mesh setup.

[0038] Figure 3 This is an example diagram of particle system settings.

[0039] Figure 4 This is an example diagram of a C# statement for receiving particle collision event information.

[0040] Figure 5 This is an example diagram of C# statements for spraying patches to fit the real physical environment.

[0041] Figure 6 This is an example diagram of C# statements for thermal analysis of shotcrete effects.

[0042] Figure 7 This is an example diagram of the in-situ operation simulation method and system of the shotcrete robot based on mixed reality. DETAILED DESCRIPTION

[0043] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0044] like Figure 1 As shown, the present invention provides a method and system for simulating in-situ operation of a shotcrete robot based on mixed reality, the method comprising the following steps:

[0045] Step 1: Mixed Reality Environment Configuration: Use the Mixed Reality Toolkit (MRTK) to create a mixed reality scene in Unity for simulating the in-situ operation of the shotcrete robot in a real physical environment. Create a shotcrete robot model. Create a SpatialMeshObserver to obtain the spatial mesh data scanned by the mixed reality device and add a Mesh Collider to the scanned spatial mesh.

[0046] Step 2: Spray particle simulation: Configure the particle system in Unity to simulate the spray effect of spray, set the particle life cycle, speed, size and emission rate parameters, and enable the Collision particle system collision detection module;

[0047] Step 3: Spray patch generation and surface fitting: When the spray particles collide with the spatial grid, a spray patch object is generated based on the coordinates of the collision point. According to the surface normal direction of the collision point, the rotation matrix of the spray patch object is adjusted, and an offset in the normal direction is applied to make the spray patch fit the real physical environment.

[0048] Step 4: Spray patch object management: Use object pool technology to instantiate and recycle spray patch objects; set the life cycle of spray patch objects;

[0049] Step 5: Persistence and visual analysis of the spraying effect: Record the sprayed patch object data and generate a three-dimensional heat map of the spraying coverage area to achieve quantitative analysis and visual feedback of the spraying effect;

[0050] Step 6: Compile, deploy, and simulate on-site: Use Visual Studio to compile the project into ARM64 format and publish it as an APPX installation package, which is then deployed to the mixed reality device. Wear the mixed reality device in real space and simulate the in-situ operation of the shotcrete robot at the actual work site.

[0051] Furthermore, in step 1, the following steps may be performed in sequence:

[0052] Step 1.1: Create a new scene in Unity and use the Mixed Reality Toolkit (MRTK) to create a mixed reality scene.

[0053] Step 1.2: Create a shotcrete robot model and set the model hierarchy and initial position in advance;

[0054] Step 1.3: Enable Spatial Awareness System, create SpatialMeshObserver, obtain the spatial mesh data scanned by the mixed reality device, and add a Mesh Collider collision body to the scanned mesh, such as Figure 2 shown.

[0055] Furthermore, in step 2, the following steps may be performed in sequence:

[0056] Step 2.1: Set the particle emission shape to cone, set the cone angle range and the initial particle velocity, such as Figure 3 As shown;

[0057] Step 2.2: Enable the gravity effect module of the particle system and set the gravity coefficient to simulate the actual motion trajectory of the spraying particles under the influence of gravity;

[0058] Step 2.3: Bind the custom shader to the particle material, enable the collision detection module, and create a List <particlecollisionevent>List to store detailed information of collision events; use GetCollisionEvents method to obtain all collision events between particle system and space grid, store event information in collisionEvents list, and return the number of collision events numCollisionEvents. Figure 4 shown.

[0059] Furthermore, in step 3, the following steps may be performed in sequence:

[0060] Step 3.1: Traverse all collision events and obtain the coordinates of the collision point and the surface normal direction through the particle collision event;

[0061] Step 3.2: Calculate the rotation matrix of the spray patch object based on the surface normal direction and apply an offset to fit the surface; fine-tune the position of the spray patch on the surface by setting the position of the spray patch to the collision point plus the surface normal multiplied by the offset. The collision point position is , the normal vector of the collision surface is , the offset is , the spray patch position is ; Use the Quaternion.LookRotation method in Unity and use the negative direction of the normal as the parameter to rotate so that the spray patch faces the opposite direction of the collision surface and is displayed correctly on the surface, such as Figure 5 As shown;

[0062] Step 3.3: Assign the spray patch object to a separate spray patch layer and exclude that layer from the particle system's collision detection.

[0063] Furthermore, in step 4, the following steps may be performed in sequence:

[0064] Step 4.1: Initialize the spray patch object pool, preset the object pool capacity, and use Queue <gameobject>To store the spray patch object;

[0065] Step 4.2: When the number of spray patch generation requests exceeds the object pool capacity, the earliest generated spray patch is recycled and added back to the object pool.

[0066] Furthermore, in step 5, the following steps may be performed in sequence:

[0067] Step 5.1: Record the location, generation time, and texture density data of all spray patches. Create a SprayPatchData class and store the location, generation time, and texture density of the spray patches in this class. Then add this object to the sprayPatchDataList list.

[0068] Step 5.2: Generate a 3D heat map of the spray coverage area based on the spray patch data. The GenerateHeatmap method iterates over each SprayPatchData in the sprayPatchDataList list. Based on the properties of each spray patch, use the Color.Lerp method to interpolate between gray and red. Use the Instantiate method to instantiate a heat map point at the location of the spray patch and apply the calculated color to the point's material, as shown in the following example: Figure 6 shown.

[0069] Furthermore, in step 6, the project is compiled into ARM64 format using Visual Studio, an APPX installation package is generated, and it is deployed to the mixed reality device. Then, the spraying robot operation simulation is carried out in situ at the construction site, and the key indicators such as the spraying thickness distribution and coverage are visualized in situ through the mixed reality device. Figure 7 shown.

[0070] The mixed reality-based in-situ simulation method and system for shotcrete robots provided by this invention seamlessly integrates virtual shotcrete operations with the real physical environment by combining mixed reality technology with real-time physics simulation. This approach significantly improves the realism of shotcrete simulation and the real-time operation feedback by dynamically generating spray patches and precisely fitting them to the surrounding surface. By efficiently combining a proprietary particle collision detection algorithm with the Unity engine, this invention avoids the reliance on traditional paper drawings and the limitations of desktop simulation software. This allows construction workers to intuitively adjust shotcrete parameters and verify the results in real time in complex engineering sites (such as tunnels and construction sites), significantly enhancing the fluidity of construction planning and decision-making. In applications across multiple fields, particularly in tunnel support and building structure reinforcement, users can directly observe key indicators such as shotcrete thickness distribution and coverage through mixed reality devices, and quickly optimize construction plans based on three-dimensional heat maps. This method not only simplifies the trial-and-error process inherent in traditional shotcrete processes, but also significantly reduces material waste and construction risks, thereby improving project efficiency. In addition, the application of this invention greatly optimizes the immersion and accuracy of on-site human-computer interaction, provides innovative tools for industrial automation and digital construction, and promotes the in-depth application and large-scale development of intelligent construction technology in the infrastructure field.

[0071] The present invention is not limited to the above-mentioned embodiments. Anyone can derive other forms of products under the inspiration of the present invention. However, no matter what changes are made in the shape or structure, any product with the same or similar technical solutions as the present application falls within the scope of protection of the present invention.< / gameobject> < / particlecollisionevent>

Claims

1. A mixed reality-based in-situ operation simulation method and system for a shotcrete robot, characterized in that: The method comprises the following steps: Step 1: Mixed Reality Environment Configuration: Use the Mixed Reality Toolkit (MRTK) to create a mixed reality scene in Unity for simulating the in-situ operation of the shotcrete robot in a real physical environment. Create a shotcrete robot model. Create a SpatialMeshObserver to obtain the spatial mesh data scanned by the mixed reality device and add a Mesh Collider to the scanned spatial mesh. Step 2: Spray particle simulation: Configure the particle system in Unity to simulate the spray effect of spray, set the particle life cycle, speed, size and emission rate parameters, and enable the Collision particle system collision detection module; Step 3: Spray patch generation and surface fitting: When the spray particles collide with the spatial grid, a spray patch object is generated based on the coordinates of the collision point. According to the surface normal direction of the collision point, the rotation matrix of the spray patch object is adjusted, and an offset in the normal direction is applied to make the spray patch fit the real physical environment. Step 4: Spray patch object management: Use object pool technology to instantiate and recycle spray patch objects; set the life cycle of spray patch objects; Step 5: Persistence and visual analysis of the spraying effect: Record the sprayed patch object data and generate a three-dimensional heat map of the spraying coverage area to achieve quantitative analysis and visual feedback of the spraying effect; Step 6: Compile, deploy, and simulate on-site: Use Visual Studio to compile the project into ARM64 format and publish it as an APPX installation package, which is then deployed to the mixed reality device. Wear the mixed reality device in real space and simulate the in-situ operation of the shotcrete robot at the actual work site.

2. The mixed reality-based in-situ operation simulation method and system for a shotcrete robot according to claim 1, characterized in that: In step 1, perform the following steps in sequence: Step 1.1: Create a new scene in Unity and use the Mixed Reality Toolkit (MRTK) to create a mixed reality scene. Step 1.2: Create a shotcrete robot model and set the model hierarchy and initial position in advance; Step 1.3: Enable the Spatial Awareness System, create a SpatialMeshObserver, obtain the spatial mesh data scanned by the mixed reality device, and add a Mesh Collider to the scanned mesh.

3. The mixed reality-based in-situ operation simulation method and system for a shotcrete robot according to claim 1, characterized in that: In step 2, perform the following steps in sequence: Step 2.1: Set the particle emission shape to cone, set the cone angle range and the initial particle velocity; Step 2.2: Enable the gravity effect module of the particle system and set the gravity coefficient to simulate the actual motion trajectory of the spraying particles under the influence of gravity; Step 2.3: Bind the custom shader to the particle material and enable the collision detection module to receive particle collision event information.

4. The mixed reality-based in-situ operation simulation method and system for a shotcrete robot according to claim 1, characterized in that: In step 3, perform the following steps in sequence: Step 3.1: Obtain the coordinates of the collision point and the surface normal direction through the particle collision event; Step 3.2: Calculate the rotation matrix of the spray patch object based on the surface normal direction and apply an offset to fit the surface; Step 3.3: Assign the spray patch object to a separate spray patch layer and exclude that layer from the particle system's collision detection.

5. The mixed reality-based in-situ operation simulation method and system for a shotcrete robot according to claim 1, characterized in that: In step 4, perform the following steps in sequence: Step 4.1: Initialize the spray patch object pool and preset the object pool capacity; Step 4.2: When the number of spray patch generation requests exceeds the object pool capacity, the earliest generated spray patch is recycled.

6. The mixed reality-based in-situ operation simulation method and system for a shotcrete robot according to claim 1, characterized in that: In step 5, the following steps may be performed in sequence: Step 5.1: Record the location, generation time and texture density data of all spray patches; Step 5.2: Generate a three-dimensional thermal map of the spray coverage area based on the spray patch data.

7. The mixed reality-based in-situ operation simulation method and system for a shotcrete robot according to claim 1, characterized in that: In step 6, by using Visual Studio to compile the project into ARM64 format, generate an APPX installation package, and deploy it to the mixed reality device, the shotcrete robot operation simulation is performed on-site at the construction site, and key indicators such as the shotcrete thickness distribution and coverage are visualized on-site through the mixed reality device.

Images