A mixed reality based guniting robot in-situ operation simulation method and system

Abstract

The application discloses a kind of based on mixed reality's guniting robot in-situ operation simulation method and system, its method includes: mixed reality environment configuration;Guniting particle simulation;Spraying patch generation and surface fitting;Spraying patch object management;Guniting effect's persistence and visual analysis;Compiling deployment and in-situ simulation.The application based on mixed reality technology proposes a kind of real environment driven guniting robot in-situ operation simulation method.Virtual guniting effect is accurately linked with real environment surface using mixed reality technology;Combined with physical engine and dynamic particle collision detection algorithm, the simulation of guniting trajectory, guniting thickness distribution and coverage rate is realized.The application provides a kind of digital construction preview tool, solves the problem that traditional process relies on manual test spraying, offline simulation deviation is big and lacks real-time feedback, significantly improves construction quality and intelligent level.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of civil engineering and intelligent construction, in particular to a mixed reality based in-situ operation simulation method and system for a guniting robot. BACKGROUND

[0002] Guniting is a core technology in the construction, tunnel and mine engineering, which aims to reinforce the target surface by spraying concrete or mortar material. Currently, the operation planning and effect verification of the guniting robot mainly rely on traditional technical means. Before construction, local test spraying is performed by manually operating the guniting equipment, and the coverage effect is judged by experience. This method mainly has the following defects: (1) it is divorced from the real environment: offline simulation relies on simplified models and cannot reflect the geometric complexity and dynamic changes of the real physical scene; (2) the construction personnel cannot directly observe the guniting effect in the real operation site and dynamically adjust the parameters; (3) physical test spraying and repeated correction result in significant increase in material, time and labor costs. In order to break through the bottleneck of traditional technology, mixed reality (MR) technology provides an innovative direction for guniting simulation. MR technology deeply integrates the virtual world with the real world, creating a mixed environment for virtual and real interaction. Users can directly control virtual objects through natural interaction methods such as gestures and voice. Using the depth sensor and space mapping function of the mixed reality device, the geometric structure of the real environment is scanned and reconstructed in real time, generating a high-precision spatial mesh model, providing a real physical basis for guniting simulation, and rendering the guniting particle trajectory and deposition effect in real time, and mapping the guniting effect to the real environment surface in real time.

[0003] Therefore, the present application proposes a mixed reality based in-situ operation simulation method and system for a guniting robot, aiming to realize high-precision dynamic fusion of guniting effect and real environment through mixed reality technology, and improve construction efficiency and decision accuracy by combining physical simulation algorithms. SUMMARY

[0004] In view of the above problems, the present application aims to provide a mixed reality based in-situ operation simulation method and system for a guniting robot, which comprises the following steps:

[0005] Step 1: Mixed reality environment configuration: use the mixed reality toolkit MRTK (Mixed Reality Toolkit) to create a mixed reality scene in Unity for in-situ operation simulation of a guniting robot in a real physical environment; create a guniting robot model; create a SpatialMeshObserver to obtain spatial mesh data scanned by a mixed reality device, and add a Mesh Collider collision body to the scanned spatial mesh;

[0006] Step 2: Spray particle simulation: Configure the particle system in Unity to simulate the spray effect of the spray, set the particle life cycle, speed, size, and emission rate parameters, and enable the Collision particle system collision detection module;

[0007] Step 3: Spray patch generation and surface fitting: When the spray particles collide with the spatial mesh, generate a spray patch object based on the coordinates of the collision points; adjust the rotation matrix of the spray patch object according to the surface normal direction of the collision points, and apply a normal direction offset to make the spray patch fit the real physical environment;

[0008] Step 4: Spray patch object management: Use object pool technology to instantiate and recycle spray patch objects; set the life cycle of the spray patch object;

[0009] Step 5: Persistence and visual analysis of spray effect: Record spray patch object data and generate a three-dimensional heat map of the spray coverage area to achieve quantitative analysis and visual feedback of the spray effect;

[0010] Step 6: Compile and deploy in situ simulation: Compile the project into ARM64 format and publish it as an APPX installation package through Visual Studio, and deploy it to a mixed reality device; wear a mixed reality device in a real space and perform in-situ simulation of the spray robot in the actual work site.

[0011] Further, in step 1, the following steps can be performed in sequence:

[0012] Step 1.1: Create a new scene in Unity and use the Mixed Reality Toolkit (MRTK) to create a mixed reality scene;

[0013] Step 1.2: Create a spray robot model and set the model hierarchy, initial position in advance;

[0014] Step 1.3: Enable Spatial Awareness System, create SpatialMeshObserver, obtain spatial mesh data scanned by the mixed reality device, and add Mesh Collider collision body to the scanned mesh.

[0015] Further, in step 2, the following steps can be performed in sequence:

[0016] Step 2.1: Set the particle emission shape to be conical, set the cone angle range, and set the initial particle speed;

[0017] Step 2.2: Enable the gravity effect module of the particle system, set the gravity coefficient to simulate the real motion trajectory of the spray particles under the action of gravity;

[0018] Step 2.3: Bind the custom shader to the particle material and enable the collision detection module to receive particle collision event information.

[0019] Furthermore, in step 3, the following steps can be performed sequentially:

[0020] Step 3.1: Obtain the coordinates of the collision point and the direction of the surface normal through particle collision events;

[0021] Step 3.2: Calculate the rotation matrix of the spray patch object based on the surface normal direction, and apply an offset to fit the surface;

[0022] Step 3.3: Assign the spray patch object to an independent spray patch layer and exclude that layer in the collision detection of the particle system.

[0023] Furthermore, in step 4, the following steps can be performed sequentially:

[0024] Step 4.1: Initialize the spray patch object pool and preset the object pool capacity;

[0025] Step 4.2: When the number of spray patch generation requests exceeds the object pool capacity, reclaim the earliest generated spray patch.

[0026] Furthermore, in step 5, the following steps can be performed sequentially:

[0027] Step 5.1: Record the location, generation time, and texture density data of all sprayed patches;

[0028] Step 5.2: Generate a three-dimensional thermal map of the sprayed area based on the spray patch data. Red indicates high-density areas and gray indicates low-density areas.

[0029] Furthermore, in step 6, the project is compiled into ARM64 format using Visual Studio to generate an APPX installation package, which is then deployed to a mixed reality device. In-situ simulation of the shotcrete robot operation is then performed at the engineering site, and key indicators such as shotcrete thickness distribution and coverage are visualized in-situ using the mixed reality device.

[0030] The advantages and positive effects of this invention are as follows:

[0031] (1) This invention utilizes mixed reality spatial mapping and dynamic particle collision detection technology to achieve a precise fusion of virtual spraying effects and the real physical environment. By generating a spatial mesh and binding collision bodies through real-time environmental scanning, the motion trajectory and surface adhesion effect of the sprayed particles strictly follow real physical laws, making users subjectively believe that the spraying deposition effect directly acts on the surface of the real environment. This approach avoids the separation problem between the virtual model and the physical environment in traditional simulation software, ensuring the realism of the spraying simulation and the accuracy of construction feedback, while also improving users' ability to predict construction quality.

[0032] (2) This invention achieves efficient and stable simulation performance through dynamic spray patch generation and object pool management technology. Spray patches are generated in real time based on particle collision events, and a normal direction offset algorithm is used to ensure seamless bonding between the patches and the surface; combined with the object pool recycling mechanism and lifecycle control, the system maintains a stable frame rate when running on mixed reality devices. This efficient resource management method not only reduces hardware performance dependence but also significantly improves the smoothness of simulation in complex scenarios, and is especially suitable for in-situ operation verification in large-scale dynamic environments such as tunnels and construction sites.

[0033] (3) The three-dimensional thermal mapping analysis and in-situ deployment of the present invention realizes the quantitative evaluation and real-time decision support of the spraying effect. By recording the location, density and time data of the sprayed patches, a thermal map of the covered area is generated, and users can intuitively identify areas of uneven spraying. This method transforms traditional post-construction inspection into a process-controllable digital pre-simulation, reduces material waste and shortens the iteration cycle of construction plans.

[0034] In summary, this invention constructs a high-precision in-situ simulation system for shotcrete robots by deeply integrating mixed reality technology, real-time physical simulation, and intelligent resource management. This invention overcomes the technical challenge of dynamically adapting virtual shotcrete to the real environment, significantly improving the scientific rigor and operability of construction planning. In fields such as tunnel support and building reinforcement, users can verify shotcrete schemes and optimize construction parameters in situ using mixed reality equipment without relying on physical test spraying or offline simulation software. This technology reduces engineering trial-and-error costs, improves construction safety in complex scenarios, and provides a reusable technical framework for intelligent construction and industrial automation, accelerating the transformation of the infrastructure sector towards digitalization and intelligence. Attached Figure Description

[0035] The above and / or other aspects and advantages of the present invention will become clearer and more readily understood from the following detailed description taken in conjunction with the accompanying drawings, which are merely illustrative and do not limit the invention, wherein:

[0036] Figure 1 This is a simulation method and system flowchart for in-situ operation of shotcrete robots based on mixed reality.

[0037] Figure 2 This is an example diagram of a mixed reality space grid setting.

[0038] Figure 3 This is an example diagram of a particle system setup.

[0039] Figure 4 This is a sample diagram of C# statements for receiving particle collision event information.

[0040] Figure 5 This is an example diagram of C# statements used to apply spray-painted patches to a real physical environment.

[0041] Figure 6 This is a C# statement example diagram for the thermal analysis of the spraying effect.

[0042] Figure 7 This is an example diagram of a simulation method and system for in-situ operation of a shotcrete robot based on mixed reality. Detailed Implementation

[0043] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0044] like Figure 1 As shown, this invention provides a method and system for simulating in-situ operation of a shotcrete robot based on mixed reality. The method includes the following steps:

[0045] Step 1: Mixed Reality Environment Configuration: Use the Mixed Reality Toolkit (MRTK) in Unity to create a mixed reality scene for simulating in-situ operation of a shotcrete robot in a real physical environment; create a shotcrete robot model; create a SpatialMeshObserver to obtain spatial mesh data scanned by the mixed reality device, and add Mesh Collider colliders to the scanned spatial mesh;

[0046] Step 2: Particle Spray Simulation: Configure the particle system in Unity to simulate the spraying effect of the spray, set the particle lifecycle, speed, size and emission rate parameters, and enable the Collision particle system collision detection module;

[0047] Step 3: Spray Patch Generation and Surface Adhesion: When the sprayed particles collide with the spatial mesh, a spray patch object is generated based on the coordinates of the collision point; according to the surface normal direction of the collision point, the rotation matrix of the spray patch object is adjusted, and the offset of the normal direction is applied to make the spray patch adhere to the real physical environment.

[0048] Step 4: Spray Patch Object Management: Use object pooling technology to instantiate and recycle spray patch objects; set the lifecycle of spray patch objects;

[0049] Step 5: Persistent and visual analysis of spraying effect: Record the data of the sprayed patch object and generate a three-dimensional heat map of the sprayed area to achieve quantitative analysis and visual feedback of the spraying effect;

[0050] Step 6: Compile, deploy, and simulate in situ: Compile the project into ARM64 format using Visual Studio and publish it as an APPX installation package, then deploy it to the mixed reality device; wear the mixed reality device in the real space to simulate the in situ operation of the shotcrete robot at the actual work site.

[0051] Furthermore, in step 1, the following steps can be performed sequentially:

[0052] Step 1.1: Create a new scene in Unity and use the Mixed Reality Toolkit (MRTK) to create a mixed reality scene;

[0053] Step 1.2: Create a model of the shotcrete robot and pre-set the model hierarchy and initial position;

[0054] Step 1.3: Enable Spatial Awareness System, create a SpatialMeshObserver, acquire spatial mesh data scanned by the mixed reality device, and add Mesh Collider colliders to the scanned meshes, such as... Figure 2 As shown.

[0055] Furthermore, in step 2, the following steps can be performed sequentially:

[0056] Step 2.1: Set the particle emission shape to a cone, and set the cone angle range and initial particle velocity, such as... Figure 3 As shown;

[0057] Step 2.2: Activate the gravity influence module of the particle system and set the gravity coefficient to simulate the real motion trajectory of the sprayed particles under the action of gravity;

[0058] Step 2.3: Bind the custom shader to the particle material and enable the collision detection module, then create a List. <particlecollisionevent>The collision event details are stored in a list; all the collision events that occur in the particle system and the space grid are obtained using the GetCollisionEvents method, and the event information is stored in the collisionEvents list, and the number of collision events numCollisionEvents that occur is returned, as shown in the following code: Figure 4

[0059] Further, in step 3, the following steps can be executed in turn:

[0060] Step 3.1: Traverse all the collision events, and obtain the coordinates of the collision point and the surface normal direction through the particle collision event;

[0061] Step 3.2: Calculate the rotation matrix of the spray patch object according to the surface normal direction, and apply an offset to fit the surface; by setting the position of the spray patch to the collision point plus the surface normal multiplied by the offset, the position of the spray patch on the surface is fine-tuned, the collision point position is , the normal vector of the collision surface is , the offset is , and the position of the spray patch is ; using the Quaternion.LookRotation method in Unity, with the negative direction of the normal as the parameter to rotate, so that the spray patch is just facing the opposite direction of the collision surface, to correctly display on the surface, as shown in the following code: Figure 5

[0062] Step 3.3: Assign the spray patch object to a separate spray patch layer, and exclude this layer in the collision detection of the particle system.

[0063] Further, in step 4, the following steps can be executed in turn:

[0064] Step 4.1: Initialize the spray patch object pool, preset the object pool capacity, and use Queue <gameobject>to store the spray patch object;

[0065] Step 4.2: When the spray patch generation request exceeds the object pool capacity, the earliest generated spray patch is recycled and re-added to the object pool.

[0066] Further, in step 5, the following steps can be executed in turn:

[0067] Step 5.1: Record the position, generation time and texture density data of all spray patches, create a SprayPatchData class, and store the position, generation time and texture density of the spray patch in this class, and then add this object to the sprayPatchDataList list;

[0068] Step 5.2: Generate a three-dimensional heat map of the spray coverage area according to the spray patch data, and the GenerateHeatmap method traverses each SprayPatchData in the sprayPatchDataList list. According to the properties of each spray patch, use the Color.Lerp method to interpolate between gray and red; use the Instantiate method to instantiate the heat map point at the position of the spray patch, and apply the calculated color to the material of the point, as shown in Figure 6 .

[0069] Further, in step 6, after the project is compiled into ARM64 format using Visual Studio, the APPX installation package is generated and deployed to the mixed reality device, and the shotcrete robot operation simulation is carried out in situ in the engineering site, and the key indicators such as shotcrete thickness distribution and coverage rate are visualized in situ through the mixed reality device, as shown in Figure 7 .

[0070] The application provides a mixed reality-based guniting robot in-situ operation simulation method and system, which realizes seamless fusion of virtual guniting operation and real physical environment by combining mixed reality technology with real-time physical simulation technology. This way greatly improves the reality of guniting simulation and the real-time of operation feedback by dynamically generating a sprayed patch and precisely fitting the environment surface. Through the combination of the self-developed particle collision detection algorithm and the Unity engine, the application avoids the dependence on traditional paper drawings and the limitations of desktop simulation software, enabling construction personnel to intuitively adjust guniting parameters and verify the effect in real time in complex engineering sites (such as tunnels and construction sites), significantly improving the smoothness of construction planning and decision-making. In multiple applications, especially in tunnel support, building structure reinforcement and other scenarios, users can directly observe key indicators such as guniting thickness distribution and coverage rate through mixed reality devices, and quickly optimize the construction scheme based on a three-dimensional heat map. This method not only simplifies the trial-and-error operation process in traditional guniting technology, but also significantly reduces material waste and construction risk, improving engineering efficiency. In addition, the application of the application greatly optimizes the immersion and accuracy of on-site human-computer interaction, providing an innovative tool for industrial automation and digital construction, and promoting the in-depth application and large-scale development of intelligent construction technology in the infrastructure field.

[0071] The application is not limited to the above-mentioned embodiments, and anyone can derive other various forms of products under the inspiration of the application, but regardless of any changes in shape or structure, any technical solution with the same or similar technical solutions as the present application falls within the protection scope of the application.< / gameobject> ​​< / particlecollisionevent>

Claims

1. A method and system for simulating in-situ operation of a shotcrete robot based on mixed reality, characterized in that, The method includes the following steps: Step 1: Mixed Reality Environment Configuration: Use the Mixed Reality Toolkit (MRTK) in Unity to create a mixed reality scene for simulating in-situ operation of a shotcrete robot in a real physical environment; create a shotcrete robot model; create a SpatialMeshObserver to obtain spatial mesh data scanned by the mixed reality device, and add Mesh Collider colliders to the scanned spatial mesh; Step 2: Particle Spray Simulation: Configure the particle system in Unity to simulate the spraying effect of the spray, set the particle lifecycle, speed, size and emission rate parameters, and enable the Collision particle system collision detection module; Step 3: Spray Patch Generation and Surface Adhesion: When the sprayed particles collide with the spatial mesh, a spray patch object is generated based on the coordinates of the collision point; according to the surface normal direction of the collision point, the rotation matrix of the spray patch object is adjusted, and the offset of the normal direction is applied to make the spray patch adhere to the real physical environment. Step 4: Spray Patch Object Management: Use object pooling technology to instantiate and recycle spray patch objects; set the lifecycle of spray patch objects; Step 5: Persistent and visual analysis of spraying effect: Record the data of the sprayed patch object and generate a three-dimensional heat map of the sprayed area to achieve quantitative analysis and visual feedback of the spraying effect; Step 6: Compile, deploy, and simulate in situ: Compile the project into ARM64 format using Visual Studio and publish it as an APPX installation package, then deploy it to the mixed reality device; wear the mixed reality device in the real space to simulate the in situ operation of the shotcrete robot at the actual work site.

2. The in-situ operation simulation method and system for shotcrete robots based on mixed reality as described in claim 1, characterized in that, In step 1, the following steps can be performed sequentially: Step 1.1: Create a new scene in Unity and use the Mixed Reality Toolkit (MRTK) to create a mixed reality scene; Step 1.2: Create a model of the shotcrete robot and pre-set the model hierarchy and initial position; Step 1.3: Enable Spatial Awareness System, create SpatialMeshObserver, obtain spatial mesh data scanned by the mixed reality device, and add Mesh Collider colliders to the scanned meshes.

3. The in-situ operation simulation method and system for shotcrete robots based on mixed reality as described in claim 1, characterized in that, In step 2, the following steps can be performed sequentially: Step 2.1: Set the particle emission shape to a cone, and set the cone angle range and the initial particle velocity; Step 2.2: Activate the gravity influence module of the particle system and set the gravity coefficient to simulate the real motion trajectory of the sprayed particles under the action of gravity; Step 2.3: Bind the custom shader to the particle material and enable the collision detection module to receive particle collision event information.

4. The in-situ operation simulation method and system for shotcrete robots based on mixed reality as described in claim 1, characterized in that, In step 3, the following steps can be performed sequentially: Step 3.1: Obtain the coordinates of the collision point and the direction of the surface normal through particle collision events; Step 3.2: Calculate the rotation matrix of the spray patch object based on the surface normal direction, and apply an offset to fit the surface; Step 3.3: Assign the spray patch object to an independent spray patch layer and exclude that layer in the collision detection of the particle system.

5. The in-situ operation simulation method and system for shotcrete robots based on mixed reality as described in claim 1, characterized in that, In step 4, the following steps can be performed sequentially: Step 4.1: Initialize the spray patch object pool and preset the object pool capacity; Step 4.2: When the number of spray patch generation requests exceeds the object pool capacity, reclaim the earliest generated spray patch.

6. The in-situ operation simulation method and system for shotcrete robots based on mixed reality as described in claim 1, characterized in that, In step 5, the following steps can be performed sequentially: Step 5.1: Record the location, generation time, and texture density data of all sprayed patches; Step 5.2: Generate a three-dimensional thermal map of the sprayed area based on the spray patch data.

7. The in-situ operation simulation method and system for shotcrete robots based on mixed reality as described in claim 1, characterized in that, In step 6, the project is compiled into ARM64 format using Visual Studio to generate an APPX installation package. After being deployed to a mixed reality device, the operation of the shotcrete robot is simulated in situ at the engineering site. The shotcrete thickness distribution and key coverage indicators are visualized in situ using the mixed reality device.

Images