A motion-level planning device and method for coal mine robots based on digital cousins ​​and MR

By combining digital cousins ​​with mixed reality technology, motion planning results for coal mine robots are generated, solving the problem of low efficiency in joint planning of the robot body and robotic arm in existing technologies. This achieves efficient and visualized motion planning and human-machine collaboration, improving the adaptability and feasibility of the planning results.

CN120056111BActive Publication Date: 2026-11-10TAIYUAN UNIVERSITY OF TECHNOLOGY
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202510251811.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2025-03-04
Publication Date
2026-11-10
Estimated Expiration
2045-03-04

Smart Images

  • Figure CN120056111B_ABST
    Figure CN120056111B_ABST
Patent Text Reader

Abstract

This application discloses a motion-level planning device and method for coal mine robots based on digital cousins ​​and MR (Motion Learning), relating to the field of human-machine hybrid augmented intelligence technology. The device and method involve: an information data acquisition module to acquire scene information data; a digital cousin scene information determination module to determine the digital cousin scene information; a motion data determination module to input the digital cousin scene information into an imitation learning model to obtain motion data; a motion pre-planning module to determine the motion pre-planning result based on the motion data; an adjustment module to adjust the motion pre-planning result using a multimodal interaction method to obtain an adjusted planning result; a spatiotemporal extrapolation module to process the adjusted planning result using a preset motion planning algorithm and perform spatiotemporal extrapolation according to the time dimension to obtain a spatiotemporal extrapolation result; and a motion planning module to determine the motion planning result based on the spatiotemporal extrapolation result. This application aims to improve the efficiency and adaptability of motion planning.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of human-machine hybrid augmented intelligence technology, and in particular to a motion-level planning device and method for coal mine robots based on digital cousins ​​and MR. Background Technology

[0002] Currently, the research and application of coal mine robots has become an important part of the intelligent construction of coal mines. The initial goal of applying coal mine robots was to achieve "robot replacement of humans." However, with the deepening of research and application, limited by the level of intelligence of coal mine robots, they can only complete tasks independently in a few simple scenarios. In most cases, the deep involvement of coal mine operators is still required to collaborate with coal mine robots to complete the work tasks.

[0003] The working process of a coal mining robot can be divided into three stages: perception, decision-making, and execution. Decision-making serves as the link between perceived information and actual actions, and is the core factor determining the quality of the robot's work. Following the principle of moving from macro to micro, the operational objectives of a coal mining robot can be decomposed into three levels: task-level, behavior-level, and action-level. Among these, action-level planning is a crucial step in the decision-making process, refining high-level task objectives into specific action instructions to ensure that the coal mining robot executes the predetermined task requirements accurately.

[0004] Currently, most underground coal mines use body-manipulator collaborative robots. However, existing motion planning schemes for coal mine robots are all based on the robot's movement path or the manipulator's trajectory, without considering the robot body and manipulator as a whole for joint planning. As a result, the motion planning efficiency is low and the adaptability is poor.

[0005] Most motion planning schemes for coal mine robots rely on pure algorithm calculations, lack spatiotemporal extrapolation capabilities, cannot be pre-executed in a virtual environment, and lack intuitive visualization of planning results. They cannot consider complex spatiotemporal relationships in the planning process, nor can they quickly verify the feasibility and effectiveness of the motion planning results.

[0006] Currently, motion planning for coal mine robots based on digital twins extends the planning process to virtual space. However, establishing a digital twin environment relies on high-precision modeling, a large number of sensors, and real-time updated accurate data. This process is time-consuming, costly, and cannot provide good cross-domain generalization capabilities.

[0007] Due to the dynamic and complex nature of the underground environment and tasks, human-robot collaboration is currently the main operating mode for coal mine robots. However, in the context of human-robot collaboration, existing motion planning methods for coal mine robots do not take into account the role of humans in motion planning under the human-robot collaborative operation mode. In other fields besides coal mining, some technical solutions have introduced human involvement in robot motion planning, but these are limited to defining motion targets or providing solutions that are completely independent of the robot's autonomous planning. They do not fully leverage the advantages of human flexibility and adjustment, nor do they cleverly combine machine intelligence with human intelligence.

[0008] Therefore, improving the efficiency and adaptability of motion planning is of paramount importance. Summary of the Invention

[0009] The purpose of this application is to provide a motion planning device and method for coal mine robots based on digital cousins ​​and MR, which can improve motion planning efficiency and adaptability.

[0010] To achieve the above objectives, this application provides the following solution:

[0011] In a first aspect, this application provides a motion-level planning device for a coal mine robot based on digital cousins ​​and MR, comprising:

[0012] The information data acquisition module is used to acquire scene information data; the scene information data includes a single-frame RGB image collected in an underground coal mine.

[0013] The digital cousin scene information determination module is connected to the information data acquisition module and is used to determine the digital cousin scene information. The digital cousin scene information is determined based on the scene information data using a 3D asset library virtual model. The 3D asset library virtual model is a preset model library that covers work objects, work tools, terrain structures, mechanical equipment, and auxiliary equipment, and is geared towards underground coal mine work tasks.

[0014] The motion data determination module, connected to the digital cousin scene information determination module, is used to input the digital cousin scene information into the imitation learning model to obtain motion data. The motion data includes: coal mine robot motion pose information and robotic arm operation pose information. The imitation learning model is obtained through simulation training and zero-shot transfer based on the coal mine robot behavior model. The coal mine robot behavior model is obtained through reinforcement learning simulation training of the coal mine robot model. The coal mine robot model is a physical model constructed using 3D modeling tools based on the coal mine robot's entity parameters. The entity parameters include: size proportions, structural data, and shape data.

[0015] An action pre-planning module, connected to the action data determination module, is used to determine the action pre-planning result based on the action data;

[0016] An adjustment module, connected to the action pre-planning module, is used to adjust the action pre-planning result using a multimodal interaction method to obtain an adjusted planning result;

[0017] The spatiotemporal simulation module, connected to the adjustment module, is used to process the adjustment planning results using a preset motion planning algorithm and perform spatiotemporal simulation according to the time dimension to obtain the spatiotemporal simulation results; the spatiotemporal simulation results include the movement path of the coal mine robot and the movement trajectory of the robotic arm.

[0018] The motion planning module, connected to the spatiotemporal simulation module, is used to determine the motion planning result based on the spatiotemporal simulation result; the motion planning result includes: position coordinates, attitude information, and path points.

[0019] Secondly, this application provides a motion-level planning method for coal mine robots based on digital cousins ​​and MR, wherein the motion-level planning method for coal mine robots based on digital cousins ​​and MR is implemented using a motion-level planning device for coal mine robots based on digital cousins ​​and MR; the motion-level planning method for coal mine robots based on digital cousins ​​and MR includes:

[0020] Acquire scene information data; the scene information data includes a single-frame RGB image collected underground in a coal mine;

[0021] Determine the digital cousin scene information; the digital cousin scene information is determined based on the scene information data according to the 3D asset library virtual model; the 3D asset library virtual model is a preset model library covering work objects, work tools, terrain structure, mechanical equipment and auxiliary equipment, oriented towards coal mine underground work tasks;

[0022] The digital cousin scene information is input into the imitation learning model to obtain action data; the action data includes: the motion pose information of the coal mine robot and the operation pose information of the robotic arm; the imitation learning model is obtained by simulation training and zero-shot transfer based on the behavior model of the coal mine robot; the behavior model of the coal mine robot is obtained by reinforcement learning simulation training of the coal mine robot model; the coal mine robot model is a physical model constructed using 3D modeling tools based on the entity parameters of the coal mine robot; the entity parameters include: size proportions, structural data, and shape data;

[0023] The motion pre-planning result is determined based on the motion data;

[0024] A multimodal interaction method is used to adjust the pre-planned action results to obtain adjusted planning results;

[0025] The adjustment planning results are processed using a preset motion planning algorithm, and spatiotemporal extrapolation is performed according to the time dimension to obtain spatiotemporal extrapolation results; the spatiotemporal extrapolation results include the movement path of the coal mine robot and the movement trajectory of the robotic arm;

[0026] The motion planning result is determined based on the spatiotemporal simulation result; the motion planning result includes: position coordinates, attitude information and path points.

[0027] According to the specific embodiments provided in this application, this application has the following technical effects:

[0028] This application provides a motion-level planning device and method for coal mine robots based on digital cousins ​​and MR (Motion Learning). First, the digital cousin scenario is determined. Then, through reinforcement learning simulation training and zero-shot transfer learning, an imitation learning model is obtained to generate motion pre-planning results. The pre-planning results are further adjusted through multimodal interaction. Subsequently, spatiotemporal extrapolation and pre-execution are performed. Finally, the final motion planning result is generated, enabling the coal mine robot to perform operations. By adjusting and extrapolating the pre-planning results, the generalization ability of motion planning is improved, and based on the imitation learning model, a better planning result that balances machine intelligence and human intelligence can be obtained. Therefore, this application can improve the efficiency and adaptability of motion planning. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 A structural diagram of a motion-level planning device for a coal mine robot based on digital cousins ​​and MR.

[0031] Figure 2 A flowchart for the motion pre-planning stage of a coal mine robot based on digital cousins;

[0032] Figure 3 This is a schematic diagram illustrating the principle of digital cousin scene generation and understanding in step S1.

[0033] Figure 4 A schematic diagram illustrating the principle of integrating the predefined behavior of the coal mine robot with the reinforcement learning model in step S2;

[0034] Figure 5 This is a schematic diagram illustrating the principle of imitation learning and zero-shot transfer in step S3.

[0035] Figure 6A flowchart for the motion optimization stage of a coal mine robot based on mixed reality;

[0036] Figure 7 A schematic diagram illustrating the principle of presenting the pre-planning results for step S5;

[0037] Figure 8 A schematic diagram illustrating the principle of adjusting the planning results based on multimodal interaction in step S6;

[0038] Figure 9 This is a schematic diagram illustrating the principle of spatiotemporal deduction and pre-execution in step S7. Detailed Implementation

[0039] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0040] The related technologies disclosed include "a path planning method for coal mine search and rescue robots based on an improved A* algorithm," which optimizes the neighborhood search method, increases the search range, adjusts the heuristic function weights, applies B-spline smoothing, and introduces the D algorithm to finally optimize the path. Also disclosed is "a motion planning method and device for a robotic arm on a coal mine autonomous inspection platform," which uses obstacle object simplification processing and combines a pre-trained motion planner based on a Q-neural network to plan the robotic arm's movement operations in complex scenarios in real time.

[0041] The two solutions mentioned above provide separate motion planning methods for the coal mining robot body and the robotic arm it is equipped with. However, most current coal mining robots are in the form of body-robotic arm collaboration, and the two solutions mentioned above cannot meet the complete requirements of motion planning. In addition, both solutions are based on algorithm calculations and are not combined with the spatiotemporal extrapolation capabilities of virtual space, which makes them insufficient in dealing with complex and dynamic scenarios.

[0042] The related technology discloses a "digital twin system for a multi-functional quadruped robot in coal mines and its operation method", which includes a robot virtual planning subsystem that can perform robot path planning and synchronize the results to the robot's autonomous control module to complete the movement according to the path planning results.

[0043] While digital twin technology was used to migrate the path planning of coal mine robots from physical space to virtual space, the human factor was not considered. In a human-machine coexistence environment, human decision-making ability plays a crucial role and should be fully utilized. Furthermore, the virtual space construction based on digital twin technology is costly and has weak generalization ability, potentially making it unsuitable for new environments or conditions.

[0044] The related technology discloses "a method and system for human-machine collaborative motion planning of a mobile assembly robot". In each control cycle, the load type and pose information are obtained through the identification and positioning device. Based on the load information, the motion of the assembly robot is reasonably allocated and the end-load assembly motion trajectory is planned.

[0045] Although human involvement is introduced into the motion planning of mobile assembly robots, the robot's motion is still essentially accomplished by motion allocation and trajectory planning algorithms. The main role of humans in this process is to define the position and direction of the load at the end of the robotic arm, without actually participating in the planning of the robot's specific motion.

[0046] The related technology discloses a "human-robot collaborative path planning method for mobile robots in unstructured environments", which mixes and filters the path planned by the operator and the path planned autonomously by the robot to form a human-in-the-loop path planning.

[0047] However, the robot's autonomous planning and the operator's path planning are carried out simultaneously and independently, failing to fully utilize the robot's global computing advantages and the operator's intuitive judgment. A more ideal approach would be to first derive a preliminary optimal solution through the robot's computing power, and then have the operator make flexible adjustments based on their rich intuition and experience.

[0048] Digital cousins ​​are a novel concept proposed by Professor Fei-Fei Li's team at Stanford University. It retains the advantages of digital twins while significantly reducing the cost of generating simulated environments from real-world settings, and simultaneously improving the generalization ability of learning. Mixed reality (MR) technology can serve as an interface for deep human involvement in robot motion planning, thereby achieving an organic integration of machine intelligence and human intelligence in robot motion planning. Combining the digital cousin concept with MR technology holds promise for providing a new approach to motion-level planning for coal mine robots.

[0049] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0050] In one exemplary embodiment, such as Figure 1 As shown, a motion-level planning device for a coal mine robot based on digital cousin and MR is provided. The motion-level planning device for a coal mine robot based on digital cousin and MR includes: an information data acquisition module, a digital cousin scene information determination module, a motion data determination module, a motion pre-planning module, an adjustment module, a spatiotemporal extrapolation module, and a motion planning module.

[0051] The information data acquisition module is used to acquire scene information data; the scene information data includes single-frame RGB images collected underground in the coal mine.

[0052] The digital cousin scene information determination module is connected to the information data acquisition module. The digital cousin scene information determination module is used to determine the digital cousin scene information. The digital cousin scene information is determined based on the scene information data using the 3D asset library virtual model. The 3D asset library virtual model is a pre-set model library covering work objects, work tools, terrain structures, mechanical equipment, and auxiliary equipment, oriented towards underground coal mine work tasks.

[0053] The motion data determination module is connected to the digital cousin scene information determination module. The motion data determination module is used to input the digital cousin scene information into the imitation learning model to obtain motion data. The motion data includes: the motion pose information of the coal mine robot and the operation pose information of the robotic arm. The imitation learning model is obtained by simulation training and zero-shot transfer based on the behavior model of the coal mine robot. The behavior model of the coal mine robot is obtained by reinforcement learning simulation training of the coal mine robot model. The coal mine robot model is a physical model built using 3D modeling tools based on the entity parameters of the coal mine robot. The entity parameters include: size scale, structural data and shape data.

[0054] The motion pre-planning module is connected to the motion data determination module. The motion pre-planning module is used to determine the motion pre-planning result based on the motion data.

[0055] The adjustment module is connected to the action pre-planning module. The adjustment module is used to adjust the action pre-planning results using a multimodal interaction method to obtain the adjusted planning results.

[0056] The spatiotemporal simulation module is connected to the adjustment module. The spatiotemporal simulation module is used to process the adjustment planning results using a preset motion planning algorithm and perform spatiotemporal simulation according to the time dimension to obtain the spatiotemporal simulation results. The spatiotemporal simulation results include the movement path of the coal mine robot and the movement trajectory of the robotic arm.

[0057] The motion planning module is connected to the spatiotemporal simulation module. The motion planning module is used to determine the motion planning results based on the spatiotemporal simulation results. The motion planning results include: position coordinates, attitude information, and path points.

[0058] In one embodiment, the digital cousin scene information determination module includes: a semantic segmentation submodule, a label annotation submodule, a point cloud data determination submodule, a semantic similarity determination submodule, a scene construction submodule, and a matching submodule.

[0059] The semantic segmentation submodule is connected to the information data acquisition module. The semantic segmentation submodule is used to perform semantic segmentation on scene information data to obtain segmented information data.

[0060] The labeling submodule is connected to the semantic segmentation submodule. The labeling submodule is used to label the segmented information data with objects using a large language model to obtain the labeling information.

[0061] The point cloud data determination submodule is connected to the label annotation submodule. The point cloud data determination submodule is used to determine the point cloud data based on the annotation information; the point cloud data includes three-dimensional position information.

[0062] The semantic similarity determination submodule is connected to the point cloud data determination submodule. The semantic similarity determination submodule is used to determine semantic similarity based on the annotation information and the virtual model of the 3D asset library.

[0063] The scene building submodule is connected to the semantic similarity determination submodule. The scene building submodule is used to build digital cousin scenes using the Unity3d engine.

[0064] The matching submodule is connected to the scene construction submodule. The matching submodule is used to perform digital kinship matching in the digital kinship scene based on point cloud data and semantic similarity to obtain digital kinship scene information.

[0065] This application also provides a motion-level planning method for coal mine robots based on digital cousins ​​and MR. This method is implemented using a motion-level planning device for coal mine robots based on digital cousins ​​and MR. The motion-level planning method for coal mine robots based on digital cousins ​​and MR includes:

[0066] Acquire scene information data. Scene information data includes single-frame RGB images captured underground in a coal mine.

[0067] Determine the digital cousin scene information. The digital cousin scene information is determined based on the scene information data using the 3D asset library virtual model; the 3D asset library virtual model is a pre-set model library covering work objects, work tools, terrain structures, mechanical equipment, and auxiliary equipment, oriented towards underground coal mine work tasks.

[0068] As an optional implementation method, determining digital cousin scenario information specifically includes:

[0069] The scene information data is semantically segmented to obtain segmented information data; a large language model is used to annotate the segmented information data with object labels to obtain annotation information; point cloud data is determined based on the annotation information; the point cloud data includes three-dimensional position information.

[0070] Based on the annotation information and the virtual model of the 3D asset library, semantic similarity is determined; a digital cousin scene is constructed using the Unity 3D engine; based on point cloud data and semantic similarity, digital cousin matching is performed within the digital cousin scene to obtain digital cousin scene information.

[0071] The digital parent-child scene information is input into the imitation learning model to obtain motion data. The motion data includes: the motion pose information of the coal mining robot and the operational pose information of the robotic arm; the imitation learning model is obtained through simulation training and zero-shot transfer based on the coal mining robot's behavior model; the coal mining robot's behavior model is obtained through reinforcement learning simulation training of the coal mining robot model; the coal mining robot model is a physical model constructed using 3D modeling tools based on the coal mining robot's entity parameters; entity parameters include: size scale, structural data, and shape data.

[0072] Methods for determining the behavior model of coal mine robots include:

[0073] Acquire task data; task data refers to the operational data of the coal mining robot when performing tasks. A reinforcement learning simulation environment is constructed using the ML-Agents toolkit; the task data is decomposed to obtain basic behavioral data, including operational data for movement, rotation, grasping, and placement. Using a reinforcement learning algorithm and based on a defined reinforcement learning reward and punishment mechanism, the coal mining robot model is optimized and trained on multiple tasks within the reinforcement learning simulation environment based on the basic behavioral data, resulting in a coal mining robot behavior model.

[0074] As an optional implementation method, the method for determining the imitation learning model includes:

[0075] Obtain a demonstration dataset; the demonstration dataset contains the state, actions, and reward data of the coal mine robot behavior model in different scenarios; filter the demonstration dataset to obtain filtered data; based on the filtered data and the coal mine robot behavior model, perform simulation training in Unity 3D by adding domain randomization, and then transfer the trained model to the coal mine robot to obtain the imitation learning model; domain randomization includes: randomly changing lighting, object color, and object friction coefficient.

[0076] Determine the motion pre-planning results based on motion data.

[0077] A multimodal interaction method is used to adjust the pre-planning results of actions, resulting in an adjusted planning result.

[0078] In one embodiment, a multimodal interaction method is used to adjust the action pre-planning result to obtain an adjusted planning result, specifically including:

[0079] By using a mixed reality headset to perform interference analysis on the motion pre-planning results, it is determined whether adjustments should be made, and the results are obtained.

[0080] If the judgment result is yes, then the gaze interaction in multimodal interaction is used to select the path point in the space of the action pre-planning result as the movement path point.

[0081] Based on gesture interaction in multimodal interaction, the motion pre-planning results are interpolated and marked to determine the key points for optimizing the robot arm trajectory and obtain the robot arm trajectory adjustment points.

[0082] Based on voice interaction in multimodal interaction, the motion optimization is confirmed according to the movement path points and the adjustment points of the robotic arm trajectory, and the adjustment planning results are obtained.

[0083] The pre-defined motion planning algorithm is used to process the adjusted planning results, and spatiotemporal extrapolation is performed according to the time dimension to obtain the spatiotemporal extrapolation results. The spatiotemporal extrapolation results include the movement path of the coal mining robot and the movement trajectory of the robotic arm. The pre-defined motion planning algorithms include: the A* algorithm and the RRT-Connect algorithm; wherein, the A* algorithm is used to determine the movement path of the coal mining robot, and the RRT-Connect algorithm is used to determine the movement trajectory of the robotic arm.

[0084] In one embodiment, a preset action planning algorithm is used to process the adjustment planning results, and spatiotemporal extrapolation is performed according to the time dimension to obtain spatiotemporal extrapolation results, specifically including:

[0085] The pre-defined motion planning algorithm is used to optimize the adjustment planning results to obtain the motion optimization results; based on the motion optimization results, spatiotemporal extrapolation is performed according to the time dimension to obtain the spatiotemporal extrapolation results.

[0086] The motion planning results are determined based on the spatiotemporal simulation results. The motion planning results include: position coordinates, attitude information, and waypoints.

[0087] The determination of action planning results based on spatiotemporal simulation results specifically includes:

[0088] The spatiotemporal simulation results are used to determine whether they match the expected action information. If so, the spatiotemporal simulation results are converted into data according to the preset standard instruction format to obtain converted data. The action planning results are then determined based on the converted data.

[0089] This application also provides an application scenario in which the above-mentioned motion-level planning method for coal mine robots based on digital cousins ​​and MR is applied. Specifically, the motion-level planning method for coal mine robots based on digital cousins ​​and MR provided in this embodiment can be applied in underground coal mine transportation scenarios. The underground coal mine transportation scenario includes an information acquisition stage, a motion planning stage, and a transportation execution stage. Environmental information (scenario information) collected underground enters the motion planning stage from the information acquisition stage. Through reinforcement learning and imitation learning in a digital cousin environment, the motion planning stage achieves pre-planning of the coal mine robot's actions. The pre-planning results are then optimized to obtain the corresponding motion planning results, which are then entered into the downstream transportation execution stage. The robot moves according to the motion planning results to realize underground coal mine transportation operations. The motion-level planning method for coal mine robots based on digital cousins ​​and MR provided in this embodiment belongs to the process of motion pre-planning, adjustment, and optimization within the motion planning stage.

[0090] like Figure 2 As shown, the technical concept of the method adopted in this application mainly includes two stages: pre-planning of coal mine robot movements based on digital cousins ​​and optimization of coal mine robot movements based on mixed reality.

[0091] The pre-planning stage of motion for coal mine robots based on digital cousins ​​includes the following steps.

[0092] S1: Digital Cousin Scene Generation and Understanding. The principle is detailed in the attached document. Figure 3 As shown.

[0093] S101: Scene data collection and object label generation. Single-frame RGB images are collected from real underground coal mine scenes; a pre-trained semantic segmentation model is used to generate masks of objects related to the task in the images; a large language model is used to label each object in the image, and the descriptive information is passed to the semantic segmentation model to ensure that each object has a unique label.

[0094] The semantic segmentation model is Grounded SAM 2, which can combine language and visual input to identify and segment complex objects, making it suitable for semantic segmentation in complex scenarios. The large language model is GPT-4o, which inherits the core capabilities of GPT-4 and has advantages in speed and cost.

[0095] S102: Object Point Cloud Generation and State Estimation. A monocular depth estimation model is used to generate a depth map of the scene, thereby obtaining the 3D position information of each object; the depth map is converted into point cloud data, and combined with the object's mask information, a 3D point cloud representation of each object is generated; the pose and size of the object are estimated based on the point cloud data, preparing for subsequent virtual object matching and placement.

[0096] The monocular depth estimation model is DepthAnythingV2, which captures layer and depth changes more accurately than other depth estimation models.

[0097] S103: Digital Cousin Matching Based on Language-Visual Model Fusion. This involves calculating the semantic similarity between real-world object labels (annotation information) and virtual models in a 3D asset library using a multimodal model, thus performing preliminary category filtering of the 3D asset library. A visual feature embedding model is then used to calculate the geometric feature distance between each real-world object and the virtual model, determining the semantic similarity based on the annotation information and the virtual models in the 3D asset library. Virtual objects with the closest shape and semantics are selected as digital cousin objects.

[0098] The 3D asset library is a pre-built model library for underground coal mine operations, covering the work objects, tools, terrain structures, machinery and equipment, and auxiliary facilities involved in underground coal mine operations.

[0099] The multimodal model is CLIP, which has the ability to understand textual descriptions and establish associations with visual information; the visual feature embedding model is DINOv2, which has high-quality visual feature embedding capabilities and can accurately capture the geometric and semantic features of objects.

[0100] S104: Digital Cousin Scene Generation and Setup. A digital cousin scene is created using the Unity 3D engine. Based on the depth information and spatial position of each object, the corresponding digital cousin objects are placed in their respective positions within the scene, and the object sizes are adjusted to match real-world dimensions. Appropriate physical properties are set for the digital cousin objects in the scene to ensure their physical behavior is consistent with that of actual objects. In other words, based on point cloud data and semantic similarity, digital cousin matching is performed within the digital cousin scene to obtain the digital cousin scene information.

[0101] Physical properties are set using rigid bodies, colliders, and joint components in the Unity 3D engine.

[0102] S105: Enhanced generalization ability of digital cousin scenes. Randomize the appearance of objects using the Unity 3D engine's shaders and material library to improve the robot's visual generalization ability; moderately randomize the position and size of objects in the Unity 3D engine to improve the robot's task adaptability under different layouts and positions; add random obstacles or dynamic elements in Unity3D to further enhance the diversity of the scene.

[0103] S2: Integration of predefined behavior and reinforcement learning model for coal mine robots; its principle is as follows: Figure 4 As shown.

[0104] S201: Coal Mine Robot Model Construction. Create a coal mine robot model using 3D modeling tools, ensuring the model's size, structure, and shape are completely consistent with the physical entity of the coal mine robot, and integrate it into the digital scene; use the joint components of the Unity 3D engine to control the rotation and range of motion of each moving part of the coal mine robot model; add a physics engine to the coal mine robot model to ensure it conforms to actual motion laws.

[0105] The 3D modeling tool is Blender, which integrates the modeling and rendering workflow and supports FBX, OBJ and STL formats, making it easy to integrate seamlessly with Unity3d.

[0106] The joint assembly includes a Configurable Joint, a Hinge Joint, a Fixed Joint, and a Spring Joint. The Configurable Joint is used to control the complex translation and rotation of the coal mining robot body, the Hinge Joint is used to control the rotation of the robotic arm joints, the Fixed Joint is used to fix the body to the robotic arm base, and the Spring Joint is used to control the grasping of the robotic arm end effector.

[0107] S202: Building a Reinforcement Learning Simulation Environment. Install and configure the ML-Agents toolkit, and attach the core scripts to the coal mine robot model and digital cousin environment; in the training configuration YAML file, select an appropriate reinforcement learning algorithm and set the hyperparameters; establish Socket communication between the mlagents Python library and Unity3d to achieve real-time data interaction between the Python training and the Unity3d simulation environment.

[0108] The core scripts include the Agent script and the EnvironmentManager script. The Agent script is attached to the robot model and is responsible for the robot model's behavior and learning. The EnvironmentManager script is attached to the digital parent environment and is responsible for the overall management and coordination of the environment.

[0109] The reinforcement learning algorithm is the Deep Deterministic Policy Gradient (DDPG) algorithm, which is suitable for continuous action spaces and has good results in robot training.

[0110] S203: Definition of Task Skills and Reward / Punishment Mechanisms. The complex tasks of the coal mining robot are broken down into basic behaviors. For each behavior, a reinforcement learning reward / punishment mechanism is defined and written into the OnActionReceived() method. The OnActionReceived() method is called in the AddReward() or EndEpisode() method to provide the robot with corresponding feedback for each success or failure. Basic behaviors include movement, rotation, grasping, and placement.

[0111] The reinforcement learning reward and punishment mechanism specifically rewards the coal mine robot agent when it successfully completes a behavior, and punishes it when it fails to complete the behavior or collides with an obstacle.

[0112] S204: Behavioral Model Training and Multi-Task Learning Optimization. Training is initiated using a reinforcement learning framework. Through repeated execution and reward feedback, the policy of each behavioral model is continuously optimized. Under the multi-task learning framework, a scheduling mechanism is used to switch between different behaviors, ensuring that the reinforcement learning model achieves good policy optimization results on each behavioral task. The reinforcement learning framework is TensorFlow or PyTorch.

[0113] S205: Demonstration Data Generation and Storage. Run the trained model strategy in a simulation environment, placing the coal mine robot model in diverse scenarios, recording the state, actions, and reward data for each scenario to generate a complete demonstration dataset. Store the generated dataset in a structured format in a database to ensure that the demonstration data can be directly accessed and used during subsequent imitation learning or training adjustments.

[0114] The database is InfluxDB, which is a time-series database that can handle large amounts of timestamp data and is suitable for the continuous data streams generated by robotic agents during training.

[0115] S3: Imitation learning and zero-shot transfer; its principle is as follows: Figure 5 As shown.

[0116] S301: Selection and optimization of the demonstration dataset. In the demonstration dataset, task completion rate, path length, action smoothness, and task completion time are weighted and scored. A scoring threshold is set, and only data above this threshold is retained. Data augmentation methods are used to optimize the demonstration data, including adding different environmental conditions or location variations to ensure that the data covers more scenarios and boundary situations, thereby improving the model's generalization ability.

[0117] S302: Imitation Learning Model Training. Using a filtered and optimized demonstration dataset (i.e., the filtered data), train the robot's imitation learning model in Unity 3D. Input state-action pairs into the imitation learning model so it learns how to imitate the operations in the demonstration data. Add domain randomization during simulation training to adapt the model to a wider range of physical and visual conditions, improving the robustness of the imitation learning model in real-world environments. Specifically, based on the filtered data and the coal mine robot's behavior model, perform simulation training in Unity 3D by adding domain randomization. Domain randomization includes randomly varying lighting, object colors, and object friction coefficients.

[0118] S303: Model Performance Evaluation and Optimization. Test the model's performance in a simulation environment to ensure it can accurately perform tasks under different conditions, with a focus on evaluating the model's adaptability in domain-randomized environments. Based on the test results, make necessary fine-tuning to ensure its stability in key task steps and changing environments, in order to further optimize the model's generalization effect.

[0119] S304: Zero-Shot Transfer and Domain Adaptation. The simulation-trained model is directly transferred to the actual coal mining robot without additional real-world training, directly testing the model's initial performance. This involves transferring the trained model to the coal mining robot to obtain an imitation learning model. During the transfer process, the model's performance in the real environment is observed. If deviations occur, domain adaptation can be applied to make adaptive adjustments to the model.

[0120] S4: Generation of motion pre-planning results. S401: Model deployment and interface configuration. The trained imitation learning model is loaded and deployed to the host computer of the coal mining robot so that the model can run in the actual environment; an appropriate communication protocol is selected to configure the communication interface between the robot's host computer and the imitation learning model to ensure that the robot can obtain the instructions generated by the model in real time. The communication protocol is ROS.

[0121] S402: Environmental Perception and Data Transmission Test. Start and initialize the coal mine robot sensor system to ensure that the robot can acquire environmental information in real time, providing accurate environmental data for subsequent motion planning; before the actual task, conduct a data transmission test to ensure that the model can receive the coal mine robot sensor data and output the corresponding motion planning results.

[0122] The sensor system for coal mine robots includes an IMU, a 3D lidar, a depth camera, and a visible light camera.

[0123] S403: Motion pre-planning result generation and formatting. Based on the input task objectives and the coal mining robot's perception of the work object, tools, and environmental conditions, a complete movement path and robotic arm trajectory required for the coal mining robot to complete the task are generated using an imitation learning model. A series of motion data are output, including the pose information of the coal mining robot's movement and the robotic arm's operation.

[0124] S404: Formatting and storing motion pre-planning results. Organize motion data into an array, where each node contains position coordinates, orientation angle, timestamp, and other auxiliary information; save the array-based motion data as a JSON file to a specific location for transmission to the mixed reality program during subsequent motion tuning phases.

[0125] like Figure 6 As shown, the motion optimization stage of a coal mine robot based on mixed reality includes the following steps.

[0126] S5: Presentation of motion pre-planning results, the principle of which is as follows: Figure 7 As shown.

[0127] S501: Basic development of mixed reality program. Import and configure the MRTK toolkit in the Unity 3D engine, select OpenXR as the XR development plugin, and add the Microsoft HoloLens function group to it; import the coal mine robot model built in S201 into the scene, and modify it into a semi-transparent material coal mine robot AR model to ensure the virtual-real fusion effect of the mixed reality program.

[0128] S502: Integration of Mixed Reality Program Functions. Gesture interaction, gaze interaction, and voice interaction functions are added through the OpenXR interaction configuration file to allow coal mine operators to subsequently optimize motion planning results; Vuforia Engine is imported into the mixed reality program, using the coal mine robot model as the Model Target; the LineRenderer component is integrated into the coal mine robot AR model to support motion trajectory rendering.

[0129] Gesture interaction is achieved by detecting gestures using the HandJointUtils() and GestureRecognizer() classes.

[0130] The gaze interaction uses the Focusable component to make objects respond to gaze. The voice interaction uses the SpeechCommand() class to define recognizable voice commands.

[0131] S503: Mixed Reality Application Deployment and Spatial Alignment. The developed mixed reality application is packaged into a UWP platform application and deployed to a mixed reality headset via Visual Studio, worn by the coal mine operator. When a physical entity of the coal mine robot is detected in the mixed reality headset's field of view, a Model Target is loaded, precisely aligning the coal mine robot model with the physical entity's pose. The mixed reality headset is a Microsoft HoloLens 2.

[0132] S504: Pre-planning result rendering and interaction optimization. The motion data from the JSON file stored in S404 is loaded into the mixed reality program. The Line Renderer component is used to draw the pre-planned results of the coal mine robot's movement path and robotic arm trajectory, and interpolation points are marked at key locations. A ManipulationHandler component is added to each interpolation point object, enabling it to be dragged via gesture interaction.

[0133] S6: Adjustment of planning results based on multimodal interaction; its principle is as follows: Figure 8 As shown.

[0134] S601: Observation and Analysis of Pre-planning Results. The coal mine operator observes the pre-planning results of the coal mine robot's movements using a mixed reality headset to determine whether the coal mine robot can accurately execute the work tasks according to the planning results, and whether the movement path and robotic arm interfere with obstacles in the environment or the next action of the coal mine operator. In turn, the operator analyzes whether the pre-planning results need to be adjusted and how to adjust them.

[0135] S602: Waypoint selection based on gaze interaction. Coal mine operators select multiple key points in space through gaze interaction as waypoints for optimizing the movement path of the coal mine robot; to ensure the accuracy of point selection and avoid misoperation, the system is designed so that when the coal mine operator's gaze stays on a target point for more than 3 seconds, the point is considered to be confirmed as selected.

[0136] S603: Gesture-based adjustment of robotic arm trajectory points. Coal mine operators use gestures to move interpolation points marked on the pre-planned trajectory of the robotic arm, which serve as key points for optimizing the trajectory. When the coal mine operator adjusts the interpolation points, the workspace of the robotic arm is displayed in the mixed reality program to avoid adjusting the points to positions that the robotic arm cannot reach.

[0137] S604: Voice-based motion optimization confirmation. After completing the selection of waypoints for the coal mine robot and the adjustment of the robotic arm trajectory points, the coal mine operator issues a replanning command via voice. To avoid misidentification, the replanning command must be confirmed again via voice.

[0138] S7: Spatiotemporal deduction and pre-execution; its principle is as follows: Figure 9 As shown.

[0139] S701: Coal Mine Robot Motion Optimization. Records the pose information of the coal mine robot's waypoints and robotic arm trajectory points after the coal mine operator sets and adjusts them, and transmits them to the auxiliary computing device. The trajectory is replanned by the preset motion planning algorithm deployed in the auxiliary computing device, and the replanned result is presented in the mixed reality headset.

[0140] The auxiliary computing device is an embedded edge computing box, housed in an explosion-proof enclosure, which can be carried by coal mine operators on their backs. The preset motion planning algorithm is a combination of the A* algorithm and the RRT-Connect algorithm, where the A* algorithm is used for movement path planning and the RRT-Connect algorithm is used for robotic arm trajectory planning.

[0141] Furthermore, the communication protocol between the auxiliary computing device and the mixed reality headset is Socket.

[0142] S702: Spatiotemporal simulation of motion optimization results. Based on the motion optimization results of the coal mining robot, a spatiotemporal simulation is performed on the auxiliary computing device, that is, the motion process of the coal mining robot body and the robotic arm is pre-executed step by step according to the time dimension in the spatiotemporal simulation scenario.

[0143] S703: Spatiotemporal simulation result push. Sends the spatiotemporal simulation results to the mixed reality program in the mixed reality headset to generate a visual preview of the coal mining robot's movement path and the robotic arm's trajectory, displaying the expected pose of the coal mining robot and robotic arm at each moment.

[0144] S8: Generation and transmission of planning results.

[0145] S801: Motion optimization result analysis and re-optimization. The coal mine operator judges whether the motion optimization results are reasonable and feasible, and whether there are any actions that do not meet expectations, based on the spatiotemporal simulation results in the mixed reality headset. If further adjustments to the motion optimization results are still needed, the coal mine operator can repeatedly execute S602-S703 until it is ensured that the path fully meets the requirements before execution.

[0146] S802: Path and Trajectory Data Conversion. Export the final confirmed coal mine robot movement path and robotic arm trajectory data, i.e., motion planning results, from the hybrid display program, including position coordinates, attitude information, and critical path points, and convert them into an instruction format that conforms to the robot control system parsing standard.

[0147] S803: Command Transmission and Execution. The formatted coal mine robot path and robotic arm trajectory data, i.e., the motion planning results, are transmitted to the robot control system through the configured communication interface. The data is further parsed into movement commands for the robot body and fine operation commands for the robotic arm, ensuring that the robot completes the task accurately and efficiently according to the motion planning results and task requirements.

[0148] The beneficial effects of this application are:

[0149] (1) The motion-level planning of the coal mine robot is divided into two stages: motion pre-planning and motion optimization. The first stage realizes the motion pre-planning of the coal mine robot through reinforcement learning and imitation learning in a digital parent environment. The second stage realizes the further optimization of the pre-planning results by the coal mine operator based on the first stage using mixed reality as the interface. This fully utilizes the global computing and macro planning capabilities of machine intelligence, and also reflects the flexible adaptability and local optimization advantages of human intelligence.

[0150] (2) In the pre-planning stage of the action of the coal mine robot, the concept of digital cousin is introduced to extend the data of the physical space to the simulation environment for learning. Compared with digital twin, digital cousin does not pursue perfect reconstruction of the physical space in all the tiny details, but focuses on preserving higher-level details, such as spatial relationships and semantic information between objects. This not only greatly reduces the generation of virtual environment, but also helps to improve the cross-domain generalization ability of learning strategies.

[0151] (3) The pre-planning model of the coal mine robot in the digital cousin environment is established by using reinforcement learning algorithm. It can regard the state of the coal mine robot body and the robotic arm as a joint state space for joint motion planning. While simplifying the motion planning process, it enhances the task adaptability and versatility of the pre-planning model. In addition, it can achieve effective motion planning through interactive training, thereby saving the complex kinematic analysis and modeling process and improving planning efficiency.

[0152] (4) Mixed reality technology is used as the interface and channel for coal mine operators to participate deeply in the motion planning of coal mine robots. Its virtual-real fusion characteristics and its good support for multimodal interaction are fully utilized to enhance the active role of humans in the motion planning of coal mine robots, strengthen the dynamic interaction and feedback between humans and machines in human-machine collaborative planning, and enable humans and machines to make the final motion planning results of coal mine robots more flexible and reliable in the complex and dynamic environment of underground coal mines.

[0153] (5) In the motion optimization of coal mine robots based on mixed reality, the auxiliary computing device carried by the coal mine operator can perform spatiotemporal simulation of the motion planning results optimized by the coal mine operator in the virtual scene, and present the simulation results intuitively in the mixed reality headset, which is in line with the coal mine operator's intuition, reduces the understanding cost of the coal mine operator, and enables him to better focus on the motion optimization process; in addition, the deployment location of the auxiliary computing device can reduce communication latency and speed up the planning process.

[0154] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0155] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A motion-level planning device for coal mine robots based on digital cousins ​​and MR, characterized in that, The motion-level planning device for coal mine robots based on digital cousins ​​and MR includes: The information data acquisition module is used to acquire scene information data; the scene information data includes a single-frame RGB image collected in an underground coal mine. The digital cousin scene information determination module is connected to the information data acquisition module and is used to determine the digital cousin scene information. The digital cousin scene information is determined based on the scene information data using a 3D asset library virtual model. The 3D asset library virtual model is a preset model library that covers work objects, work tools, terrain structures, mechanical equipment, and auxiliary equipment, and is geared towards underground coal mine work tasks. The motion data determination module, connected to the digital cousin scene information determination module, is used to input the digital cousin scene information into the imitation learning model to obtain motion data. The motion data includes: coal mine robot motion pose information and robotic arm operation pose information. The imitation learning model is obtained through simulation training and zero-shot transfer based on the coal mine robot behavior model. The coal mine robot behavior model is obtained through reinforcement learning simulation training of the coal mine robot model. The coal mine robot model is a physical model constructed using 3D modeling tools based on the coal mine robot's entity parameters. The entity parameters include: size proportions, structural data, and shape data. An action pre-planning module, connected to the action data determination module, is used to determine the action pre-planning result based on the action data; An adjustment module, connected to the action pre-planning module, is used to adjust the action pre-planning result using a multimodal interaction method to obtain an adjusted planning result; The spatiotemporal simulation module, connected to the adjustment module, is used to process the adjustment planning results using a preset motion planning algorithm and perform spatiotemporal simulation according to the time dimension to obtain the spatiotemporal simulation results; the spatiotemporal simulation results include the movement path of the coal mine robot and the movement trajectory of the robotic arm. The motion planning module, connected to the spatiotemporal simulation module, is used to determine the motion planning result based on the spatiotemporal simulation result; the motion planning result includes: position coordinates, attitude information, and path points.

2. The motion-level planning device for coal mine robots based on digital cousins ​​and MR according to claim 1, characterized in that, The digital cousin scenario information determination module includes: The semantic segmentation submodule, connected to the information data acquisition module, is used to perform semantic segmentation on the scene information data to obtain segmented information data. The labeling submodule, connected to the semantic segmentation submodule, is used to label the segmented information data with object labels using a large language model to obtain labeling information. A point cloud data determination submodule, connected to the label annotation submodule, is used to determine point cloud data based on the annotation information; the point cloud data includes three-dimensional position information. The semantic similarity determination submodule, connected to the point cloud data determination submodule, is used to determine the semantic similarity based on the annotation information and the 3D asset library virtual model; The scene construction submodule, connected to the semantic similarity determination submodule, is used to construct a digital cousin scene using the Unity 3D engine; The matching submodule, connected to the scene construction submodule, is used to perform digital kinship matching within the digital kinship scene based on the point cloud data and the semantic similarity, to obtain digital kinship scene information.

3. A motion-level planning method for coal mine robots based on digital cousins ​​and MR, characterized in that, The motion-level planning method for coal mine robots based on digital cousins ​​and MR is implemented using the motion-level planning device for coal mine robots based on digital cousins ​​and MR as described in any one of claims 1-2. The motion-level planning method for coal mine robots based on digital cousins ​​and MR includes: Acquire scene information data; the scene information data includes a single-frame RGB image collected underground in a coal mine; Determine the digital cousin scene information; the digital cousin scene information is determined based on the scene information data according to the 3D asset library virtual model; the 3D asset library virtual model is a preset model library covering work objects, work tools, terrain structure, mechanical equipment and auxiliary equipment, oriented towards coal mine underground work tasks; The digital cousin scene information is input into the imitation learning model to obtain action data; the action data includes: the motion pose information of the coal mine robot and the operation pose information of the robotic arm; the imitation learning model is obtained by simulation training and zero-shot transfer based on the behavior model of the coal mine robot; the behavior model of the coal mine robot is obtained by reinforcement learning simulation training of the coal mine robot model; the coal mine robot model is a physical model constructed using 3D modeling tools based on the entity parameters of the coal mine robot; the entity parameters include: size proportions, structural data, and shape data; The motion pre-planning result is determined based on the motion data; A multimodal interaction method is used to adjust the pre-planned action results to obtain adjusted planning results; The adjustment planning results are processed using a preset motion planning algorithm, and spatiotemporal extrapolation is performed according to the time dimension to obtain spatiotemporal extrapolation results; the spatiotemporal extrapolation results include the movement path of the coal mine robot and the movement trajectory of the robotic arm; The motion planning result is determined based on the spatiotemporal simulation result; the motion planning result includes: position coordinates, attitude information and path points.

4. The motion-level planning method for coal mine robots based on digital cousins ​​and MR according to claim 3, characterized in that, Determine the information for the digital cousin scenario, specifically including: The scene information data is semantically segmented to obtain segmented information data; A large language model is used to annotate the segmented information data with object labels to obtain annotation information; Point cloud data is determined based on the annotation information; the point cloud data includes three-dimensional position information. Based on the annotation information and the virtual model of the 3D asset library, the semantic similarity is determined; A digital cousin scene was built using the Unity 3D engine; Based on the point cloud data and the semantic similarity, digital cousin matching is performed within the digital cousin scenario to obtain digital cousin scenario information.

5. The motion-level planning method for coal mine robots based on digital cousins ​​and MR according to claim 3, characterized in that, A multimodal interaction method is used to adjust the action pre-planning result to obtain an adjusted planning result, specifically including: The motion pre-planning results are analyzed using a mixed reality headset to determine whether adjustments are needed, and a judgment result is obtained. If the judgment result is yes, then the gaze interaction in multimodal interaction is used to select a path point in the space of the action pre-planning result as the path point for movement. Based on gesture interaction in multimodal interaction, the pre-planning results of the action are interpolated and marked to determine the key points for optimizing the robot arm trajectory and obtain the robot arm trajectory adjustment points. Based on voice interaction in multimodal interaction, the motion optimization is confirmed according to the movement path points and the adjustment points of the robotic arm trajectory, and the adjustment planning results are obtained.

6. The motion-level planning method for coal mine robots based on digital cousins ​​and MR according to claim 3, characterized in that, The adjustment planning results are processed using a preset action planning algorithm, and spatiotemporal extrapolation is performed according to the time dimension to obtain spatiotemporal extrapolation results, specifically including: The adjustment planning results are optimized using a preset action planning algorithm to obtain the action optimization results; Based on the action optimization results, a spatiotemporal extrapolation is performed along the time dimension to obtain the spatiotemporal extrapolation results.

7. The motion-level planning method for coal mine robots based on digital cousins ​​and MR according to claim 3, characterized in that, The action planning results are determined based on the spatiotemporal simulation results, specifically including: Based on the spatiotemporal deduction results, determine whether they match the expected action information; If so, the spatiotemporal deduction results are converted according to a preset standard instruction format to obtain converted data; The motion planning result is determined based on the transformed data.

8. The motion-level planning method for coal mine robots based on digital cousins ​​and MR according to claim 3, characterized in that, The method for determining the behavior model of the coal mine robot includes: Acquire task data; the task data refers to the operational data of the coal mining robot when performing tasks. A reinforcement learning simulation environment was built using the ML-Agents toolkit; The task data is decomposed to obtain basic behavioral data; the basic behavioral data includes operation data corresponding to movement, rotation, grasping, and placement. Using a reinforcement learning algorithm and based on a set reinforcement learning reward and punishment mechanism, the coal mine robot model is trained on a multi-task basis in the reinforcement learning simulation environment according to the basic behavioral data to obtain the coal mine robot behavior model.

9. The motion-level planning method for coal mine robots based on digital cousins ​​and MR according to claim 3, characterized in that, The method for determining the imitation learning model includes: Obtain a demonstration dataset; the demonstration dataset consists of the state, actions, and reward data of the coal mine robot behavior model in different scenarios. The sample dataset is filtered to obtain filtered data; Based on the filtered data and the behavior model of the coal mine robot, simulation training is performed in Unity 3D by adding domain randomization, and the trained model is then transferred to the coal mine robot to obtain the imitation learning model; the domain randomization includes: randomly changing the lighting, object color, and object friction coefficient.

10. The motion-level planning method for coal mine robots based on digital cousins ​​and MR according to claim 3, characterized in that, The preset motion planning algorithms include the A* algorithm and the RRT-Connect algorithm; the A* algorithm is used to determine the movement path of the coal mining robot, and the RRT-Connect algorithm is used to determine the movement trajectory of the robotic arm.

Citation Information

Patent Citations

  • Digital twin evolution mechanism and method for intelligent mine scene

    CN113516903A

  • Underground coal mine multifunctional quadruped robot digital twinning system and operation method thereof

    CN115329922A