Mine robot remote control system and method based on augmented reality

Abstract

The application discloses a kind of mine robot remote control system and method based on augmented reality, belong to the cross technical field of mine rescue and robot control. Through SLAM algorithm, the high-precision virtual-real registration of robot back data and mine digital twin model is realized, and the real environment and virtual information are integrated rendered in AR field of view;Intuitive man-machine interaction is realized by using depth camera and CNN neural network to identify natural gesture instruction;Based on UWB positioning network, centimeter-level accurate positioning and path deviation alarm are provided;Through global mini-map and gesture pointing, frame selection mechanism, seamless view switching and intelligent task collaboration of multiple robots are realized.The application overcomes the defects of the existing two-dimensional screen control system, such as lack of spatial perception, complex interaction, non-intuitive positioning and low collaboration efficiency, and constructs a man-machine-environment intelligent closed-loop control ecology, which can significantly improve the efficiency and safety of detection and rescue tasks in mine disaster environment.

Description

Technical Field

[0001] This invention relates to the field of remote control of mining robots and augmented reality (AR) technology, specifically to a remote control system and method for mining robots based on augmented reality. Background Technology

[0002] In mine accident rescue, ground operators rely on remotely controlled robots to enter dangerous areas. Existing mainstream systems suffer from fundamental flaws: the user interface, based on a two-dimensional screen displaying video streams and sensor data transmitted from the robot, makes it difficult for operators to build accurate three-dimensional situational awareness, often leading to misjudgments and collisions due to missing depth information; control relies on multi-button joysticks and complex menus, placing a heavy burden on instruction memorization, lengthy training periods for new operators, and making them highly susceptible to misoperation in emergency situations; robot positioning information is presented in abstract digital coordinates, lacking an intuitive spatial reference system, resulting in significant deviations in operators' judgments of relative position and direction of movement, greatly extending task execution time; when multiple robots need to work collaboratively, operators must frequently switch between multiple independent control windows, failing to form a unified global task view, leading to low collaborative efficiency. Therefore, existing technologies have bottlenecks in all four dimensions of "perception, interaction, positioning, and collaboration," urgently requiring a revolutionary solution that can achieve intuitive perception, natural interaction, precise positioning, and integrated collaboration. Summary of the Invention

[0003] The present invention aims to solve the above-mentioned technical problems by providing a remote control system and method for mining robots based on augmented reality.

[0004] To solve the above-mentioned technical problems, the technical solution provided by the present invention is as follows:

[0005] A remote control method for a mining robot based on augmented reality includes the following steps:

[0006] Step 1: Construct a digital twin model of the mine environment, which includes the mine tunnel structure, the location of key obstacles, and environmental feature information;

[0007] Step 2: Using AR virtual-real fusion technology, register and integrate the multi-source data returned by the robot with the digital twin model to form an enhanced operational field of view;

[0008] Step 3: Generate control commands through natural gesture recognition to achieve robot movement, steering, configuration switching and function control;

[0009] Step 4: Achieve precise positioning of the robot and operator based on the UWB positioning system, and complete navigation and path deviation detection by combining path planning algorithms;

[0010] Step 5: Through the multi-machine collaborative interface and gesture control mechanism, realize the global situation monitoring, seamless switching of perspectives and collaborative task allocation of multiple robots.

[0011] Preferably, the specific steps of the AR virtual-real fusion include:

[0012] Step 1: Receive and synchronize multi-source heterogeneous data from the robot, including 3D laser point cloud, 2D visual image and IMU pose data, and use graph optimization-based SLAM algorithm to match the point cloud with the digital model in real time.

[0013] Step 2: Solve for the rigid body transformation matrix using a nonlinear least squares problem, as shown in the following formula:

[0014] ;in, For real-time point cloud data points, For the corresponding point in the digital twin model, Let be the rigid body transformation matrix containing rotation and translation to be solved. This is a regularization term used to smooth and optimize the pose graph;

[0015] Step 3: Perform multi-layered 3D rendering on the AR display device, including a real video layer, a virtual annotation layer, and a status information layer. The virtual annotation layer includes 3D bounding boxes of obstacles, distance markers, and transparent highlight rendering of safety passages.

[0016] Preferably, the specific implementation process of the natural gesture recognition includes:

[0017] Step 1: Continuously acquire depth and color images of the operator's hand using an infrared depth camera, and simultaneously extract the three-dimensional coordinates of key points of the hand skeleton;

[0018] Step 2: Preprocess the acquired image data by extracting gesture features layer by layer using a convolutional neural network and classifying them; the formula is shown below: ;in, For the first The input feature map of the layer, and The first Convolution kernels and biases of layers, It is a non-linear activation function;

[0019] Step 3: Output the probability distribution of gesture categories through a fully connected layer and a Softmax classifier, and match it with 20 preset core control commands;

[0020] Step 4: Optimize recognition accuracy by combining skeletal tracking data, and provide tactile or visual cues when gesture recognition is incorrect.

[0021] Preferably, the core control commands include forward, backward, left translation, right translation, acceleration, deceleration, left turn, right turn, in-situ rotation alignment, pause / hold, continue, emergency stop, configuration switch A, configuration switch B, confirm / execute, cancel / undo, camera left scan, camera right scan, pointing selection, and box selection formation.

[0022] Preferably, the specific implementation process of the precise positioning and path deviation detection includes:

[0023] Step 1: Calculate the robot and operator positions based on the UWB positioning network. The absolute and relative positions are calculated by measuring the time difference of arrival of radio signals between the tag and multiple fixed base stations. The physical model equations are as follows:

[0024] ;in, Let these be the coordinates of the label to be determined. For the first The known coordinates of each base station For the signal to reach the base station With reference base station Time difference, The speed of light;

[0025] Step 2: The operator marks environmental feature points with gestures, and the system compares the real location with the corresponding points in the virtual model to dynamically correct the virtual-real registration deviation;

[0026] Step 3: Calculate the Euclidean distance between the robot's current position and the planned path in real time. When the deviation exceeds the preset threshold, trigger visual and tactile alarms, dynamically generate correction vectors, and update the path display.

[0027] Preferably, the specific implementation process of the multi-machine collaborative interface includes:

[0028] Step 1: Display a global mini-map in the AR view, presenting the real-time positions of all robots, key target points, and the working area environment in a two-dimensional top-down view;

[0029] Step 2: Use gestures to point at the robot icon on the mini-map or in the AR field of view to achieve seamless switching of the first-person perspective and maintain visual continuity;

[0030] Step 3: Select multiple robots using a double-handed selection gesture, point to the target area to trigger the collaborative task allocation algorithm, solve for the objective function to achieve optimal task division and path planning, as shown in the formula below:

[0031] ;in These represent the planning results / solutions for robot A and robot B, respectively. These are weighting coefficients. For time cost, For overlapping cost / penalty terms, This is an energy consumption cost item.

[0032] The present invention also provides an augmented reality-based remote control system for mining robots, comprising:

[0033] The AR virtual-real fusion module is used to construct an enhanced operational field of view that precisely matches the real mine environment;

[0034] The gesture interaction unit is used to recognize the operator's natural gestures and convert them into control commands;

[0035] The spatial positioning calibration module is used to achieve centimeter-level precise positioning between the robot and the operator;

[0036] A multi-robot collaboration interface is used to achieve unified management and task collaboration among multiple robots.

[0037] Preferably, the AR virtual-real fusion module includes:

[0038] The multi-source data fusion unit is used to receive and synchronize the robot's 3D laser point cloud, 2D image and IMU pose data;

[0039] The virtual-real registration unit uses a graph-optimized SLAM algorithm to match real-time point clouds with the mine digital twin model;

[0040] An integrated rendering unit is used to blend and display real video, virtual annotations, and robot status panels in AR display devices;

[0041] The gesture interaction unit includes:

[0042] Infrared depth camera, used to capture depth and color images of the operator's hand;

[0043] Convolutional neural networks are used for real-time recognition and classification of gesture images;

[0044] Feedback devices are used to provide tactile or visual cues when gesture recognition errors occur;

[0045] The spatial positioning calibration module includes:

[0046] UWB positioning network consists of multiple base stations and tags, and calculates location based on TDOA model;

[0047] The virtual-real deviation correction unit allows operators to mark feature points with gestures to optimize registration accuracy;

[0048] The path deviation detection and alarm unit is used to monitor robot path deviation in real time and prompt corrections.

[0049] The multi-machine collaborative interface includes:

[0050] A global mini-map displays all robots and key targets in real time in a two-dimensional top-down view.

[0051] A seamless first-person perspective switching mechanism supports switching the controlled object via gesture pointing;

[0052] The collaborative task allocation unit supports selecting robots and assigning collaborative tasks via gestures.

[0053] By employing the above methods and systems, the present invention has the following advantages:

[0054] This invention combines SLAM algorithms to achieve millimeter-level virtual-real registration of robot-transmitted data and virtual models; utilizes convolutional neural networks to recognize natural gesture commands, enabling intuitive human-computer interaction; achieves centimeter-level precise positioning and path deviation alarms based on UWB positioning networks; and achieves seamless switching of multiple robot perspectives and intelligent task collaboration through a multi-robot collaborative interface and gesture control mechanism. It realizes a leap from "single-machine control" to "team management," from "two-dimensional perception" to "three-dimensional immersion," and from "complex interaction" to "natural interaction," significantly improving the efficiency and safety of mine disaster rescue. Furthermore, this invention overcomes the shortcomings of existing two-dimensional control systems, such as lack of spatial perception, complex interaction, unintuitive positioning, and low collaborative efficiency, constructing a "human-machine-environment" intelligent closed-loop control ecosystem, significantly improving the efficiency and safety of detection and rescue tasks in mine disaster environments. Attached Figure Description

[0055] Figure 1 This is a diagram of the overall system architecture of the present invention;

[0056] Figure 2 This is a schematic diagram of the gesture interaction unit of the present invention. Detailed Implementation

[0057] The present invention will now be described in further detail with reference to the full text.

[0058] Combined with appendix Figure 1 and Figure 2 This paper presents a remote control system and method for mining robots based on augmented reality (AR). The core innovation of this system lies in the construction of a closed-loop intelligent control ecosystem of "human-machine-environment" through deep integration of multiple modules and core algorithm-driven operation.

[0059] The technical solution of the present invention consists of four key technical modules that work together: an AR virtual-real fusion module, a gesture interaction unit, a spatial positioning calibration module, and a multi-machine collaborative interface.

[0060] The AR virtual-real fusion module is the perception core of the system, responsible for constructing an enhanced operational field of view that precisely matches the real environment. This module first receives and fuses multi-source heterogeneous data from the robot via a time synchronization mechanism, including 3D laser point clouds, 2D visual images, and inertial measurement unit (IMU) pose data. Subsequently, it executes the core virtual-real registration process: employing a graph-optimized SLAM (Simultaneous Localization and Mapping) algorithm, it matches and aligns the real-time acquired real-world point cloud with a pre-constructed high-precision digital twin model of the mine. This process is achieved by solving a nonlinear least-squares problem, with the objective function being:

[0061] ;

[0062] in, For real-time point cloud data points, For the corresponding point in the digital twin model, Let be the rigid body transformation matrix containing rotation and translation to be solved. This is a regularization term used to smooth and optimize the pose graph. This algorithm ensures that virtual information can be superimposed on the real world with millimeter-level precision. Finally, in the high-definition optical display system of AR glasses, the system integrates the real scene (including the robot's first-person view video, etc.), virtual annotations (including the transparent and highlighted rendering of 3D bounding boxes of obstacles, distances, safety passages, etc.), and the robot's status panel (including battery level, speed, working configuration, etc.) into a unified rendering, providing the operator with an immersive perception environment that feels like "seeing through" the surroundings.

[0063] The gesture interaction unit defines a next-generation paradigm for natural human-computer interaction. This unit utilizes an infrared depth camera deployed in the operating space to continuously capture depth and color images of the operator's hand at a high frame rate. The captured raw data is fed into a carefully designed convolutional neural network (CNN) for real-time recognition and analysis. This network extracts gesture features layer by layer through multi-layer convolution and pooling operations; its core computation can be represented as:

[0064] ;

[0065] in, For the first The input feature map of the layer, K (l) and b (l) These represent the convolution kernel and bias of the l-th layer, respectively. The activation function is non-linear. The network ends with a fully connected layer connected to a Softmax classifier, outputting a probability distribution covering over twenty core commands, including basic movements (such as pushing / pulling the palm forward / backward), fine steering (wrist swinging left / right), configurational switching (clenching a fist, opening the palm), and emergency commands (crossing hands). The specific distribution is as follows:

[0066] 1. Moving forward: With one hand outstretched and palm facing forward, push forward in the direction of your line of sight (displacement > threshold and hold for 0.3–0.5 seconds);

[0067] 2. Backwards: With one hand outstretched and palm facing forward, pull backwards along the direction of your line of sight (displacement > threshold and maintain).

[0068] 3. Leftward lateral movement: Spread your five fingers with one hand, palm facing down or forward, and move it horizontally to the left a certain distance;

[0069] 4. Right lateral movement: Spread your five fingers with one hand, palm facing down or forward, and move it horizontally to the right a certain distance;

[0070] 5. Acceleration: Perform the "pinch → open" action once with one hand; or twice in a row to represent greater acceleration;

[0071] 6. Deceleration: Perform the "open → pinch" motion once with one hand; or perform it twice in a row to represent a greater deceleration.

[0072] 7. Left turn: Keep one hand in a natural position and swing / rotate the wrist to the left (angular velocity or angle > threshold);

[0073] 8. Turn right: Keep one hand in a natural position and swing / rotate the wrist to the right;

[0074] 9. In-place rotation alignment: Open your arms and make a twisting motion in opposite directions with your left and right hands (like wringing a towel) to trigger in-place rotation alignment;

[0075] 10. Pause / Hold: Hold one hand with palm facing forward for ≥1 second to enter stop / hold mode;

[0076] 11. Continue: After pausing, make a "forward" gesture or a short "palm forward → retract" motion to release the hold;

[0077] 12. Emergency Stop: Cross your arms in an X shape and hold for ≥0.5 seconds to immediately trigger an emergency stop (highest priority).

[0078] 13. Configuration Switch A: Hold a clenched fist for ≥0.5s to switch to the preset configuration A (e.g., "crab shape / low pass").

[0079] 14. Configuration switch B: Hold palm open (five fingers spread) for ≥0.5s to switch to preset configuration B (e.g., "snake-like / slit-like").

[0080] 15. Confirmation / Execution: Hold the thumb raised for ≥0.3 seconds to confirm dangerous actions, begin detection / scanning, etc.

[0081] 16. Cancel / Undo: Thumb down or "shake your palm left and right once" to cancel the previous command / exit the current mode;

[0082] 17. Camera left sweep: Brush two fingers together (index and middle fingers) to the left in the air to drive the gimbal / view to turn left;

[0083] 18. Camera right sweep: Swipe right in the air with two fingers together to drive the gimbal / view to turn right;

[0084] 19. Pointing Selection: Point your index finger at the robot entity / mini-map icon / target point in the AR field of view to trigger "Select / Focus / Switch to First-Person View";

[0085] 20. Select Group: Use both hands to select (draw circles in the air to form a "selection area") multiple robot icons to trigger "Group Selection / Batch Assignment Task".

[0086] The system combines skeletal tracking data to ensure the accuracy and robustness of the recognition results, achieving high recognition accuracy and extremely low response latency. Simultaneously, the system provides operational assistance mechanisms. For example, in novice mode, a semi-transparent overlay of currently executable gesture icons is displayed in the gesture interaction area. Furthermore, when the operator makes an incorrect or ambiguous gesture, gentle vibration feedback is provided via a smartwatch or haptic glove, enabling silent guidance and correction.

[0087] The spatial positioning calibration module is fundamental to achieving precise navigation and control. The system employs ultra-wideband (UWB) technology to construct a sophisticated cooperative positioning network. Both the robot and the operator's AR system wear UWB tags, and absolute and relative positions are calculated by measuring the time difference of arrival (TDOA) of radio signals between the tags and multiple fixed base stations. This physical model is described by the following set of equations:

[0088] ;in, Let these be the coordinates of the label to be determined. For the first The known coordinates of each base station For the signal to reach the base station Time difference with reference base station 1 The speed is the speed of light. Solving this overdetermined system of equations yields 3D coordinates with centimeter-level accuracy. To further eliminate accumulated errors, the system introduces a virtual-real deviation correction function: the operator can mark a significant feature point (such as a pipe joint) in the real environment using a specific gesture (such as pointing with the index finger). The system then compares and fine-tunes the real position of this point with the corresponding position in the virtual model, dynamically optimizing the registered parameters. In the AR interface, the planned path is rendered in real time as a highlighted colored dashed line. When the system detects that the robot deviates from the planned path by more than a predetermined threshold due to slippage or other reasons, the path line color changes from a safety color (such as green) to a warning color (such as red), accompanied by a dynamic arrow indicating the correction direction.

[0089] The multi-robot collaborative interface enables a leap from "single-robot control" to "team management." Located in the corner of the operator's AR main field of view, this interface presents a two-dimensional top-down view of the entire work area as a constantly floating global mini-map. All online robots and key target points (such as survivor locations and charging stations) are clearly displayed with dynamic markers of different colors and icons. To control a specific robot, the operator needs no menu input; simply pointing their finger at the robot's physical form in the AR view or the corresponding icon on the mini-map will seamlessly switch the system to the robot's first-person view video stream and data interface in a very short time through spatial calculation and intent recognition. For complex collaborative tasks, the operator can use hand gestures to select one or more robots in the AR view, then point to a target area. The system's background task allocation algorithm will automatically calculate the optimal task division and movement path for the selected robot group (e.g., instructing robot A to scan the left side of the area while robot B simultaneously scans the right side), and display the planning results intuitively in the AR global view and in the individual robot perspectives as different colored path lines.

[0090] Example 1:

[0091] This embodiment uses a "life detection and rescue in a narrow mine passage" process as an example to illustrate the implementation process of the present invention in detail.

[0092] Phase 1: System initialization and precise 3D registration of the environment, specifically including 4 processes:

[0093] Process 1: Hardware system deployment and startup preparation process is mainly divided into the following 4 stages:

[0094] (1) In the mine ground command center, operators wear AR head-mounted display devices (such as Microsoft HoloLens2), which integrate infrared depth cameras, IMU sensors and display systems.

[0095] (2) At key locations in the mine operation area, four or more UWB positioning base stations are precisely deployed to form a positioning network covering the entire operation area. The coordinates of the base stations are accurately measured by a total station and entered into the system.

[0096] (3) The morphological robot and the flying robot are powered on and self-tested in sequence, and establish a low-latency communication link with the central processing system;

[0097] (4) The system loads a pre-built high-precision digital twin model of the mine, which includes environmental information such as the channel structure and the location of key obstacles.

[0098] Process 2: The multi-source data acquisition and synchronous transmission process is mainly divided into the following three stages:

[0099] (1) The morphological robot enters the narrow channel of the target, and its onboard 16-line lidar scans the environment at a frequency of 10Hz to generate dense three-dimensional point cloud data. ;

[0100] (2) The RGB-D camera synchronously acquires color images and depth information, and the IMU provides real-time pose data for the robot;

[0101] (3) All sensor data are synchronized at the microsecond level through hardware timestamps, packaged and transmitted to the ground AR virtual-real fusion module via 5G network.

[0102] Process 3: SLAM Virtual-Real Registration and Precise Alignment. This process is implemented by initiating the core SLAM virtual-real registration algorithm through the AR virtual-real fusion module. Specifically, this process includes:

[0103] (1) Feature extraction and matching: Extract ISS feature points from the real-time point cloud P and match them with the corresponding point set in the digital twin model. Perform FPFH feature descriptor matching;

[0104] (2) Pose graph optimization: Construct a pose graph containing robot pose nodes and environmental feature points, and calculate the optimal rigid body transformation matrix T* by solving the following nonlinear optimization problem:

[0105] ;

[0106] Among them, the regularization term Specifically, it can be expressed as follows:

[0107] ;

[0108] Regularization / smoothing terms are used to constrain the solution of the entire pose graph to be more consistent and smoother, avoiding drift or instability caused by fitting only data terms;

[0109] : A collective term for pose-related variables (which can be understood as the entire pose set or the set of optimization variables);

[0110] : Weight coefficient, used to adjust the proportion of the regularization term in the overall objective function (the larger the value, the more emphasis is placed on smoothness / consistency constraints);

[0111] Sum all constraint edges in the graph; (Also written as E in the text) represents the set of pose constraint edges, where each edge connects two pose nodes. ;

[0112] : The index of two nodes in the pose graph (e.g., the poses at two time points of adjacent keyframes, odometry-constrained connections, or the poses of two frames connected by loop closures).

[0113] : respectively represent the first The and the first The pose of each node (generally SE(3) rigid body transformation: rotation + translation);

[0114] From node To the node The relative pose transformation (the relative relationship obtained by multiplying the current estimated absolute pose);

[0115] Logarithmic mapping from Lie groups to Lie algebras (mapping the transformation matrix of SE(3) into a vector form of "error" / "increment", commonly a 6-dimensional vector: 3-dimensional rotation + 3-dimensional translation);

[0116] : The squared norm weighted by covariance (commonly used in Mahalanobis distance form). Representing an edge The measurement uncertainty / noise covariance is used to weight the confidence level of different constraints.

[0117] Process 4: Real-time rendering and display of the AR environment. After successful registration, the system performs multi-layered 3D rendering in the operator's AR field of view, specifically divided into:

[0118] (1) Real video layer: Displays a real-time RGB video stream from the robot's first-person perspective, maintaining a 30fps refresh rate;

[0119] (2) Virtual annotation layer (example):

[0120] Semi-transparent blue grid lines were overlaid on the walls on both sides of the passage, and information such as the width of the passage was dynamically marked at key locations.

[0121] At a certain distance in front of the passage, the real stone is rendered as a red 3D bounding box, and the presence or absence of obstacles and the height information of the obstacles are marked.

[0122] At a certain distance to the right front of the depths of the passage, render a continuously flashing orange sphere labeled with information such as "suspected life signal".

[0123] Status information layer: The robot status is displayed floating in the upper right corner of the field of view, including robot number, current battery level, robot configuration, and other information.

[0124] The second phase: natural gesture interaction and intelligent navigation control, specifically including four processes:

[0125] Process 1: Gesture command initiation and high-precision capture, the specific process is as follows:

[0126] (1) After the operator observes the AR interface and confirms that the environment is safe, he / she can make a gesture such as "pushing the palm forward" within the field of view of the depth camera as a sign of action;

[0127] (2) The infrared depth camera simultaneously captures the depth map of the hand. and color pictures ;

[0128] (3) The hand skeleton tracking algorithm extracts the three-dimensional coordinates of key hand points in real time to form gesture skeleton data. Process 2: Real-time inference of CNN deep neural network, the specific process is as follows:

[0129] The gesture interaction unit feeds the pre-processed multimodal data into a pre-trained convolutional neural network, which performs hierarchical feature extraction. The specific hierarchical extraction levels are as follows:

[0130] (1) Shallow feature extraction (layers 1-3):

[0131] ;

[0132] Feature map extracted from the first layer;

[0133] Activation function (usually a non-linear activation function such as ReLU or Sigmoid);

[0134] Input image or feature map from the previous layer;

[0135] The first layer's convolutional kernel (filter);

[0136] : The bias term of the first layer.

[0137] (2) Deep feature abstraction (4th-Lth layer):

[0138] ;

[0139] The first layer's convolutional kernel (filter);

[0140] Previous layer (first) Feature maps of layers;

[0141] : No. Convolutional kernels of layers;

[0142] : No. Layer bias terms;

[0143] (3) Classification decision:

[0144] ;

[0145] : The output of the classification (e.g., the probability of the gesture category);

[0146] : Weight matrix, used to weight the first... Layer features are mapped to the final classification space;

[0147] : No. Feature map of the last layer;

[0148] Bias term, usually related to category;

[0149] Activation function: Used to transform the network output into a probability distribution, typically used for multi-class classification problems.

[0150] When the probability value of the "forward" command in the output vector y exceeds the set threshold, the system determines that the command is valid.

[0151] Process 3: Instruction execution and robot motion control, the specific process is as follows:

[0152] (1) Valid instructions are sent to the variable-cell robot via wireless network. The robot motion control module parses the instructions and starts moving forward at a specific speed.

[0153] (2) The AR interface updates in real time, displaying a green dashed navigation path extending from the robot's location to the life signal point;

[0154] (3) The robot pose data is transmitted back in real time through the UWB system to update the virtual robot model in the AR scene.

[0155] Process 4: Adaptive obstacle avoidance and dynamic path planning, the specific process is as follows:

[0156] (1) Collision warning: When the robot approaches a virtually marked obstacle, the system performs real-time detection based on the AABB (Axis-aligned bounding box) collision detection algorithm. The AABB collision detection method determines whether a collision has occurred by generating a rectangular box (in 2D) or a cube box (in 3D) parallel to the coordinate axes for the object. The specific steps are as follows:

[0157] Define AABB: The AABB of each object is represented by two points: the minimum point. and maximum point These represent the bottom left and top right corners (2D) or the bottom front and top back corners (3D) of an object, respectively.

[0158] Checking for AABB overlap: For two AABBs, assuming they are A and B respectively, to determine if they intersect, the following conditions must be checked:

[0159] If the maximum x-coordinate of A is less than the minimum x-coordinate of B, or the minimum x-coordinate of A is greater than the maximum x-coordinate of B, then the two will not collide.

[0160] If the maximum y-coordinate of A is less than the minimum y-coordinate of B, or the minimum y-coordinate of A is greater than the maximum y-coordinate of B, then the two will not collide.

[0161] Collision detection: If any of the above conditions are met, it means that AABB does not overlap and the objects do not collide; otherwise, the objects collide.

[0162] (2) The conditions are met Timely triggering of warnings; This is the robot's current position. The location of obstacles is marked virtually; This is the minimum safe distance threshold.

[0163] (2) Intelligent prompts: A text prompt for turning and obstacle avoidance is dynamically popped up next to the obstacle. At the same time, the path planning algorithm is recalculated to generate a smooth B-spline detour curve to replace the original path.

[0164] (3) Multimodal interaction: The operator observes the prompt, keeps the right hand forward command, and at the same time extends the left hand to make a "slightly swing to the left" turning gesture. The CNN model recognizes this compound command, and the robot receives the heading adjustment parameters of the command.

[0165] (4) Configuration adaptation: If the width of the passage decreases after detouring, the operator makes a "fist" gesture. After the system recognizes it, it sends a configuration switching command to the variable-cell robot. The robot reconstructs from "snake shape" to "crab shape" within a specific time, so that the cross-section is reduced to a passable size.

[0166] Phase Three: Integration of Positioning and Precise Target Achievement, specifically including three processes:

[0167] Process 1: UWB / TDOA high-precision positioning calculation, the specific process is as follows:

[0168] The spatial positioning calibration module operates continuously, and the specific process consists of the following three steps:

[0169] Step 1: Signal Measurement: The robot's UWB tag performs bidirectional ranging with each base station, measuring the signal arrival time difference. ;

[0170] Step 2: Location Calculation: The localization engine solves the following system of equations based on the TDOA model:

[0171] ;in, Let these be the coordinates of the label to be determined. For the first The known coordinates of each base station For the signal to reach the base station With reference base station Time difference, The speed of light;

[0172] Step 3: Kalman filtering: Smooth the positioning results by fusing IMU data to suppress measurement noise.

[0173] Process 2: Location information visualization and intelligent alarm, the specific process is as follows:

[0174] (1) Navigation guidance: The calculated robot position is mapped to the AR interface in real time, and a red three-dimensional arrow is generated pointing from the robot position to the life signal target point. Information such as the azimuth angle is dynamically displayed next to the arrow.

[0175] (2) Deviation detection: Real-time calculation of the robot's current position To the planned path Euclidean distance:

[0176] ;

[0177] (3) Alarm Trigger: When At this time, the system triggers a multi-level alarm: the path line changes from green to rapidly flashing red, a yellow correction vector arrow is generated, and the haptic device provides vibration feedback.

[0178] Step 3: Target confirmation and detection initiation, the specific process is as follows:

[0179] (1) The operator makes course corrections through subtle turning gestures to bring the robot back to a safe path;

[0180] (2) When the UWB displays that the robot has entered a certain distance from the target point, the AR interface pops up "The target is directly in front of us, ready to detect" and marks the relative distance;

[0181] (3) When the operator makes a "thumbs up" confirmation gesture, the robot automatically starts the life detection radar to perform a fine scan and transmits the detection data back in real time.

[0182] Phase Four: Multi-robot intelligent collaboration and task closure, specifically including four processes:

[0183] Process 1: Global Situation Awareness and Monitoring, the specific process is as follows:

[0184] (1) The system maintains a unified global coordinate system, and the positions of all robots are represented in the same coordinate system;

[0185] (2) The global mini-map in the upper right corner of the AR field of view is displayed in real time in the form of a 2D top view: the morphological robot moves along the channel, the flying robot hovers at the entrance, and the location of the life signal continues to flash;

[0186] (3) The mini-map supports gesture zooming and rotation, making it convenient for operators to observe the overall situation from different angles.

[0187] Process 2: Seamless first-person perspective switching mechanism, the specific process is as follows

[0188] (1) When the operator needs the assistance of the flying robot, he will naturally raise his finger and point to the flying robot icon on the mini map;

[0189] (2) The visual recognition algorithm combines the gesture direction vector and the icon screen coordinates to calculate the intersection of gaze and confirm the selection intention;

[0190] (3) The system completes the context switching within a certain time: fade out of the perspective of the morphological robot and fade in the video stream of the flying robot looking down, maintaining visual continuity.

[0191] Process 3: Collaborative task allocation and path planning, the specific process is as follows:

[0192] (1) The operator selects the morphological robot and the flying robot by using the "two-handed selection" gesture, and then points to the area to be detected;

[0193] (2) The system triggers a collaborative task allocation algorithm to solve the following optimization problem:

[0194] ;

[0195] Among all feasible solutions, the planning schemes for robots A and B are jointly optimized to minimize the objective function.

[0196] : These represent the planning results / solutions for robot A and robot B, respectively, and can usually be specified as "path sequence (waypoint set) + corresponding task allocation (scanning area / target point sequence) + time annotation", etc.

[0197] Two robots participating in the collaboration (a metamorphic robot and a flying robot).

[0198] : Weighting coefficients are used to balance the importance of different optimization objectives (e.g., prioritizing time, avoiding duplication, or saving energy).

[0199] The time-related cost term can be defined as the total completion time or the sum / average time of the two robots, etc.

[0200] Overlap cost / penalty term: measures the degree of repetition between two robots in the task area or path (such as the overlapping area of ​​the scan area, the proportion of repeated coverage, the length of repeated passage in the same channel, etc.), and it is generally desirable to be as small as possible.

[0201] Energy consumption cost can be estimated from path length, speed / acceleration, slope / terrain resistance, load, etc., and it is generally desirable to keep it as low as possible.

[0202] (3) Calculate the path lines of different colors and display them in different colors.

[0203] Process 4: Task completion and information collaboration, the specific process is as follows:

[0204] (1) After the morphobot confirms the discovery of a survivor, it publishes a "SurvivorFound" message through the system event bus, which includes the precise location coordinates and vital signs data;

[0205] (2) All robot platforms and ground command centers receive this information synchronously and update their respective status displays;

[0206] (3) The mother robot uses the A* algorithm to plan the optimal rescue route based on global map information and highlights it for the operator in the AR interface;

[0207] (4) The system generates a task report, which records the data of the entire process from detection to rescue planning, for subsequent analysis and optimization.

[0208] The present invention and its embodiments have been described above. This description is not restrictive, and the embodiments shown throughout are only one of the embodiments of the present invention. The actual structure is not limited to this. In conclusion, if those skilled in the art are inspired by this description and design similar structures and embodiments without departing from the spirit of the present invention, they should all fall within the protection scope of the present invention.

Claims

1. A remote control method for a mining robot based on augmented reality, characterized in that, Includes the following steps: Step 1: Construct a digital twin model of the mine environment, which includes the mine tunnel structure, the location of key obstacles, and environmental feature information; Step 2: Using AR virtual-real fusion technology, register and integrate the multi-source data returned by the robot with the digital twin model to form an enhanced operational field of view; Step 3: Generate control commands through natural gesture recognition to achieve robot movement, steering, configuration switching and function control; Step 4: Achieve precise positioning of the robot and operator based on the UWB positioning system, and complete navigation and path deviation detection by combining path planning algorithms; Step 5: Through the multi-machine collaborative interface and gesture control mechanism, realize the global situation monitoring, seamless switching of perspectives and collaborative task allocation of multiple robots.

2. The remote control method for a mining robot based on augmented reality according to claim 1, characterized in that: The specific steps of the AR virtual-real fusion include: Step 1: Receive and synchronize multi-source heterogeneous data from the robot, including 3D laser point cloud, 2D visual image and IMU pose data, and use graph optimization-based SLAM algorithm to match the point cloud with the digital model in real time. Step 2: Solve for the rigid body transformation matrix using a nonlinear least squares problem, as shown in the following formula: ;in, For real-time point cloud data points, For the corresponding point in the digital twin model, Let be the rigid body transformation matrix containing rotation and translation to be solved. This is a regularization term used to smooth and optimize the pose graph; Step 3: Perform multi-layered 3D rendering on the AR display device, including a real video layer, a virtual annotation layer, and a status information layer. The virtual annotation layer includes 3D bounding boxes of obstacles, distance markers, and transparent highlight rendering of safety passages.

3. The remote control method for a mining robot based on augmented reality according to claim 1, characterized in that: The specific implementation process of natural gesture recognition includes: Step 1: Continuously acquire depth and color images of the operator's hand using an infrared depth camera, and simultaneously extract the three-dimensional coordinates of key points of the hand skeleton; Step 2: Preprocess the acquired image data by extracting gesture features layer by layer using a convolutional neural network and classifying them; the formula is shown below: ;in, For the first The input feature map of the layer, and The first Convolution kernels and biases of layers, It is a non-linear activation function; Step 3: Output the probability distribution of gesture categories through a fully connected layer and a Softmax classifier, and match it with 20 preset core control commands; Step 4: Optimize recognition accuracy by combining skeletal tracking data, and provide tactile or visual cues when gesture recognition is incorrect.

4. The remote control method for a mining robot based on augmented reality according to claim 3, characterized in that: The core control commands include forward, backward, left lateral, right lateral, acceleration, deceleration, left turn, right turn, in-place rotation alignment, pause / hold, continue, emergency stop, configuration switch A, configuration switch B, confirm / execute, cancel / undo, camera left scan, camera right scan, point selection, and box selection formation.

5. The remote control method for a mining robot based on augmented reality according to claim 1, characterized in that: The specific implementation process of the precise positioning and path deviation detection includes: Step 1: Calculate the robot and operator positions based on the UWB positioning network. The absolute and relative positions are calculated by measuring the time difference of arrival of radio signals between the tag and multiple fixed base stations. The physical model equations are as follows: ;in, Let these be the coordinates of the label to be determined. For the first The known coordinates of each base station For the signal to reach the base station With reference base station Time difference, The speed of light; Step 2: The operator marks environmental feature points with gestures, and the system compares the real location with the corresponding points in the virtual model to dynamically correct the virtual-real registration deviation; Step 3: Calculate the Euclidean distance between the robot's current position and the planned path in real time. When the deviation exceeds the preset threshold, trigger visual and tactile alarms, dynamically generate correction vectors, and update the path display.

6. The remote control method for a mining robot based on augmented reality according to claim 1, characterized in that: The specific implementation process of the multi-machine collaborative interface includes: Step 1: Display a global mini-map in the AR view, presenting the real-time positions of all robots, key target points, and the working area environment in a two-dimensional top-down view; Step 2: Use gestures to point at the robot icon on the mini-map or in the AR field of view to achieve seamless switching of the first-person perspective and maintain visual continuity; Step 3: Select multiple robots using a double-handed selection gesture, point to the target area to trigger the collaborative task allocation algorithm, solve for the objective function to achieve optimal task division and path planning, as shown in the formula below: ;in These represent the planning results / solutions for robot A and robot B, respectively. These are weighting coefficients. For time cost, For overlapping cost / penalty terms, This is an energy consumption cost item.

7. A remote control system for a mining robot based on augmented reality, characterized in that: include: The AR virtual-real fusion module is used to construct an enhanced operational field of view that precisely matches the real mine environment; The gesture interaction unit is used to recognize the operator's natural gestures and convert them into control commands; The spatial positioning calibration module is used to achieve centimeter-level precise positioning between the robot and the operator; A multi-robot collaboration interface is used to achieve unified management and task collaboration among multiple robots.

8. The augmented reality-based remote control system for mining robots according to claim 7, characterized in that: The AR virtual-real fusion module includes: The multi-source data fusion unit is used to receive and synchronize the robot's 3D laser point cloud, 2D image and IMU pose data; The virtual-real registration unit uses a graph-optimized SLAM algorithm to match real-time point clouds with the mine digital twin model; An integrated rendering unit is used to blend and display real video, virtual annotations, and robot status panels in AR display devices; The gesture interaction unit includes: Infrared depth camera, used to capture depth and color images of the operator's hand; Convolutional neural networks are used for real-time recognition and classification of gesture images; Feedback devices are used to provide tactile or visual cues when gesture recognition errors occur; The spatial positioning calibration module includes: UWB positioning network consists of multiple base stations and tags, and calculates location based on TDOA model; The virtual-real deviation correction unit allows operators to mark feature points with gestures to optimize registration accuracy; The path deviation detection and alarm unit is used to monitor the robot's path deviation in real time and prompt corrections. The multi-machine collaborative interface includes: A global mini-map displays all robots and key targets in real time in a two-dimensional top-down view. A seamless first-person perspective switching mechanism supports switching the controlled object via gesture pointing; The collaborative task allocation unit supports selecting robots and assigning collaborative tasks via gestures.

Images