Remote control method for mining large electric shovel based on AR (Augmented Reality) or MR (Magnetic Resonance)

Abstract

The invention discloses a mining large electric shovel remote control method based on AR or MR, and belongs to the technical field of electric shovel intellectualization. According to the method, a three-dimensional digital model is constructed by using a multi-source data fusion algorithm, virtual and real scenes are synchronously presented in AR or MR head display equipment in combination with an electric shovel digital twinborn model, and the telescopic stroke of a push rod, the bucket opening and closing state and the walking paths of left and right crawler belts are dynamically overlaid and displayed; a virtual-real fusion multi-modal interaction system is constructed, space-time consistency calibration of a control instruction is realized, a dynamic mapping model is established, multi-dimensional control input is analyzed into specific action parameters, and through continuous learning of a reinforcement learning model, a bucket movement path and a walking path of an electric shovel are rehearsed and corrected in time, so that the accuracy of the operation of the electric shovel is improved. And meanwhile, a historical operation process can be stored and played back in the head-mounted display equipment. The problems that traditional manual operation is high in risk, existing remote control is poor in dynamic adaptability, and instruction distortion is caused by network delay are solved, and operation safety, operation efficiency and equipment reliability are remarkably improved.

Description

Technical Field

[0001] This invention belongs to the field of intelligent electric shovel technology, specifically relating to a remote control method for large mining electric shovels based on AR (Augmented Reality) or MR (Mixed Reality). Background Technology

[0002] Traditional operation of large electric shovels in mining relies on manual on-site control. Operators are exposed to hazardous environments such as high dust levels, extreme weather, and flying rocks from blasting, which can easily lead to occupational diseases. Furthermore, manual operation is limited by experience differences, and equipment failures caused by operator errors severely restrict mine production efficiency. To address this issue, remote control technology is becoming increasingly common, such as using multiple sensors to collect equipment status data in real time and combining it with network transmission for remote control. However, existing technologies still face significant bottlenecks, including poor adaptability to dynamic scenarios and insufficient network stability. Traditional methods in existing technologies cannot perceive changes in the ore accumulation morphology after blasting in real time, which can easily lead to deviations in the cutting angle, thus reducing efficiency. Summary of the Invention

[0003] To address the problems of existing technologies, this invention proposes a remote control method for large electric shovels used in mining based on AR or MR. Hardware-level synchronous data acquisition is achieved through sensors at the shovel end. A high-fidelity 3D digital model of the working scene is constructed using a multi-source data fusion algorithm. Combined with the shovel's digital twin model, a real-time rendering engine in an AR or MR headset enables synchronized presentation of the virtual and real scenes, dynamically overlaying and displaying the push rod extension / retraction stroke, bucket opening / closing status, and left / right track movement paths. A multimodal interactive system integrating virtual and real elements is constructed. A Kalman filter algorithm is used to achieve spatiotemporal consistency calibration of control commands, and a dynamic mapping model between remote operation signals and shovel response data is established. Multi-dimensional control inputs are parsed into action parameters for shovel bucket opening / closing, slewing platform rotation, push rod extension / retraction, hoisting rope lifting / lowering, and left / right track movement. Through continuous learning using a reinforcement learning model, the shovel's bucket movement path and walking path are pre-simulated and corrected in a timely manner. Simultaneously, historical operation processes can be saved and replayed in the AR or MR headset.

[0004] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0005] A remote control method for large electric shovels used in mining based on AR or MR mainly includes the following steps:

[0006] The first step is to use sensors at the end of the electric shovel to perform environmental perception and data collection, thereby obtaining multi-source perception data.

[0007] The electric shovel end sensors include binocular cameras symmetrically installed on the top of the electric shovel cab, an inclinometer installed at the push rod, a lidar installed at the front end of the electric shovel boom, a tension sensor installed at the lifting rope, high-precision encoders installed at the push motor, the lifting drum motor and the left and right travel motors respectively, a magnetic sensor installed at the bucket, and an inertial measurement unit installed at the center of the slewing platform.

[0008] A binocular camera mounted on the top of the electric shovel's cab acquires real-time binocular visual images and stereoscopic visual data of the working scene. An inclinometer installed at the push rod collects the angle between the push rod and the horizontal plane. A LiDAR deployed at the front end of the shovel's boom acquires real-time 3D point cloud data of the working area. A tension sensor installed at the lifting rope collects real-time load status data of the bucket. High-precision encoders installed at the push motor, lifting drum motor, and left and right travel motors collect real-time operating status data of the push rod, lifting rope, and left and right travel tracks. A magnetic sensor installed at the bucket collects bucket opening and closing status data. An inertial measurement unit installed at the center of the slewing platform collects real-time attitude data of the equipment, i.e., the rotation status data of the slewing platform. All sensor data are synchronously acquired and preprocessed at the hardware level to form time-aligned multi-source sensing data.

[0009] The multi-source sensing data includes the operating status data of the electric shovel actuator.

[0010] The second step involves processing the stereo vision data and 3D point cloud data obtained from the first step using a multi-source data fusion algorithm to construct a 3D digital model of the work scene, which can significantly improve the recognition accuracy of ore accumulation patterns in complex scenes.

[0011] The third step involves processing the multi-source sensing data obtained in the first step to obtain device response data, drawing a 3D model of the electric shovel and performing kinematic analysis, thereby constructing a digital twin model of the electric shovel, and realizing the synchronous presentation of virtual and real scenes and the synchronous updating of the digital twin model of the electric shovel in AR or MR head-mounted displays.

[0012] The fourth step involves building a multimodal interaction system on the remote control terminal based on the AR or MR head-mounted display device from the third step. This system establishes a dynamic mapping model between remote operation signals and the response data of the electric shovel equipment, and interprets the multi-dimensional control inputs into action parameters for the opening and closing of the electric shovel bucket, the rotation of the slewing platform, the extension and retraction of the push rod, the raising and lowering of the lifting rope, and the movement of the left and right walking tracks, thereby achieving a remote operation that combines virtual and real elements.

[0013] Control input is achieved through a remote controller integrating a mode switching button, a bucket opening / closing button, a slewing knob, and two single-degree-of-freedom force feedback joysticks on the left and right. The joystick pose change signals are processed using a Kalman filter algorithm to predict network transmission delay and dynamically calibrate command timing, ensuring the accuracy and real-time performance of control commands. The mode switching button is mapped to digging and walking modes; the bucket opening / closing button's on / off state is mapped to the opening and closing status of the electric shovel bucket; the slewing knob's torsional motion is mapped to the rotation speed of the slewing platform; in digging mode, the forward / backward angle of the left joystick is mapped to the extension / retraction rate of the push rod, and the forward / backward angle of the right joystick is mapped to the lifting / lowering rate of the hoisting rope; in walking mode, the forward / backward angles of the left and right joysticks are mapped to the forward or backward speed of the left and right tracks.

[0014] The fifth step involves collecting historical operation data and equipment response information from multiple sets of electric shovels as initial training data. A reinforcement learning model based on this training data is used to optimize the push-lift coordination strategy. Using the control information from the digging mode in the fourth step as a foundation, the digital twin model of the electric shovel is used to perform real-time pre-simulation of the bucket's movement path, i.e., the digging trajectory, and this path is then synchronously displayed on an AR or MR head-mounted display.

[0015] When the predicted bucket movement path indicates a collision risk, the reinforcement learning model automatically generates a modified trajectory to correct the control input in step four, and prioritizes its transmission to the electric shovel actuator via a 5G URLLC (Ultra-Reliable Low-Latency Communication) channel. Simultaneously, the haptic feedback module adjusts the joystick resistance gradient in real time based on the bucket load pressure.

[0016] For the walking task, the reinforcement learning model optimizes the differential speed control strategy of the left and right walking tracks of the electric shovel and performs a pre-simulation of the walking path based on the training data. When abnormal resistance on one side of the electric shovel's track is detected, the torque distribution of the left and right walking motors is automatically adjusted. Based on the control information of the fourth walking mode, the digital twin model of the electric shovel is presented in the AR or MR head-mounted display device to pre-simulate the walking path. If the distance between the vehicle body or bucket and the obstacle is predicted to be less than a safety threshold (the safety threshold is half the length of the electric shovel body), the control input of the fourth step is corrected. If the distance between the vehicle body or bucket and the obstacle is less than one-quarter of the vehicle body length, a deceleration command is sent through the 5G URLLC channel. If the distance between the vehicle body or bucket and the obstacle is less than one-twentieth of the vehicle body length, an emergency stop command is sent through the 5G URLLC channel.

[0017] Step six involves recording the operational data from the mining and walking modes in step four, along with the equipment response data processed in step 3.1, and feeding this data back into the reinforcement learning model in step five. This continuously updates the training dataset of the reinforcement learning model, optimizing the control strategy through continuous learning to gradually improve the accuracy and efficiency of remote operations. Simultaneously, it supports saving and replaying historical operation processes on AR or MR headsets for experience summarization and skill enhancement. The operational data consists of the angle changes of the left and right single-degree-of-freedom force feedback joysticks.

[0018] The beneficial effects of this invention are as follows:

[0019] (1) By processing the multi-source sensing data collected by the electric shovel end sensor through the multi-source data fusion algorithm, a high-fidelity three-dimensional digital model of the operation scene is constructed, which significantly improves the recognition accuracy of ore accumulation pattern in complex scenes.

[0020] (2) The Kalman filter algorithm is used to process the joystick pose signal. The network transmission delay is predicted by the state vector and the command timing is dynamically calibrated to achieve spatiotemporal consistency calibration of the control command, which significantly improves the accuracy and real-time performance of the electric shovel control.

[0021] (3) The scene of fusion of three-dimensional digital model and binocular visual image is presented synchronously in AR or MR head-mounted display device. At the same time, the digital twin model of electric shovel is superimposed and the dynamic display of push rod extension stroke, lifting rope lifting trajectory, bucket opening and closing status, and left and right track walking path operation parameters are displayed. The simulated operation environment is constructed based on the mapping of rocker resistance and load pressure. At the same time, when the collision risk is detected in the pre-drilling electric shovel digging path or the distance between the vehicle body or bucket and the obstacle is less than the safety threshold in the walking mode, the control input is corrected by priority transmission through 5G URLLC channel, which effectively improves the intuitiveness and safety of remote operation.

[0022] (4) Based on the reinforcement learning model, the push-lifting collaborative strategy and differential speed control strategy are generated by training historical operation data. By continuously recording operation data and equipment response information, the training dataset of the reinforcement learning model is constantly updated. At the same time, it supports saving and replaying historical operation processes for experience summarization and skill improvement, which effectively improves the operation efficiency of electric shovels. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the present invention.

[0024] Figure 2 This is a diagram showing the deployment locations of the sensors at the end of the electric shovel.

[0025] In the diagram: 1. Binocular camera; 2. Inclinometer; 3. LiDAR; 4. Tension sensor; 5. High-precision encoder; 6. Magnetic sensor; 7. Inertial measurement unit. Detailed Implementation

[0026] like Figure 1 The diagram shown is a schematic diagram of the present invention. The present invention will now be described in further detail with reference to specific embodiments.

[0027] A remote control method for large electric shovels used in mining based on AR or MR mainly includes the following steps:

[0028] The first step is to use sensors at the end of the electric shovel to perform environmental perception and data collection, thereby obtaining multi-source perception data.

[0029] The electric shovel end sensor, such as Figure 2 As shown, the device includes a binocular camera 1 symmetrically mounted on the top of the electric shovel's cab, an inclinometer 2 mounted on the push rod, a lidar 3 mounted on the front end of the electric shovel's boom, a tension sensor 4 mounted on the lifting rope, high-precision encoders 5 mounted on the push motor, the lifting drum motor, and the left and right travel motors, respectively, a magnetic sensor 6 mounted on the bucket, and an inertial measurement unit 7 mounted at the center of the slewing platform.

[0030] A binocular camera 1, mounted on the top of the electric shovel's cab, acquires real-time binocular visual images and stereoscopic visual data of the working scene. An inclinometer 2, installed at the push rod, collects the angle between the push rod and the horizontal plane. A lidar 3, deployed at the front end of the electric shovel's boom, acquires real-time 3D point cloud data of the working area. A tension sensor 4, installed at the lifting rope, collects real-time load status data of the bucket. High-precision encoders 5, installed at the push motor, lifting drum motor, and left and right travel motors, collect real-time operating status data of the electric shovel's push rod, lifting rope, and left and right travel tracks. A magnetic sensor 6, installed at the bucket, collects bucket opening and closing status data. An inertial measurement unit 7, installed at the center of the slewing platform, collects real-time attitude data of the equipment, i.e., the rotation status data of the slewing platform. All sensor data are synchronously acquired and preprocessed at the hardware level to form time-aligned multi-source sensing data.

[0031] The multi-source sensing data includes the operating status data of the electric shovel actuator. The electric shovel actuator includes a push rod, a lifting rope, left and right travel tracks, a bucket, and a slewing platform. The electric shovel actuator is controlled by a corresponding drive mechanism, including a push motor that drives the push rod to extend and retract, a lifting drum motor that drives the lifting rope to rise and fall, a left and right travel motor that drives the left and right travel tracks to move, a bucket motor that drives the bucket to open and close, and a slewing motor that drives the slewing platform to rotate.

[0032] The second step involves processing the stereoscopic vision data and 3D point cloud data obtained from the first step using a multi-source data fusion algorithm to construct a 3D digital model of the work scene. This significantly improves the accuracy of identifying ore accumulation patterns in complex scenarios. The specific steps are as follows:

[0033] Step 2.1: Perform stereo matching on the stereo vision data using a stereo matching algorithm to generate a high-precision depth map. The formula is as follows:

[0034]

[0035] In the formula, z is the visual depth, f is the camera focal length, B is the baseline distance, and d is the disparity value of the matched pixel.

[0036] Step 2.2: The improved iterative nearest-point algorithm is used to register the 3D point cloud data. The objective function is:

[0037]

[0038] In the formula, R is the rotation matrix, p i Let q be the 3D coordinates of the i-th point in the 3D point cloud data. j Let be the 3D coordinates of the j-th point in the high-precision depth map, t be the translation vector, λ be the regularization coefficient, and N be the number of effective matching point pairs.

[0039] In the formula, λ is used to suppress noise interference and achieve accurate registration between the lidar point cloud and the binocular depth map; Tr(R T R) means matrix R T The trace of R is used to penalize the non-orthogonality of the rotation matrix, ensuring the rotation is reasonable.

[0040] Step 2.3: The registered point cloud and depth map are probabilistically fused using a raster. The fusion confidence is calculated using a Bayesian update rule, as shown in formula (3):

[0041]

[0042] In the formula, P lidar P represents the confidence level of the lidar sensor in detecting a target or area, with a value ranging from [0-1]. vision This represents the confidence level of the binocular camera in detecting a target or region, with a value ranging from [0 to 1].

[0043] The third step is to construct a digital twin model of the electric shovel and achieve synchronized presentation of the virtual and real scenes and synchronized updates of the digital twin model of the electric shovel within an AR or MR headset. The specific steps are as follows:

[0044] Step 3.1: Perform second-order low-pass filtering on the tiltmeter angle data, load status data, actuator motion status data, bucket opening and closing status data, and attitude data of the electric shovel obtained in the first step to obtain the electric shovel's equipment response data.

[0045] Step 3.2: Measure the geometric parameters of the electric shovel and draw its 3D model. These geometric parameters include the length, width, and height of the cab; the length, width, and height of the vehicle body; the length, width, and height of the tracks; the center distance between the left and right tracks; the height and diameter of the slewing platform; the length, width, and height of the push rod; the length, width, and height of the bucket; the length, width, and height of the boom; the angle between the boom and the horizontal plane; and the diameter and thickness of the top wheel. The drawn 3D model of the electric shovel consists of an upper structure, an lower structure, and a digging structure. The upper structure includes the cab, vehicle body, slewing platform, and boom; the lower structure includes the left and right tracks; and the digging structure includes the push rod and bucket.

[0046] Step 3.3: Based on the 3D model of the electric shovel from Step 3.2, perform kinematic analysis of the electric shovel. The specific steps are as follows:

[0047] Step 3.3.1: For the upper structure, its kinematic equation is a single-degree-of-freedom rotation. The rotation angle of the slewing platform can be obtained from the data collected by the inertial measurement unit 7 processed in step 3.1.

[0048] Step 3.3.2, for the undercarriage structure, the instantaneous distances traveled by the left and right tracks, collected by the high-precision encoders 5 at the left and right walking motors processed in Step 3.1, can be used to determine the coordinate change of the undercarriage structure center relative to the previous moment, which is (Δψ, Δs). Δψ represents the yaw angle change, and Δs represents the displacement change in the heading direction, specifically expressed as follows:

[0049]

[0050] In the formula, s R s represents the instantaneous distance traveled by the right track. L B represents the instantaneous distance traveled by the left track. S The center distance between the left and right walking tracks.

[0051]

[0052] Step 3.3.3: For the excavation structure, the elongation of the push rod relative to the push rod motor and the angle of the push rod relative to the horizontal plane are obtained from the state data of the push rod motor processed in Step 3.1 and the data from the inclinometer 2. Based on the kinematic equations, the polar coordinates of the hinge point between the push rod and the bucket relative to the center of the push rod motor are derived as (ρ, θ), where ρ is the elongation of the push rod, i.e., the distance between the hinge point of the push rod and the bucket and the center of the push rod motor, and θ is the angle of the push rod relative to the horizontal plane. The opening and closing state of the bucket is determined by the data collected by the magnetic sensor 6.

[0053] Step 3.4: Construct a digital twin model of the electric shovel using the response data of the electric shovel in Step 3.1, the three-dimensional model of the electric shovel in Step 3.2, and the kinematic equations derived in Step 3.3.

[0054] Step 3.5: Install a real-time rendering engine in the AR or MR head-mounted display device to merge the real-time acquired binocular visual images of the work scene with the 3D digital model of the work scene obtained in the second step and the digital twin model of the electric shovel from step 3.4, so as to realize the synchronous presentation of the virtual and real scene and the digital twin model of the electric shovel. Import the electric shovel equipment response data processed in step 3.1 into the constructed digital twin model of the electric shovel to realize the synchronous reconstruction of the digital twin model of the electric shovel. Simultaneously overlay and display the digital twin model of the electric shovel and dynamically display the electric shovel bucket opening and closing, slewing platform rotation, push rod extension and retraction, lifting rope lifting and lowering, and left and right walking track walking operation parameters in the virtual and real scene.

[0055] The fourth step involves building a multimodal interaction system on the remote control terminal based on the AR or MR head-mounted display device from the third step. This system establishes a dynamic mapping model between remote operation signals and the response data of the electric shovel equipment, and interprets the multi-dimensional control inputs into action parameters for the opening and closing of the electric shovel bucket, the rotation of the slewing platform, the extension and retraction of the push rod, the raising and lowering of the lifting rope, and the movement of the left and right walking tracks, thereby achieving a remote operation that combines virtual and real elements.

[0056] Control input is achieved through a remote controller integrating a mode switching button, a bucket opening / closing button, a slewing knob, and two single-degree-of-freedom force feedback joysticks on the left and right. The joystick pose change signals are processed using a Kalman filter algorithm to predict network transmission delay and dynamically calibrate command timing, ensuring the accuracy and real-time performance of control commands. The mode switching button is mapped to digging and walking modes; the bucket opening / closing button's on / off state is mapped to the opening and closing status of the electric shovel bucket; the slewing knob's torsional motion is mapped to the rotation speed of the slewing platform; in digging mode, the forward / backward angle of the left joystick is mapped to the extension / retraction rate of the push rod, and the forward / backward angle of the right joystick is mapped to the lifting / lowering rate of the hoisting rope; in walking mode, the forward / backward angles of the left and right joysticks are mapped to the forward or backward speed of the left and right tracks.

[0057] The specific steps of the Kalman filter algorithm are as follows:

[0058] Step 4.1, Set the initial state vector Let the initial covariance matrix be P0, where P0 represents the uncertainty of the state estimate.

[0059] Step 4.2, perform status monitoring:

[0060]

[0061] In the formula, For prior state estimation, Let A be the state vector from the previous time step, B be the state transition matrix, and u be the control input matrix. k This is the control input for the current moment.

[0062] The state transition matrix and control input matrix are expressed as follows:

[0063]

[0064] In the formula, T s Where J is the sampling period, and J is the moment of inertia of the rocker arm.

[0065] The state vector of the previous moment can be expressed as:

[0066]

[0067] In the formula, θ handle The joystick tilt angle, Let Δt be the angular velocity of the joystick. delay This is due to network transmission delay.

[0068] Step 4.3, predict the covariance matrix:

[0069] P k|k-1 =AP k-1 A T +Q (10)

[0070] In the formula, P k|k-1 Let P be the prior covariance matrix. k-1 Let be the covariance matrix of the previous time step, and Q be the process noise covariance matrix, representing the model uncertainty.

[0071] Step 4.4, calculate the Kalman gain:

[0072] K k =P k|k-1 H T HP k|k-1 H T +R) -1 (11)

[0073] In the formula, H is the observation matrix, which represents mapping the state to the measurement space, and R is the measurement noise covariance matrix, which represents the sensor error.

[0074] Step 4.5: Perform state correction and covariance update.

[0075]

[0076] In the formula, For posterior state vector estimation, z k These are actual measured values.

[0077] P k =(IK k H)P k|k-1 (13)

[0078] In the formula, P kLet I be the covariance matrix at the current time, and let I be the identity matrix.

[0079] The Kalman filter algorithm achieves optimal state estimation through iterative prediction and updating, combining the predictive model and actual measurements. Its core lies in the dynamic management of the covariance matrix, balancing the uncertainties of model prediction and sensor measurements, ultimately achieving both accuracy and real-time control of the electric shovel drive mechanism.

[0080] The fifth step involves collecting historical operation data and equipment response information from multiple sets of electric shovels as initial training data. A reinforcement learning model based on this training data is used to optimize the push-lift coordination strategy. Using the control information from the digging mode in the fourth step as a foundation, the digital twin model of the electric shovel is used to perform real-time pre-simulation of the bucket's movement path, i.e., the digging trajectory, and this path is then synchronously displayed on an AR or MR head-mounted display.

[0081] When the predicted bucket movement path indicates a collision risk, the reinforcement learning model automatically generates a modified trajectory to correct the control input in step four, and prioritizes its transmission to the electric shovel actuator via a 5G URLLC (Ultra-Reliable Low-Latency Communication) channel. Simultaneously, the haptic feedback module adjusts the joystick resistance gradient in real time based on the bucket load pressure.

[0082] The joystick resistance is mapped to the load pressure as follows:

[0083] F feedback =k f F load +b f (14)

[0084] In the formula, F feedback F represents the magnitude of the joystick feedback force. load k represents the actual bucket resistance. f b is the force conversion coefficient. f This is the intercept coefficient.

[0085] For the walking task, the reinforcement learning model optimizes the differential speed control strategy of the left and right walking tracks of the electric shovel and performs a pre-simulation of the walking path based on the training data. When abnormal resistance on one side of the electric shovel's track is detected, the torque distribution of the left and right walking motors is automatically adjusted. Based on the control information of the fourth walking mode, the digital twin model of the electric shovel is presented in the AR or MR head-mounted display device to pre-simulate the walking path. If the distance between the vehicle body or bucket and the obstacle is predicted to be less than a safety threshold (the safety threshold is half the length of the electric shovel body), the control input of the fourth step is corrected. If the distance between the vehicle body or bucket and the obstacle is less than one-quarter of the vehicle body length, a deceleration command is sent through the 5G URLLC channel. If the distance between the vehicle body or bucket and the obstacle is less than one-twentieth of the vehicle body length, an emergency stop command is sent through the 5G URLLC channel.

[0086] Step six involves recording the operational data from the mining and walking modes in step four, along with the equipment response information obtained in step 3.1, and feeding this data back into the reinforcement learning model in step five. This continuously updates the training dataset of the reinforcement learning model, optimizing the control strategy through continuous learning to gradually improve the accuracy and efficiency of remote operations. Simultaneously, it supports saving and replaying historical operation processes on AR or MR headsets for experience summarization and skill enhancement. The operational data consists of the angle changes of the left and right single-degree-of-freedom force feedback joysticks.

[0087] The above embodiments are merely illustrative of the implementation methods of the present invention, but should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the protection scope of the present invention.

Claims

1. A remote control method for large electric shovels used in mining based on AR or MR, characterized in that, The remote control method for large electric shovels used in mining includes the following steps: The first step is to use sensors at the electric shovel end to perform environmental perception and data collection, thereby obtaining multi-source perception data; The electric shovel end sensors include: binocular cameras symmetrically installed on the top of the electric shovel's cab for real-time acquisition of binocular visual images and stereoscopic visual data of the working scene; an inclinometer installed at the push rod for acquiring the angle between the push rod and the horizontal plane; a lidar installed at the front end of the electric shovel's boom for real-time acquisition of three-dimensional point cloud data of the working area; a tension sensor installed at the lifting rope for real-time acquisition of the bucket's load status data; high-precision encoders installed at the push motor, lifting drum motor, and left and right travel motors for real-time acquisition of the electric shovel's push rod, lifting rope, and left and right travel track operating status data; a magnetic sensor installed at the bucket for acquiring bucket opening and closing status data; and an inertial measurement unit installed at the center of the slewing platform for real-time acquisition of the equipment's attitude data, i.e., the rotation status data of the slewing platform. All sensor data are synchronously acquired and preprocessed at the hardware level to form time-aligned multi-source sensing data, which includes the operating status data of the electric shovel actuator. The second step involves processing the stereo vision data and 3D point cloud data obtained from the first step using a multi-source data fusion algorithm to construct a 3D digital model of the work scenario. The third step is to process the multi-source perception data obtained in the first step to obtain the device response data, draw a three-dimensional model of the electric shovel and perform kinematic analysis, and then construct a digital twin model of the electric shovel. The virtual and real scenes are then presented synchronously and the digital twin model of the electric shovel is updated synchronously in the AR or MR head-mounted display device. The fourth step is to build a multimodal interaction system on the remote control terminal based on the AR or MR head-mounted display device in the third step, establish a dynamic mapping model between remote operation signals and electric shovel response data, and parse the multi-dimensional control input into the action parameters of electric shovel bucket opening and closing, slewing platform rotation, push rod extension and retraction, lifting rope lifting and lowering, and left and right walking track movement, so as to realize the remote operation of virtual and real combination. The fifth step involves collecting historical operation data and equipment response information from multiple sets of electric shovels as initial training data. The push-lifting coordination strategy is optimized using a reinforcement learning model based on the training data. Based on the control information in the digging mode of the fourth step, the electric shovel digital twin model is used to perform real-time pre-play of the bucket movement path, i.e., the digging trajectory, and the path is synchronously displayed in an AR or MR head-mounted display device. The sixth step involves recording the operational data from the mining and walking modes in the fourth step, along with the equipment response data obtained in step 3.1, and feeding this data back into the reinforcement learning model in the fifth step. This continuously updates the training dataset of the reinforcement learning model, optimizing the control strategy through continuous learning, and gradually improving the accuracy and efficiency of remote operations. Simultaneously, it supports saving and replaying historical operation processes in AR or MR headsets for experience summarization and skill enhancement. The operational data consists of the angle changes of the left and right single-degree-of-freedom force feedback joysticks.

2. The remote control method for a large electric shovel for mining based on AR or MR according to claim 1, characterized in that, In the first step, the electric shovel actuator is controlled by a corresponding drive mechanism, including a pusher motor that drives the push rod to extend and retract, a lifting drum motor that drives the lifting rope to rise and fall, a left and right travel motor that drives the left and right travel tracks to travel, a bucket motor that drives the bucket to open and close, and a slewing motor that drives the slewing platform to rotate.

3. The remote control method for a large electric shovel for mining based on AR or MR according to claim 2, characterized in that, The second step is specifically as follows: Step 2.1: Perform stereo matching on the stereo vision data using a stereo matching algorithm to generate a high-precision depth map. The formula is as follows: In the formula, z is the visual depth, f is the camera focal length, B is the baseline distance, and d is the disparity value of the matched pixel. Step 2.2: The improved iterative nearest-point algorithm is used to register the 3D point cloud data. The objective function is: In the formula, R is the rotation matrix; p i Let q be the 3D coordinates of the i-th point in the 3D point cloud data; j Δ is the 3D coordinate of the j-th point in the high-precision depth map; t is the translation vector; λ is the regularization coefficient used to suppress noise interference and achieve accurate registration between the lidar point cloud and the stereo depth map; N is the number of effective matching point pairs; Tr(R T R) is a matrix R T The trace of R; Step 2.3: The registered point cloud and depth map are probabilistically fused using a raster. The fusion confidence is calculated using a Bayesian update rule, as shown in formula (3): In the formula, P lidar P represents the confidence level of the lidar sensor in detecting a target or area, with a value ranging from [0-1]. vision This represents the confidence level of the binocular camera in detecting a target or region, with a value ranging from [0 to 1].

4. The remote control method for a large electric shovel for mining based on AR or MR according to claim 3, characterized in that, The third step is specifically as follows: Step 3.1: Perform second-order low-pass filtering on the tiltmeter angle data, load status data, actuator motion status data, bucket opening and closing status data and attitude data of the electric shovel obtained in the first step to obtain the equipment response data of the electric shovel. Step 3.2: Measure the geometric parameters of the electric shovel and draw a 3D model of the electric shovel; Step 3.3: Based on the 3D model of the electric shovel from Step 3.2, perform kinematic analysis of the electric shovel. The specific steps are as follows: Step 3.3.1: For the upper structure, its kinematic equation is a single-degree-of-freedom rotation. The rotation angle of the rotary platform is obtained from the attitude data processed in Step 3.

1. Step 3.3.2, for the undercarriage structure, the instantaneous distance traveled by the left and right tracks, collected by the high-precision encoders at the left and right travel motors processed in Step 3.1, yields the coordinate change of the undercarriage structure center relative to the previous moment as (Δψ, Δs), where Δψ is the yaw angle change and Δs is the displacement change in the heading direction, specifically expressed as: In the formula, s R s represents the instantaneous distance traveled by the right track. L B represents the instantaneous distance traveled by the left track. S The center distance between the left and right tracks; Step 3.3.3: For the excavation structure, the elongation of the push rod relative to the push rod motor and the angle of the push rod relative to the horizontal plane are obtained from the state data of the push rod motor and the inclinometer data processed in Step 3.

1. According to the kinematic equation, the polar coordinates of the hinge point between the push rod and the bucket relative to the center of the push rod motor are (ρ, θ), where ρ is the elongation of the push rod, that is, the distance between the hinge point between the push rod and the bucket and the center of the push rod motor, and θ is the angle of the push rod relative to the horizontal plane; the opening and closing state of the bucket is determined by the data collected by the magnetic sensor. Step 3.4: Construct a digital twin model of the electric shovel using the response data of the electric shovel from Step 3.1, the three-dimensional model of the electric shovel from Step 3.2, and the kinematic equations derived from Step 3.

3. Step 3.5: Install a real-time rendering engine within the AR or MR head-mounted display device to merge the real-time acquired binocular visual images of the work scene with the 3D digital model of the work scene obtained in the second step and the digital twin model of the electric shovel from Step 3.4, thereby achieving synchronous presentation of the virtual and real scenes with the digital twin model of the electric shovel; import the electric shovel equipment response data processed in Step 3.1 into the constructed digital twin model of the electric shovel to achieve synchronous reconstruction of the digital twin model of the electric shovel, and simultaneously overlay and display the digital twin model of the electric shovel and dynamically display the electric shovel bucket opening and closing, slewing platform rotation, push rod extension and retraction, lifting rope lifting and lowering, and left and right walking track walking operation parameters in the virtual and real scenes.

5. A remote control method for a large electric shovel for mining based on AR or MR according to claim 4, characterized in that, In step 3.2, the geometric parameters include the length, width and height of the cab, the length, width and height of the vehicle body, the length, width and height of the tracks, the center distance between the left and right tracks, the height and diameter of the slewing platform, the length, width and height of the push rod, the length, width and height of the bucket, the length, width and height of the boom, the angle between the boom and the horizontal plane, and the diameter and thickness of the sheave. The drawn 3D model of the electric shovel consists of an upper structure, an undercarriage structure, and an excavation structure. The upper structure includes the cab, body, slewing platform, and boom. The undercarriage structure includes the left and right tracks. The excavation structure includes the push rod and bucket.

6. A remote control method for a large electric shovel for mining based on AR or MR according to claim 5, characterized in that, In the fourth step, control input is achieved through a remote controller integrating a mode switching button, a bucket opening / closing button, a rotary knob, and two single-degree-of-freedom force feedback joysticks on the left and right. The joystick's pose change signal is processed by a Kalman filter algorithm to predict network transmission delay and dynamically calibrate command timing, ensuring the accuracy and real-time performance of control commands. The mode switching button is mapped to digging mode and walking mode, and the on / off state of the bucket opening / closing button is mapped to the opening and closing state of the electric shovel bucket. The torsional motion of the rotary knob is mapped to the rotation speed of the rotary platform. In digging mode, the forward / backward angle of the left joystick is mapped to the extension / retraction rate of the push rod, and the forward / backward angle of the right joystick is mapped to the lifting / lowering rate of the hoisting rope. In walking mode, the forward and backward angles of the left and right joysticks are mapped to the forward or backward speed of the left and right tracks.

7. A remote control method for a large electric shovel for mining based on AR or MR according to claim 6, characterized in that, The specific steps of the Kalman filter algorithm in the fourth step are as follows: Step 4.1, Set the initial state vector Set an initial covariance matrix P0, where P0 represents the uncertainty of the state estimate; Step 4.2, perform status monitoring: In the formula, For prior state estimation, Let A be the state vector from the previous time step, B be the state transition matrix, and u be the control input matrix. k For the control input at the current moment; The state transition matrix and control input matrix are expressed as follows: In the formula, T s The sampling period is J, and the moment of inertia of the joystick is J. The state vector of the previous moment can be expressed as: In the formula, θ handle The joystick tilt angle, Let Δt be the angular velocity of the joystick. delay This is due to network transmission delay; Step 4.3, predict the covariance matrix: P k|k-1 =AP k-1 From T +Q (10) In the formula, P k|k-1 Let P be the prior covariance matrix. k-1 Let be the covariance matrix of the previous time step, and Q be the process noise covariance matrix, representing the model uncertainty. Step 4.4, calculate the Kalman gain: K k =P k|k-1 H T (HP k|k-1 H T +R) -1 (11) In the formula, H is the observation matrix, which represents mapping the state to the measurement space, and R is the measurement noise covariance matrix, which represents the sensor error; Step 4.5: Perform state correction and covariance update. In the formula, For posterior state vector estimation, z k These are actual measured values; P k =(I-K k H)P k|k-1 (13) In the formula, P k Let I be the covariance matrix at the current time, and let I be the identity matrix. The Kalman filter algorithm achieves optimal estimation of the state through iterative prediction and update, combining the prediction model and actual measurement. Its core lies in the dynamic management of the covariance matrix, balancing the uncertainties of model prediction and sensor measurement, and ultimately achieving precision and real-time control of the electric shovel drive mechanism.

8. A remote control method for a large electric shovel for mining based on AR or MR according to claim 7, characterized in that, In the fifth step, when the predicted collision risk of the bucket movement path is detected during the pre-simulation, the reinforcement learning model automatically generates a modified trajectory to correct the control input in the fourth step, and transmits it to the electric shovel actuator via the 5G URLLC channel. At the same time, the haptic feedback module adjusts the rocker resistance gradient in real time according to the bucket load pressure. The joystick resistance is mapped to the load pressure as follows: F feedback =k f F load +b f (14) In the formula, F feedback F represents the magnitude of the joystick feedback force. load k represents the actual bucket resistance. f b is the force conversion coefficient. f This is the intercept coefficient.

9. A remote control method for a large electric shovel for mining based on AR or MR according to claim 8, characterized in that, In the fifth step, for the walking task, the reinforcement learning model optimizes the differential speed control strategy of the left and right tracks of the electric shovel and performs a pre-simulation of the walking path based on the training data. When abnormal resistance on one side of the electric shovel's track is detected, the torque distribution of the left and right walking motors is automatically adjusted. Based on the control information of the fourth walking mode, the digital twin model of the electric shovel is simultaneously presented in the AR or MR head-mounted display device to pre-simulate the walking path. If the distance between the vehicle body or bucket and the obstacle is predicted to be less than the safety threshold, the control input of the fourth step is corrected. If the distance between the vehicle body or bucket and the obstacle is less than one-quarter of the vehicle body length, a deceleration command is sent through the 5G URLLC channel. If the distance between the vehicle body or bucket and the obstacle is less than one-twentieth of the vehicle body length, an emergency stop command is sent through the 5G URLLC channel.

10. A remote control method for a large electric shovel for mining based on AR or MR according to claim 9, characterized in that, The safety threshold is half the length of the electric shovel body.

Images