Mining and loading operation method and system for mining electric shovel in large scene

Abstract

The invention discloses a mining electric shovel large-scene mining and loading operation method and system, and the method comprises the steps: S1, controlling an unmanned aerial vehicle to rise to a preset height, and carrying out the scanning and shooting of a mine environment through a laser radar and a TOF camera which are carried on the unmanned aerial vehicle, so as to collect the global point cloud data of the mine environment; s2, performing point cloud registration on TOF camera data and laser radar data; s3, separating the to-be-excavated material point cloud and the mine card point cloud from the global point cloud data, positioning a container of the mine card, and performing surface fitting on the to-be-excavated material point cloud; s4, according to the position of the container of the mine truck, determining the position coordinates of the unloading of the bucket of the electric shovel, and fitting the point cloud of the container of the mine truck so as to fit an outer surrounding frame of the container of the mine truck; and S5, according to the point cloud of the materials to be excavated and the point cloud of the mine trucks, the walking track of the mine trucks, the walking excavation loading track of the electric shovel and the moving track of the unmanned aerial vehicle are planned, so that the electric shovel, the multiple mine trucks and the unmanned aerial vehicle cooperatively conduct mining and loading operation.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent operation in open-pit mines, and in particular to a large-scale mining and loading operation method and system for a mining electric shovel. Background Art

[0002] In mining, the development of environmental perception and autonomous operation technology has gradually attracted widespread attention. Traditional mining excavation and loading operations rely on a large number of manual operations, which is inefficient and easily affected by the complex mining environment, posing a serious threat to the safety of operators. Therefore, mining operations are gradually shifting towards intelligence and automation, especially in the field of autonomous operations combining drones and robots, which has made significant progress. However, the current mainstream open-pit mine environmental perception technology mainly relies on ground fixed sensors and laser radars and other equipment. Although these devices can provide environmental information within a certain range, they have limited coverage, many perception blind spots, and difficulty in quickly adapting to dynamic changes, resulting in perception data that is not timely and accurate enough.

[0003] In order to solve these problems, drone technology has been introduced into open-pit mine environmental detection, significantly improving the transmission efficiency and flexibility of operation information. Tethered drones can achieve a wider range of scene perception and information transmission, effectively making up for the blind spots of ground sensors, and are more suitable for the intelligent operation needs of open-pit mining equipment. However, existing drone technology still has certain limitations in perception accuracy, real-time and intelligent decision-making, and it is difficult to meet the operation requirements in all-weather, large-scale and complex environments. In order to further improve the level of intelligence in mining operations, the collaborative operation between drones and mining electric shovels has become one of the key technologies that need to be solved urgently.

[0004] In summary, although drones are gradually showing their unique advantages in mining operations, the effective combination of their perception capabilities and the operating capabilities of mining electric shovels still faces many challenges. Further research and development of efficient and intelligent collaborative systems is needed to improve the overall efficiency and safety of mining operations. Summary of the invention

[0005] The present invention aims to solve at least one of the technical problems in the related art to a certain extent. To this end, the first purpose of the present invention is to propose a mining and loading operation method for a large-scale mining electric shovel, which can be applied to the three-dimensional perception of the working environment of the electric shovel in the large-scale open-pit mine, the dynamic planning of the excavation trajectory of the electric shovel and the walking trajectory of the mining truck, and the coordinated operation of the electric shovel, mining truck and unmanned aerial vehicle, so as to achieve the goals of high efficiency, energy saving and safety in the open-pit mining process under complex environments.

[0006] The second purpose of the present invention is to provide a large-scale mining and loading operation system for a mining electric shovel.

[0007] To achieve the above object, an embodiment of the first aspect of the present invention provides a mining operation method for a large-scale scenario of a mining electric shovel. The large-scale scenario of the mining electric shovel includes an unmanned aerial vehicle (UAV), an electric shovel, and a plurality of mining trucks. The UAV is communicatively connected to the electric shovel and the plurality of mining trucks respectively. The method includes:

[0008] S1, controlling the UAV to rise to a preset height, and scanning and photographing the mine environment through a lidar and a TOF camera mounted on the UAV to collect global point cloud data of the mine environment. The global point cloud data includes the attitude information and position information of the electric shovel, the attitude information and position information of the plurality of mining trucks, and the three-dimensional information of the material to be excavated;

[0009] S2, performing point cloud registration on the TOF camera data and the lidar data;

[0010] S3, separating the point cloud of the material to be excavated and the point cloud of the mining truck from the global point cloud data, positioning the cargo box of the mining truck, and performing surface fitting on the point cloud of the material to be excavated;

[0011] S4, determining the position coordinates of the electric shovel bucket unloading according to the position of the cargo box of the mining truck, and fitting the point cloud of the cargo box of the mining truck to fit the outer bounding box of the cargo box of the mining truck;

[0012] S5, planning the travel trajectory of the mining truck, the travel, excavation, and loading trajectory of the electric shovel, and the movement trajectory of the UAV respectively according to the point cloud of the material to be excavated and the point cloud of the mining truck, so that the electric shovel, the plurality of mining trucks, and the UAV cooperate to perform mining operations.

[0013] In addition, according to the mining operation method for a large-scale scenario of a mining electric shovel in the above embodiment of the present invention, the following additional technical features may also be provided:

[0014] According to an embodiment of the present invention, in step S1, the UAV is a tethered UAV, the lidar is a multi-line lidar, the multi-line lidar scans the surrounding environment of the electric shovel to obtain the point cloud data of the surrounding environment of the electric shovel, and the TOF camera obtains the depth map of the mine environment.

[0015] According to an embodiment of the present invention, step S2 includes:

[0016] Determining the focal lengths f in the x and y directions of the TOF camera through camera calibration x 、f y ,the optical centers c in the x and y directions x 、c y ,and converting the depth image pixel coordinates (u, v) into points (X, Y, Z) in the 3D space through the following formula:

[0017]

[0018] Z = D(u, v)

[0019] where D(u, v) is the depth value corresponding to the pixel point (u, v) in the depth image;

[0020] Through external parameter calibration, the transformation matrix T determined by the rotation matrix R and the translation vector t is calculated:

[0021]

[0022] Using the calibrated transformation matrix T, the point cloud generated by the TOF camera is transformed from the camera coordinate system to the lidar coordinate system:

[0023] P L i DAR = T · P TOF

[0024] where P LiDAR is the lidar coordinate system, and P TOF is the camera coordinate system;

[0025] After the coordinate transformation is completed, the point cloud generated by the TOF camera and the point cloud generated by the lidar are in the same coordinate system, so as to merge the TOF camera data and the lidar data for point cloud registration.

[0026] According to an embodiment of the present invention, step S3 includes:

[0027] Removing the noise points in the point cloud data through a median filter;

[0028] Reducing the point cloud density by a voxel grid method to maintain the main features while accelerating the subsequent processing speed;

[0029] Calculating the normal vector of the point cloud through the principal component analysis of the neighborhood points for feature extraction, where the normal vector of the point cloud is calculated by the following formula:

[0030]

[0031] where Ni is the neighborhood of the point Pi and Vi is the normal vector;

[0032] Using the random sample consensus algorithm to segment the point cloud of the material to be mined, the point cloud of the ore block and the obstacle point cloud from the global point cloud data, and performing surface fitting on the material heap surface and the ground.

[0033] According to an embodiment of the present invention, step S4 includes:

[0034] Identify boundary points by calculating the normal vectors and curvatures of the point cloud to perform boundary extraction on the ore truck cargo box. The point cloud curvature calculation formula is as follows:

[0035]

[0036] where N(p) is the neighborhood point set of point p, n(p) is the normal vector of point p, and κ(p) is the curvature of point p;

[0037] Utilize the aggregation characteristics of boundary points and adopt the Harris corner detection algorithm to detect the corners of the ore truck cargo box, and use the random sample consensus algorithm to fit the plane of the ore truck cargo box;

[0038] Locate the geometric center and relative coordinates of the ore truck cargo box through the boundary points and surface features of the ore truck cargo box;

[0039] Obtain the main axis direction of the ore truck cargo box through the results of the principal component analysis method, and calculate the minimum circumscribed rectangle of the ore truck cargo box. The minimum circumscribed rectangle is the outer bounding box of the ore truck cargo box.

[0040] According to an embodiment of the present invention, in step S5, plan the walking trajectory of the electric shovel based on the point cloud of the material to be excavated and the point cloud of the ore truck, including:

[0041] Obtain the position coordinates of the electric shovel based on the point cloud data provided by the drone, and determine the docking position of the ore truck;

[0042] Adopt the hybrid A* algorithm, select key path points according to the map provided by the drone, and plan the globally optimal route;

[0043] When the ore truck approaches the excavation area, enter the dynamic environment and enable the dynamic window algorithm to perform local path planning for the ore truck;

[0044] Adopt the dynamic scheduling algorithm to adjust the task priorities of multiple ore trucks in real time.

[0045] According to an embodiment of the present invention, in step S5, plan the walking, excavation, and loading trajectories of the electric shovel based on the point cloud of the material to be excavated and the point cloud of the ore truck, including: the walking trajectory planning of the electric shovel, the excavation trajectory planning of the electric shovel, and the loading trajectory planning of the electric shovel. Among them,

[0046] The travel trajectory planning of the electric shovel includes: obtaining the remaining amount of the material to be excavated around the electric shovel. When the remaining amount of the material within the maximum operation radius of the electric shovel is less than the preset threshold, the hybrid A* algorithm is used to plan the moving path of the electric shovel and navigate it to the excavation position with more material in the environment. When the electric shovel approaches the target position, the dynamic window algorithm is used to perform local path planning for the electric shovel to achieve precise positioning of the electric shovel.

[0047] The excavation trajectory planning of the electric shovel includes: based on the point cloud information of the material to be excavated and combined with the self-attitude information fed back by the sensors of the electric shovel itself, with the goal of unit volume excavation energy consumption, optimizing the excavation trajectory, and using the particle swarm algorithm to select the best excavation path in the trajectory planning. By fitting the polynomial curve of the excavation angle and the stick length, the smooth adjustment of the trajectory is realized. If the planned path does not meet the set boundary conditions, the trajectory will be automatically regenerated to ensure the continuity of the operation, where the set boundary conditions include: the maximum excavation depth and the maximum lifting force.

[0048] The loading trajectory planning of the electric shovel includes: combining the forward and inverse kinematic solutions of the electric shovel to obtain the commands corresponding to the actual lifting distance and the pushing distance, and controlling the lifting motor and the pushing motor of the electric shovel according to the commands to jointly complete the excavation task. The attitude information transmitted back by the sensors of the electric shovel is used for closed-loop feedback of the excavation action execution quality and timely correction to achieve high-precision excavation.

[0049] According to an embodiment of the present invention, using the dynamic window algorithm to perform local path planning includes:

[0050] S51, initializing the state parameters of the device. The state parameters include the position (x, y), orientation (θ), speed (v), angular velocity (ω) of the device, and setting the maximum speed (vmax) and maximum acceleration (amax) limits, and setting the time step Δt.

[0051] S52, generating a feasible speed window according to the current state. The linear speed window is v ∈ [v min , v max and the angular velocity window is ω ∈ [ω min , ω max . For each time step, the speed range is:

[0052] V d = [v current - a max ·Δt, v current + a max ·Δt]

[0053] Ω d = [ω current - a max ·Δt, ωcurrent +a max ·Δt]

[0054] S53. For each speed combination, based on the current speed and angular velocity, predict a trajectory within a future time range, and calculate the position change within the time step;

[0055] S54. Evaluate the predicted trajectory against the evaluation metrics, and select the trajectory with the lowest cost as the motion plan for the current device, where the evaluation metrics include at least one of the distance to the target, the distance to the obstacle, and the trajectory smoothness;

[0056] S55. Generate linear velocity and angular velocity control commands according to the selected optimal trajectory, and drive the motor of the device to operate;

[0057] S56. Repeat steps S52 to S55 in each control cycle, continuously update the state and trajectory planning of the device to adapt to environmental changes.

[0058] According to an embodiment of the present invention, in step S5, planning the movement trajectory of the drone according to the to-be-mined material point cloud and the ore blocking point cloud includes:

[0059] The movement trajectory planning of the drone is synchronized with the movement trajectory planning of the electric shovel. The drone always stays above the electric shovel. When the electric shovel moves, the drone moves synchronously to provide stable environmental perception services for the electric shovel.

[0060] To achieve the above object, an embodiment of the second aspect of the present invention proposes a mining operation system for a large-scale scene of a mining electric shovel. The large-scale scene of the mining electric shovel includes a drone, an electric shovel, and multiple ore trucks. The drone is communicatively connected to the electric shovel and the multiple ore trucks respectively. The system includes:

[0061] An environmental perception module, configured to control the drone to rise to a preset height, and scan and photograph the mine environment through the lidar and TOF camera carried on the drone to collect global point cloud data of the mine environment. The global point cloud data includes the attitude information and position information of the electric shovel, the attitude information and position information of the ore truck, and the three-dimensional information of the to-be-mined material;

[0062] A point cloud registration module, configured to perform point cloud registration on the TOF camera data and the lidar data;

[0063] A material and ore block point cloud segmentation module, configured to separate the to-be-mined material point cloud and the ore block point cloud from the global point cloud data, and position the cargo box of the ore truck and perform surface fitting on the to-be-mined material point cloud;

[0064] The ore truck cargo box positioning and outer bounding box fitting module is used to determine the position coordinates of the electric shovel bucket for discharging according to the position of the cargo box of the ore truck, and fit the point cloud of the cargo box of the ore truck to fit the outer bounding box of the cargo box of the ore truck;

[0065] The trajectory planning module is used to plan the travel trajectory of the ore truck, the excavation and loading trajectory of the electric shovel, and the movement trajectory of the drone according to the point cloud of the material to be excavated and the point cloud of the ore truck respectively, so that the electric shovel, multiple ore trucks and the drone can cooperate to carry out the mining and loading operation.

[0066] Compared with the prior art, the present invention is based on the tethered drone environmental perception system and guaranteed by the multi-aircraft cooperation technology, effectively combining the environmental perception ability of the drone, the autonomous operation ability of the electric shovel and the adaptive scheduling ability of the ore truck to form an efficient and intelligent cooperation system, realizing the goals of high efficiency, energy saving and safety in the open-pit mining process under complex environments.

[0067] Additional aspects and advantages of the present invention will be given in part in the following description, become apparent in part from the following description, or be understood through the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 It is a flowchart of the mining and loading operation method for a large-scale scene of an electric shovel for mines according to an embodiment of the present invention;

[0069] Figure 2 It is a schematic diagram of a large-scale scene of an electric shovel for mines according to an embodiment of the present invention;

[0070] Figure 3 It is a schematic structural diagram of a drone according to an embodiment of the present invention;

[0071] Figure 4 It is a schematic structural diagram of an electric shovel according to an embodiment of the present invention;

[0072] Figure 5 It is a schematic diagram of positioning an electric shovel and an ore truck according to an embodiment of the present invention;

[0073] Figure 6 It is a schematic block diagram of the mining and loading operation system for a large-scale scene of an electric shovel for mines according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0074] Embodiments of the present invention will be described in detail below. Examples of the embodiments are shown in the drawings, where the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the drawings are exemplary and are intended to explain the present invention and should not be construed as limiting the present invention.

[0075] The present invention will be described below with reference to the accompanying drawings to present a method for mining and loading operations in a large-scale scenario of a mining electric shovel and a system for mining and loading operations in a large-scale scenario of a mining electric shovel.

[0076] In an embodiment of the present invention, as Figure 2 shown, in the large-scale scenario of the mining electric shovel, it includes a drone 1, an electric shovel 2, and multiple mining trucks 3. The drone 1 is communicatively connected to the electric shovel 2 and the multiple mining trucks 3 respectively, and the electric shovel 2 and the multiple mining trucks 3 perform collaborative mining and loading operations on the material 4. The drone 1 provides real-time environmental perception and data transmission services; the electric shovel 2 performs autonomous path planning and excavates the material 4; the multiple mining trucks 3 are responsible for material transportation and collaborative operations with the electric shovel 2.

[0077] As Figure 1 shown, the method for mining and loading operations in the large-scale scenario of the mining electric shovel according to the embodiment of the present invention may include the following steps:

[0078] S1. Control the drone to rise to a preset height, and scan and photograph the mine environment through the lidar and TOF camera carried on the drone to collect the global point cloud data of the mine environment. The global point cloud data includes the attitude information and position information of the electric shovel, the attitude information and position information of the multiple mining trucks, and the three-dimensional information of the material to be excavated. Among them, the preset height can be calibrated according to the mining area range, the specific performance of the drone, etc.

[0079] According to an embodiment of the present invention, in step S1, the drone is a tethered drone, the lidar is a multi-line lidar, the multi-line lidar scans the surrounding environment of the electric shovel to obtain the point cloud data of the surrounding environment of the electric shovel, and the TOF camera obtains the depth map of the mine environment.

[0080] Specifically, as Figure 3 shown, the drone is configured with a lidar 1-1, a TOF camera 1-2, and an edge computing device 1-3, which obtains the three-dimensional environment data of the mining area in real time and transmits the data to the electric shovel and the mining truck through the wireless communication module 1-4. The lidar 1-1 is used to generate a high-precision terrain model, the TOF camera 1-2 is responsible for providing the depth information of the material, and the edge computing device 1-3 processes the data locally to reduce communication latency. The drone communicates with the electric shovel 2 and the mining truck 3 through a wireless network to achieve the sharing of environmental information.

[0081] Specifically, control the tethered drone equipped with a 360° lidar and a TOF camera to rise to a preset height, scan and photograph the mine, obtain the attitude, position, etc. information of the electric shovel and the mining truck, obtain the three-dimensional information of the material to be excavated, and the collected global point cloud data is transmitted to the industrial computer of the electric shovel through a wireless network.

[0082] S2. Perform point cloud registration on the TOF camera data and the lidar data.

[0083] S3. Separate the point clouds of the material to be mined and the mining truck from the global point cloud data, locate the cargo box of the mining truck, and perform surface fitting on the point cloud of the material to be mined.

[0084] S4. Determine the position coordinates of the dipper discharge of the electric shovel according to the position of the cargo box of the mining truck, and fit the point cloud of the cargo box of the mining truck to fit the outer bounding box of the cargo box of the mining truck.

[0085] S5. Plan the travel trajectories of the mining truck, the travel, excavation and loading trajectories of the electric shovel, and the movement trajectory of the UAV according to the point clouds of the material to be mined and the mining truck respectively, so that the electric shovel, multiple mining trucks and the UAV can cooperate in the mining and loading operations.

[0086] Specifically, after receiving the global point cloud data transmitted by the UAV, the industrial computer on the electric shovel preprocesses the global point cloud data. First, fuse the TOF camera data and the lidar data for point cloud registration, and then separate the point clouds of the material to be mined, the mining truck, etc. from the global point cloud, and further locate the cargo box of the mining truck and perform surface fitting on the point cloud of the material. Then, locate the position of the cargo box of the mining truck to provide the bucket position coordinates for the electric shovel to discharge, and fit the outer bounding box to provide a basis for the electric shovel to avoid obstacles.

[0087] Furthermore, the industrial computer on the electric shovel determines the stopping position of the mining truck beside the electric shovel based on the point cloud information provided by the lidar, and determines the travel trajectory of the mining truck; the electric shovel plans the excavation trajectory according to the fitted stockpile point cloud according to a certain loading task; determines the discharge trajectory according to the starting point of the excavation trajectory and the position of the mining truck; when there is no material to be mined within the working range of the electric shovel, plan the electric shovel to move to the remaining material and continue to execute the excavation and discharge trajectory planning; the tethered UAV always stays above the electric shovel and moves synchronously when the electric shovel moves. The electric shovel is equipped with a torque sensor for the push-rod gear shaft, a wire rope tension sensor, a boom angle sensor, an encoder for the wire rope drum shaft, an encoder for the push-rod gear shaft, a boom wire displacement sensor, etc., which are used to monitor the pose and load information of the electric shovel in real time.

[0088] It should be noted that as Figure 4As shown in the figure, the electric shovel can be divided into a mechanical part and a control part in terms of composition. The mechanical part consists of a working device, a slewing device, and a traveling device. The working device of the electric shovel is the main device responsible for the excavation task and is composed of a crowding mechanism and a hoisting mechanism. The crowding mechanism can be divided into a dipper stick, a bucket, a crowding gear, a crowding rack, a transmission belt, an encoder 2-1, an inclination sensor 2-2, a wire-pulling displacement sensor 2-3, a three-phase asynchronous motor of 0.75kw, etc. The core components of the hoisting mechanism include a drum, a wire rope, a sheave, a boom, an encoder, a tension and compression sensor 2-4, and two three-phase asynchronous motors of 1.1kw. During the excavation process, the power provided by the drive motor is transmitted to the bucket through the transmission mechanism. The inclination sensor, wire-pulling displacement sensor, encoder, torque sensor 2-5, and tension and compression sensor installed on the working device are used to detect the operating parameters of the excavation movement. The crowding mechanism and the hoisting mechanism cooperate to jointly drive the bucket to complete the excavation operation. The core components of the slewing device include a slewing motor, a traveling motor, a slewing bearing, a slewing platform, gears, a lower base, a crawler, and an encoder. When the slewing device is operating, the slewing motor provides power, and the power is transmitted to the slewing platform through the slewing support and pinion, thereby driving the upper body to achieve the slewing function. When the traveling device is operating, the power provided by two traveling motors is transmitted to the crawler through a speed reducer to achieve the traveling and turning functions of the electric shovel test bench.

[0089] According to an embodiment of the present invention, step S2 includes: determining the focal lengths f of the TOF camera in the x and y directions through camera calibration x , f y , the optical centers c x , c y in the x and y directions, and converting the depth image pixel coordinates (u, v) into points (X, Y, Z) in 3D space through the following formula:

[0090]

[0091] Z = D(u, v)

[0092] where D(u, v) is the depth value corresponding to the pixel point (u, v) in the depth image;

[0093] There is a rotation and translation relationship between the TOF camera and the lidar coordinate systems. Through external parameter calibration, the transformation matrix T determined by the rotation matrix R and the translation vector t can be calculated:

[0094]

[0095] Using the calibrated transformation matrix T, the point cloud generated by the TOF camera is converted from the camera coordinate system to the lidar coordinate system:

[0096] P LiDAR = T · PTOF

[0097] Among them, P LiDAR is the coordinate system of the lidar, and P TOF is the coordinate system of the camera;

[0098] After completing the coordinate transformation, the point clouds generated by the TOF camera and the lidar are in the same coordinate system, so as to merge the TOF camera data and the lidar data for point cloud registration.

[0099] According to an embodiment of the present invention, step S3 includes: removing noise points in the point cloud data through a median filter;

[0100] Reducing the point cloud density through the voxel grid method to speed up subsequent processing while maintaining the main features; calculating the normal vector of the point cloud through the principal component analysis of neighboring points for feature extraction, where the normal vector of the point cloud is calculated by the following formula:

[0101]

[0102] Among them, N i is the neighborhood of point P i , and V i is the normal vector;

[0103] Using the random sample consensus algorithm to segment the point cloud of the material to be mined, the point cloud of the mining truck, and the point cloud of the obstacle from the global point cloud data, and performing surface fitting on the material heap surface and the ground.

[0104] According to an embodiment of the present invention, step S4 includes: identifying boundary points by calculating the normal vector and curvature of the point cloud to extract the boundary of the mining truck cargo box, where the point cloud curvature calculation formula is as follows:

[0105]

[0106] Among them, N(p) is the set of neighboring points of point p, n(p) is the normal vector of point p, and κ(p) is the curvature of point p;

[0107] Utilizing the aggregation characteristics of boundary points, using the Harris corner detection algorithm to detect the corners of the mining truck cargo box, and using the random sample consensus algorithm to fit the plane of the mining truck cargo box; positioning the geometric center and relative coordinates of the mining truck cargo box through the boundary point and surface features of the mining truck cargo box; obtaining the main axis direction of the mining truck cargo box through the results of the principal component analysis method, and calculating the minimum circumscribed rectangle of the mining truck cargo box, and the minimum circumscribed rectangle is the outer bounding box of the mining truck cargo box.

[0108] Specifically, based on the obvious planar and corner geometric features of the cargo box, boundary extraction can be performed first. By calculating the normal vector and curvature of the point cloud, boundary points are identified. Boundary points usually appear where the point cloud density changes significantly. Using the aggregation characteristics of boundary points, Harris corner detection is used to detect the corner points of the cargo box. The random sample consensus (RANSAC) algorithm is used to fit the plane of the mining truck cargo box. Through boundary points and surface features, the geometric center and relative coordinates of the cargo box can be accurately located. To facilitate subsequent operations (such as loading planning), an outer bounding box needs to be fitted to the mining truck cargo box. The fitting method of the minimum circumscribed rectangle is the most compact rectangular box that encloses the cargo box point cloud. The principal axis direction of the cargo box can be obtained through the results of the principal component analysis (PCA), and then the minimum circumscribed rectangle can be calculated. Through the fitted outer bounding box, the minimum distance error Error is used to verify the fitting accuracy:

[0109]

[0110] where N is the number of points in the point set, P i is the i-th point in the point set, P fit is P i the corresponding point on the fitting model

[0111] According to an embodiment of the present invention, in step S5, based on the point cloud of the material to be mined and the point cloud of the mining truck, the travel trajectory of the electric shovel is planned, including: obtaining the position coordinates of the electric shovel according to the point cloud data provided by the unmanned aerial vehicle, and determining the docking position of the mining truck; using the hybrid A* algorithm, selecting key path points according to the map provided by the unmanned aerial vehicle, and planning the globally optimal route; when the mining truck approaches the excavation area, entering the dynamic environment, and enabling the dynamic window algorithm to perform local path planning for the mining truck; using the dynamic scheduling algorithm to adjust the task priorities of multiple mining trucks in real time.

[0112] Specifically, based on the point cloud data provided by the tethered unmanned aerial vehicle, the position coordinates of the electric shovel are obtained, and the docking position of the mining truck is determined; using the hybrid A* algorithm, key path points are selected according to the map provided by the unmanned aerial vehicle, and the globally optimal route is planned. When the mining truck approaches the excavation area, it enters the dynamic environment, and the DWA (dynamic window algorithm) is enabled for local path planning. When the mining truck travels along the planned route to the excavation area, the DWA local algorithm will ensure that the mining truck is parked directly below the bucket of the electric shovel and control the orientation error within 3°, achieving precise docking. The system uses the dynamic scheduling algorithm to adjust the task priorities of the mining trucks in real time. When a certain mining truck is approaching full load, it is given priority for unloading, and the other mining trucks continue with the loading operation. By intelligently allocating resources and optimizing the task order, the system minimizes waiting time and conflicts and improves the overall operation efficiency.

[0113] According to an embodiment of the present invention, in step S5, the walking, digging, and loading trajectories of the electric shovel are planned based on the point cloud of the material to be mined and the point cloud of the ore blocking point, including: the walking trajectory planning of the electric shovel, the digging trajectory planning of the electric shovel, and the loading trajectory planning of the electric shovel. Among them, the walking trajectory planning of the electric shovel includes: obtaining the remaining amount of the material to be mined around the electric shovel. When the remaining amount of the material within the maximum operation radius of the electric shovel is less than the preset threshold, the hybrid A* algorithm is used to plan the moving path of the electric shovel and navigate it to the digging position with more material in the environment. When the electric shovel approaches the target position, the dynamic window algorithm is used for local path planning of the electric shovel to control the positioning error of the target point within 0.1 m to achieve precise positioning of the electric shovel. The digging trajectory planning of the electric shovel includes: based on the point cloud information of the material to be mined, combined with the self-attitude information fed back by the sensors of the electric shovel itself, with the goal of unit volume digging energy consumption, optimizing the digging trajectory, and using the particle swarm algorithm to select the best digging path in the trajectory planning. By fitting the polynomial curve of the digging angle and the stick length, the smooth adjustment of the trajectory is realized. Among them, compared with the traditional algorithm, the particle swarm algorithm has the advantages of fast convergence speed and strong global optimization ability, ensuring the efficient operation of the electric shovel under different terrain and load conditions. If the planned path does not meet the set boundary conditions, the trajectory will be automatically regenerated to ensure the continuity of the operation, where the set boundary conditions include: the maximum digging depth and the maximum lifting force. The loading trajectory planning of the electric shovel includes: combining the forward and inverse kinematic solutions of the electric shovel to obtain the commands corresponding to the actual lifting distance and the pushing distance, and controlling the lifting motor and the pushing motor of the electric shovel together to complete the digging task according to the commands. The attitude information transmitted back by the sensors of the electric shovel is used for closed-loop feedback of the execution quality of the digging action and timely correction to achieve high-precision digging. In the collaborative operation, the electric shovel dynamically adjusts the bucket angle and speed according to the real-time position of the ore truck to ensure the accurate docking of the bucket and the ore truck. The electric shovel also maintains real-time communication with the ore truck, and reasonably allocates tasks through the scheduling algorithm to avoid conflicts and resource waste among multiple devices and improve the overall operation efficiency.

[0114] According to an embodiment of the present invention, the use of the dynamic window algorithm for local path planning includes the following steps:

[0115] S51, Initialize the state parameters of the device. The state parameters include the position (x, y), orientation (θ), speed (v), angular velocity (ω) of the device, and set the maximum speed (vmax) and maximum acceleration (amax) limits, and set the time step Δt;

[0116] S52, Generate a feasible speed window according to the current state. The linear speed window is v ∈ [v min , v max and the angular velocity window is ω ∈ [ω min , ω max . For each time step, the speed range is:

[0117] V d = [v current - a max ·Δt, v current + a max ·Δt]

[0118] Ω d = [ω current - a max ·Δt, ω current + a max ·Δt]

[0119] S53. For each speed combination, based on the current speed and angular velocity, predict a trajectory within a future time range, and calculate the position change within the time step;

[0120] S54. Evaluate the predicted trajectory against the evaluation metrics, and select the trajectory with the lowest cost as the motion plan for the current device, where the evaluation metrics include at least one of the distance to the target, the distance to the obstacle, and the trajectory smoothness;

[0121] S55. Generate linear velocity and angular velocity control commands according to the selected optimal trajectory, and drive the motors of the device to operate;

[0122] S56. Repeat steps S52 to S55 in each control cycle, continuously update the state and trajectory planning of the device to adapt to environmental changes.

[0123] According to an embodiment of the present invention, in step S5, plan the movement trajectory of the unmanned aerial vehicle according to the point cloud of the material to be mined and the point cloud of the ore block, including: synchronously planning the movement trajectory of the unmanned aerial vehicle and the walking trajectory of the electric shovel, the unmanned aerial vehicle always stays above the electric shovel, and when the electric shovel moves, the unmanned aerial vehicle moves synchronously to provide a stable environmental perception service for the electric shovel.

[0124] Furthermore, as Figure 5 shown, the unmanned aerial vehicle 1 is provided with a warning device. When the electric shovel 2 or the ore truck 3 moves to the boundary of the positioning area, the warning device will send a warning signal to the electric shovel 2 or the ore truck 3, driving the electric shovel 2 or the ore truck 3 to move into the positioning area to prevent the loss of positioning of the electric shovel 2 or the ore truck 3. The unmanned aerial vehicle 1 is provided with a communication relay device for communicating with the electric shovel 2 or the ore truck 3.

[0125] Corresponding to the above embodiment, the present invention also proposes a mining operation system for a large-scale scene of an electric shovel in a mine.

[0126] Figure 6 It is a block diagram of a mining operation system for a large-scale scene of an electric shovel in a mine according to an embodiment of the present invention.

[0127] As Figure 6As shown in the figure, the mining operation system 100 for large-scale scenarios of a mining electric shovel according to an embodiment of the present invention includes an unmanned aerial vehicle (UAV), an electric shovel, and multiple mining trucks in a large-scale scenario of a mining electric shovel. The UAV is communicatively connected to the electric shovel and the multiple mining trucks respectively. The system 100 may include: an environmental perception module 110, a point cloud registration module 120, a material and mining truck point cloud segmentation module 130, a mining truck cargo box positioning and outer bounding box fitting module 140, and a trajectory planning module 150.

[0128] Among them, the environmental perception module 110 is configured to control the UAV to rise to a preset height, and scan and photograph the mine environment through a lidar and a TOF camera mounted on the UAV to collect global point cloud data of the mine environment. The global point cloud data includes the attitude information and position information of the electric shovel, the attitude information and position information of the mining truck, and the three-dimensional information of the material to be mined. The point cloud registration module 120 is configured to perform point cloud registration on the TOF camera data and the lidar data. The material and mining truck point cloud segmentation module 130 is configured to separate the point cloud of the material to be mined and the point cloud of the mining truck from the global point cloud data, and position the cargo box of the mining truck and perform surface fitting on the point cloud of the material to be mined. The mining truck cargo box positioning and outer bounding box fitting module 140 is configured to determine the position coordinates of the electric shovel bucket for unloading according to the position of the cargo box of the mining truck, and fit the point cloud of the cargo box of the mining truck to fit the outer bounding box of the cargo box of the mining truck. The trajectory planning module 150 is configured to plan the travel trajectory of the mining truck, the excavation and loading trajectory of the electric shovel, and the movement trajectory of the UAV respectively according to the point cloud of the material to be mined and the point cloud of the mining truck, so that the electric shovel, the multiple mining trucks and the UAV cooperate to perform the mining operation.

[0129] The mining loading operation system for large-scale scenarios in the embodiments of the present invention mainly includes a tethered drone, an electric shovel, and a mining truck. The tethered drone specifically includes four parts: a TOF camera, a lidar, a communication relay, and an edge detection device. The TOF camera can provide fast and real-time depth information by measuring the flight time of light pulses and capture high-resolution depth images. The lidar can provide high-precision distance measurements at a farther distance and has higher depth measurement accuracy under various environmental conditions, especially suitable for large-scale scanning. The combination of the high-resolution images provided by the TOF camera and the remote measurements of the lidar can form a more detailed and accurate three-dimensional map. The depth map generated by the TOF camera is converted into point cloud data and merged with the point cloud obtained by the lidar. The blank areas of the lidar data are filled by interpolation methods to form a more complete environmental model. The tethered drone generates a three-dimensional map of the mining area through the lidar and the TOF camera, transmits the point cloud data to the mining truck and the electric shovel, and performs preprocessing locally to extract terrain changes, obstacle positions, and the real-time status information of the electric shovel. The drone also acts as a communication relay to enhance signal coverage and reduce interference, ensuring stable and low-latency data transmission between the mining truck, the electric shovel, and the control center. Through the relay device of the drone, the signal can avoid interference or attenuation during direct transmission, reduce data packet loss and communication latency, and ensure the accurate transmission of real-time information.

[0130] As the core execution equipment in mining operations, the electric shovel not only receives the three-dimensional map information transmitted by the drone but also real-time senses its own working status (such as the position of the dipper stick and the load condition) and the distribution changes of the material heap surface. The electric shovel calculates the total work of the hoisting wire rope and the dipper stick push pressure through a dynamic model and optimizes the excavation trajectory with the goal of minimizing the energy consumption per unit volume of the material. The system uses a particle swarm algorithm to select the best excavation path in trajectory planning and realizes the smooth adjustment of the trajectory by fitting the polynomial curve of the excavation angle and the dipper stick length. Compared with traditional algorithms, the particle swarm algorithm has the advantages of fast convergence speed and strong global optimization ability, ensuring the efficient operation of the electric shovel under different terrain and load conditions. If the planned path does not meet the set boundary conditions (such as the maximum excavation depth and the lifting force), the system will automatically regenerate the trajectory to ensure the continuity of the operation. In collaborative operations, the electric shovel dynamically adjusts the dipper angle and speed according to the real-time position of the mining truck to ensure the accurate docking of the dipper with the mining truck. The electric shovel also maintains real-time communication with the mining truck, reasonably allocates tasks through a scheduling algorithm, avoids conflicts and resource waste among multiple devices, and improves the overall operation efficiency.

[0131] Through the wireless communication network, real-time information exchange is maintained among the unmanned aerial vehicle (UAV), the electric shovel, and the mining truck. The mining truck receives environmental perception data from the UAV (such as 3D maps, obstacle positions) and the operation status information of the electric shovel (such as excavation progress, position). The mining truck uses multi-sensor data fusion technology to combine the environmental perception information provided by the UAV with its own sensor data. The mining truck combines the map data of the UAV with its own motion state to update the position and obstacle information in real time, improving the accuracy of path planning. The mining truck first adopts the hybrid A* algorithm to select key path points according to the map provided by the UAV and plan the globally optimal route. When the mining truck approaches the excavation area, it enters a dynamic environment and enables the DWA (Dynamic Window Algorithm) for local path planning. When the mining truck travels along the planned route to the excavation area, the DWA local algorithm ensures that the mining truck parks directly in front of the bucket of the electric shovel and controls the orientation error within 3°, achieving precise parking. The system adopts a dynamic scheduling algorithm to adjust the task priorities of the mining trucks in real time. When a certain mining truck is approaching full load, it is preferentially arranged for unloading, while other mining trucks continue with the loading operation. By intelligently allocating resources and optimizing the task sequence, the system minimizes waiting time and conflicts and improves the overall operation efficiency.

[0132] It should be noted that for the details not disclosed in the mining electric shovel large-scenario loading and unloading operation system of the embodiments of the present invention, please refer to the details disclosed in the mining electric shovel large-scenario loading and unloading operation method of the embodiments of the present invention, and specific details will not be elaborated here.

[0133] In the description of this specification, the descriptions referring to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples", etc. mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples.

[0134] In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one of such features. In the description of the present invention, the meaning of "a plurality" is at least two, such as two, three, etc., unless otherwise specifically defined.

[0135] In the present invention, unless otherwise clearly specified or limited, the terms "installed", "connected", "coupled", "fixed", etc. shall be construed broadly. For example, it may be a fixed connection, a detachable connection, or an integral body; it may be a mechanical connection or an electrical connection; it may be directly connected or indirectly connected through an intermediate medium, and it may be the internal communication of two components or the interaction relationship between two components, unless otherwise clearly limited. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0136] Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

Claims

1. A mining and loading method for a large scene with a mining electric shovel, characterized in that: The large mining electric shovel scene includes a drone, an electric shovel, and multiple mining trucks. The drone is respectively connected to the electric shovel and the multiple mining trucks for communication. The method includes: S1, controlling the drone to rise to a preset height, and scanning and photographing the mine environment through the laser radar and TOF camera carried by the drone to collect global point cloud data of the mine environment, wherein the global point cloud data includes the posture information and position information of the electric shovel, the posture information and position information of multiple mining trucks, and the three-dimensional information of the materials to be excavated; S2, perform point cloud registration on TOF camera data and lidar data; S3, separating the point cloud of the material to be excavated and the point cloud of the mining truck from the global point cloud data, positioning the cargo box of the mining truck and performing surface fitting on the point cloud of the material to be excavated; S4, determining the position coordinates of the electric shovel bucket unloading according to the position of the cargo box of the mining truck, and fitting the point cloud of the cargo box of the mining truck to fit the outer bounding box of the mining truck cargo box; S5, planning the walking trajectory of the mining truck, the walking, excavating and loading trajectory of the electric shovel and the moving trajectory of the drone respectively according to the point cloud of the material to be excavated and the point cloud of the mining truck, so that the electric shovel and the multiple mining trucks and the drone can cooperate in mining and loading operations.

2. The mining and loading method of a large-scale mining electric shovel according to claim 1 is characterized in that: In step S1, the UAV is a tethered UAV, the laser radar is a multi-line laser radar, the multi-line laser radar scans the surrounding environment of the electric shovel to obtain point cloud data of the surrounding environment of the electric shovel, and the TOF camera obtains a depth map of the mine environment.

3. The mining and loading method of a large-scale mining electric shovel according to claim 2 is characterized in that: Step S2 includes: Determine the focal length f of the TOF camera in the x and y directions by camera calibration x 、f y , the optical center c in the x and y directions x 、c y , the depth map pixel coordinates (u, v) are converted to points (X, Y, Z) in 3D space using the following formula: Z=D(u,v) Where D(u,v) is the depth value corresponding to the pixel point (u,v) in the depth image; Through external parameter calibration, the transformation matrix T determined by the rotation matrix R and the translation vector t is calculated: Use the calibrated transformation matrix T to transform the point cloud generated by the TOF camera from the camera coordinate system to the lidar coordinate system: P L i DAR =T·P TOF P TOF Among them, P L i DAR is the laser radar coordinate system, is the camera coordinate system; After completing the coordinate conversion, the point cloud generated by the TOF camera and the point cloud generated by the lidar are in the same coordinate system, so that the TOF camera data and the lidar data can be merged for point cloud registration.

4. The mining and loading method of a large-scale mining electric shovel according to claim 1 is characterized in that: Step S3 includes: Use a median filter to remove noise points in the point cloud data; Reduce the density of point clouds by using voxel grid method while keeping the main features and speeding up the subsequent processing; The normal vector of the point cloud is calculated by principal component analysis of the neighborhood points to perform feature extraction, where the normal vector of the point cloud is calculated by the following formula: Among them, Ni is the neighborhood of point Pi, Vi is the normal vector, and p is the point in the neighborhood of Pi; A random sampling consistency algorithm is used to segment the point cloud of the material to be excavated, the point cloud of the mining truck and the point cloud of the obstacle from the global point cloud data, and surface fitting is performed on the material pile surface and the ground.

5. The mining and loading method of a large-scale mining electric shovel according to claim 1 is characterized in that: Step S4 includes: The boundary points are identified by calculating the normal vector and curvature of the point cloud to extract the boundary of the mining truck cargo box, wherein the point cloud curvature calculation formula is as follows: Among them, N(p) is the neighborhood point set of point p, n(p) is the normal vector of point p, κ(p) is the curvature of point p, and n(q) is the normal vector of point q in the neighborhood; The Harris corner detection algorithm is used to detect the corner points of the cargo box of the mining truck by utilizing the clustering characteristics of the boundary points, and the random sampling consistency algorithm is used to fit the plane of the cargo box of the mining truck; Locating the geometric center and relative coordinates of the mining truck cargo box through the boundary points and surface features of the mining truck cargo box; The principal axis direction of the mining truck cargo box is obtained through the result of the principal component analysis method, and the minimum circumscribed rectangle of the mining truck cargo box is calculated, and the minimum circumscribed rectangle is the outer bounding box of the mining truck cargo box.

6. The mining and loading method of a large-scale mining electric shovel according to claim 1 is characterized in that: In step S5, the walking trajectory of the electric shovel is planned according to the point cloud of the material to be excavated and the point cloud of the mining truck, including: According to the point cloud data provided by the drone, the coordinates of the location of the electric shovel are obtained to determine the parking position of the mining truck; A hybrid A* algorithm is used to select key path points based on the map provided by the drone and plan the global optimal route; When the mining truck approaches the mining area, it enters the dynamic environment and enables the dynamic window algorithm to perform local path planning for the mining truck; A dynamic scheduling algorithm is used to adjust the task priorities of the multiple mining cards in real time.

7. The mining and loading method of a large-scale mining electric shovel according to claim 1 is characterized in that: In step S5, the walking, excavating and loading trajectory of the electric shovel is planned according to the point cloud of the material to be excavated and the point cloud of the mining truck, including: walking trajectory planning of the electric shovel, excavating trajectory planning of the electric shovel and loading trajectory planning of the electric shovel, wherein: The walking trajectory planning of the electric shovel includes: obtaining the remaining amount of materials to be excavated around the electric shovel, and when the remaining amount of materials within the maximum operating radius of the electric shovel is less than a preset threshold, using a hybrid A* algorithm to plan the moving path of the electric shovel and navigate to the excavation position with more materials in the environment; when the electric shovel approaches the target position, using a dynamic window algorithm to perform local path planning on the electric shovel to achieve accurate positioning of the electric shovel; The excavation trajectory planning of the electric shovel includes: based on the point cloud information of the material to be excavated, combined with the posture information of the electric shovel fed back by the sensor data of the electric shovel itself, taking the excavation energy consumption per unit volume as the target, optimizing the excavation trajectory, and adopting the particle swarm algorithm to select the best excavation path in the trajectory planning, and realizing the smooth adjustment of the trajectory by fitting the polynomial curve of the excavation angle and the length of the bucket arm; if the planned path does not meet the set boundary conditions, the trajectory will be automatically regenerated to ensure the continuity of the operation, wherein the set boundary conditions include: maximum excavation depth and maximum lifting force; The loading trajectory planning of the electric shovel includes: combining the forward and inverse kinematics solutions of the electric shovel to obtain instructions corresponding to the actual lifting distance and the pushing distance, and controlling the lifting motor and the pushing motor of the electric shovel according to the instructions to jointly complete the excavation task. The posture information sent back by the sensor of the electric shovel is used for closed-loop feedback of the execution quality of the excavation action and timely correction to achieve high-precision excavation.

8. The mining and loading method for a large scene with a mining electric shovel according to claim 6 or 7, characterized in that: Local path planning using the dynamic window algorithm includes: S51, initializing the state parameters of the device, the state parameters including the position (x, y), orientation (θ), speed (v), angular velocity (ω) of the device, setting the maximum speed (vmax) and maximum acceleration (amax) limits, and setting the time step Δt; S52, generate a feasible speed window according to the current state, the linear speed window v∈[v min ,v max ] and the angular velocity window ω∈[ω min ,ω max ], for each time step Δt, the feasible range of linear velocity Vd and the feasible range of angular velocity Ωd are: V d =[v current -a max ·Δt,v current +a max ·Δt] Oh d =[ω current -a max ·Δt,ω current +a max ·Δt] Among them, v current is the current linear velocity, ω current is the current angular velocity. S53, for each speed combination, based on the current speed and angular velocity, predict a trajectory within a future time range and calculate the position change within the time step; S54, evaluating the predicted trajectory with an evaluation index, and selecting a trajectory with the lowest cost as a motion plan for the current device, wherein the evaluation index includes at least one of a distance to a target, a distance to an obstacle, and trajectory smoothness; S55, generating linear velocity and angular velocity control commands according to the selected optimal trajectory to drive the motor of the device to operate; S56, repeatedly executing steps S52 to S55 in each control cycle, continuously updating the device status and trajectory planning to adapt to environmental changes.

9. The mining and loading method of a large-scale mining shovel according to claim 7 is characterized in that: In step S5, the movement trajectory of the drone is planned according to the point cloud of the material to be excavated and the point cloud of the mining truck, including: The movement trajectory planning of the UAV is carried out synchronously with the walking trajectory planning of the electric shovel. The UAV always stays above the electric shovel. When the electric shovel moves, the UAV moves synchronously to provide stable environmental perception services for the electric shovel.

10. A mining and loading system for a large scene with a mining electric shovel, characterized in that: The large mining electric shovel scene includes a drone, an electric shovel, and multiple mining trucks. The drone is respectively connected to the electric shovel and the multiple mining trucks for communication. The system includes: An environmental perception module is used to control the drone to rise to a preset height, and scan and photograph the mine environment through the laser radar and TOF camera carried by the drone to collect global point cloud data of the mine environment, wherein the global point cloud data includes the posture information and position information of the electric shovel, the posture information and position information of the mining truck, and the three-dimensional information of the material to be excavated; Point cloud registration module, used to perform point cloud registration of TOF camera data and lidar data; The material and mining truck point cloud segmentation module is used to separate the point cloud of the material to be excavated and the point cloud of the mining truck from the global point cloud data, locate the cargo box of the mining truck, and perform surface fitting on the point cloud of the material to be excavated; The mining truck cargo box positioning and outer bounding box fitting module is used to determine the position coordinates of the electric shovel bucket unloading according to the cargo box position of the mining truck, and to fit the cargo box point cloud of the mining truck to fit the outer bounding box of the mining truck cargo box; The trajectory planning module is used to plan the walking trajectory of the mining truck, the excavation and loading trajectory of the electric shovel, and the moving trajectory of the drone according to the point cloud of the material to be excavated and the point cloud of the mining truck, so that the electric shovel and multiple mining trucks and the drone can cooperate in mining and loading operations.

Images