A mining shovel large scene mining and loading operation method and system

Abstract

The application discloses a mining electric shovel large-scene mining and loading operation method and system, the method comprises the following steps: S1, controlling the unmanned aerial vehicle to ascend to a preset height, and scanning and shooting the mine field environment through the laser radar and TOF camera carried on the unmanned aerial vehicle, so as to collect the global point cloud data of the mine field environment; S2, performing point cloud registration on the TOF camera data and the laser radar data; S3, separating the to-be-excavated material point cloud and the mine card point cloud from the global point cloud data, positioning the box of the mine card, and performing surface fitting on the to-be-excavated material point cloud; S4, determining the position coordinates of the electric shovel bucket unloading according to the box position of the mine card, fitting the box point cloud of the mine card, and fitting the outer enclosing frame of the box of the mine card; and S5, respectively planning the walking track of the mine card, the walking, excavating and loading track of the electric shovel and the moving track of the unmanned aerial vehicle according to the to-be-excavated material point cloud and the mine card point cloud, so that the electric shovel and the multiple mine cards and the unmanned aerial vehicle cooperatively perform the mining and loading operation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent operation of open-pit mines, and in particular to a mining shovel large-scene mining and loading operation method and system. BACKGROUND

[0002] In mine exploitation, the development of environmental perception and autonomous operation technology has gradually attracted widespread attention. Traditional mine excavation and loading operations rely on a large number of manual operations, which is low in efficiency and easily affected by complex mine environments, posing a serious threat to the safety of operators. Therefore, mine operations are gradually changing towards intelligence and automation, especially in the field of autonomous operation combining unmanned aerial vehicles and robots, which has made remarkable progress. However, the current mainstream open-pit mine environmental perception technology mainly relies on ground fixed sensors and laser radars and other equipment. Although these devices can provide environmental information within a certain range, they have limited coverage, many blind areas, and are difficult to quickly adapt to dynamic changes, resulting in insufficient timeliness and accuracy of perception data.

[0003] In order to solve these problems, unmanned aerial vehicle technology is introduced into open-pit mine environment detection, significantly improving the transmission efficiency and flexibility of operation information. Tethered unmanned aerial vehicles can achieve greater range of scene perception and information transmission, effectively compensating for the blind area of ground sensors, and are more suitable for the intelligent operation needs of open-pit mine equipment. However, existing unmanned aerial vehicle technology still has certain limitations in terms of perception accuracy, real-time performance and intelligent decision-making, making it difficult to meet the operation requirements in all-weather and large-scene complex environments. In order to further improve the intelligent level of mine operations, the collaborative operation between unmanned aerial vehicles and mining shovels has become one of the key technologies to be solved.

[0004] In summary, although unmanned aerial vehicles gradually exhibit their unique advantages in mine operations, the effective combination of their perception capabilities and the operation capabilities of mining shovels still faces many challenges, and further research and development of efficient and intelligent collaborative systems are needed to improve the overall efficiency and safety of mine operations. SUMMARY

[0005] The present application aims to at least partially solve one of the technical problems in the related art. To this end, the first object of the present application is to propose a mining shovel large-scene mining and loading operation method, which can be applied to three-dimensional perception of the shovel operation environment in the large scene of an open-pit mine, dynamic planning of the shovel excavation trajectory and the mine truck walking trajectory, and multi-machine collaborative operation of the shovel, mine truck and unmanned aerial vehicle, achieving the goal of high efficiency, energy saving and safety in the open-pit mining process in complex environments.

[0006] The second object of the present application is to propose a mining shovel large-scene mining and loading operation system.

[0007] To achieve the above object, the first aspect of the present application provides a mining shovel large scene mining and loading operation method, the mining shovel large scene includes a UAV, a shovel and a plurality of mine trucks, the UAV is respectively connected with the shovel and a plurality of mine trucks, and the method comprises:

[0008] S1, control the UAV to rise to a preset height, and scan and shoot the mine site environment through the laser radar and TOF camera carried on the UAV, so as to collect the global point cloud data of the mine site environment, the global point cloud data includes the attitude information and position information of the shovel, the attitude information and position information of a plurality of mine trucks, and the three-dimensional information of the material to be excavated;

[0009] S2, the TOF camera data and the laser radar data are point cloud registered;

[0010] S3, the material to be excavated point cloud and the mine truck point cloud are separated from the global point cloud data, and the box of the mine truck is positioned and the surface of the material to be excavated point cloud is fitted;

[0011] S4, the position coordinates of the shovel bucket unloading are determined according to the box position of the mine truck, and the box point cloud of the mine truck is fitted to fit the outer enclosing frame of the mine truck box;

[0012] S5, the walking track of the mine truck, the walking and excavating loading track of the shovel and the moving track of the UAV are planned according to the material to be excavated point cloud and the mine truck point cloud respectively, so that the shovel and a plurality of mine trucks and the UAV cooperatively perform mining and loading operation.

[0013] In addition, the mining shovel large scene mining and loading operation method according to the above embodiment of the present application can also have the following additional technical features:

[0014] According to one embodiment of the present application, in step S1, the UAV is a tethered UAV, and the laser radar is a multi-line laser radar, which scans the surrounding environment of the shovel to obtain the point cloud data of the surrounding environment of the shovel, and the TOF camera obtains the depth map of the mine site environment.

[0015] According to one embodiment of the present application, step S2 comprises:

[0016] The focal length f of the TOF camera in x and y directions is determined by camera calibration x 、f y The optical centers c of the TOF camera in x and y directions are determined by camera calibration x 、c y The depth image pixel coordinates (u, v) are converted into points (X, Y, Z) in 3D space by the following formula:

[0017]

[0018] Z=D(u,v)

[0019] wherein D(u,v) is a depth value corresponding to a pixel point (u,v) in the depth image;

[0020] Through the extrinsic calibration, a transformation matrix T determined by a rotation matrix R and a translation vector t is calculated:

[0021]

[0022] The point cloud generated by the TOF camera is converted from the camera coordinate system to the laser radar coordinate system using the transformation matrix T obtained through the calibration:

[0023] P L i DAR =T·P TOF

[0024] wherein P LiDAR is the laser radar coordinate system, and P TOF is the camera coordinate system;

[0025] After the coordinate conversion is completed, the point cloud generated by the TOF camera and the point cloud generated by the laser radar are both in the same coordinate system, so as to merge the TOF camera data and the laser radar data for point cloud registration.

[0026] According to one embodiment of the present application, step S3 comprises:

[0027] The noise points in the point cloud data are removed through a median filter;

[0028] The point cloud density is reduced through a voxel grid method, so as to keep the main features and speed up the subsequent processing speed;

[0029] The normal vector of the point cloud is calculated through principal component analysis of the neighborhood points, so as to perform feature extraction, wherein the normal vector of the point cloud is calculated through the following formula:

[0030]

[0031] wherein Ni is the neighborhood of the point Pi, and Vi is the normal vector;

[0032] The random sample consensus algorithm is used to segment the material point cloud to be excavated, the mine truck point cloud and the obstacle point cloud from the global point cloud data, and the material pile surface and the ground are subjected to surface fitting.

[0033] According to one embodiment of the present application, step S4 comprises:

[0034] Boundary points are identified by calculating normal vectors and curvatures of point clouds to perform boundary extraction on the mine truck box, wherein a point cloud curvature calculation formula is as follows:

[0035]

[0036] wherein N(p) is a domain point set of point p, n(p) is a normal vector of point p, and kappa(p) is a curvature of point p;

[0037] By using the aggregation feature of the boundary points, a Harris corner point detection algorithm is used to detect the corner points of the mine truck box, and a random sample consensus algorithm is used to fit the plane of the mine truck box.

[0038] By using the boundary point and surface features of the mine truck box, the geometric center and relative coordinates of the mine truck box are located.

[0039] The principal axis direction of the mine truck box is obtained through the result of the principal component analysis method, and the minimum circumscribed rectangle of the mine truck box is calculated, which is the outer bounding box of the mine truck box.

[0040] According to one embodiment of the present application, in step S5, the walking trajectory of the electric shovel is planned according to the point cloud of the material to be excavated and the point cloud of the mine truck, including:

[0041] According to the point cloud data provided by the unmanned aerial vehicle, the position coordinates of the electric shovel are obtained, and the parking position of the mine truck is determined.

[0042] A hybrid A* algorithm is used to select key path points according to the map provided by the unmanned aerial vehicle, and a globally optimal route is planned.

[0043] When the mine truck approaches the excavation area, the dynamic environment is entered, and a dynamic window algorithm is enabled for local path planning of the mine truck.

[0044] A dynamic scheduling algorithm is used to adjust the task priority of multiple mine trucks in real time.

[0045] According to one embodiment of the present application, in step S5, the walking excavation loading trajectory of the electric shovel is planned according to the point cloud of the material to be excavated and the point cloud of the mine truck, including the walking trajectory planning of the electric shovel, the excavation trajectory planning of the electric shovel and the loading trajectory planning of the electric shovel, wherein,

[0046] The walking trajectory planning of the electric shovel comprises: acquiring a remaining amount of material to be excavated around the electric shovel, when the remaining amount of material within a maximum working radius of the electric shovel is less than a preset threshold, a hybrid A* algorithm is used to plan a moving path of the electric shovel and navigate to a digging position with more material in the environment, when the electric shovel approaches the target position, a dynamic window algorithm is used to plan a local path of the electric shovel to realize accurate positioning of the electric shovel;

[0047] The digging trajectory planning of the electric shovel comprises: based on point cloud information of material to be excavated, combined with self-attitude information fed back by self-sensor data of the electric shovel, taking unit volume digging energy consumption as a target, optimizing a digging trajectory, and using a particle swarm algorithm to select a best digging path in trajectory planning, through fitting a polynomial curve of a digging angle and a boom length, realizing smooth adjustment of the trajectory, if the planned path does not satisfy a set boundary condition, the trajectory is automatically regenerated to ensure operation continuity, wherein the set boundary condition comprises a maximum digging depth and a maximum lifting force;

[0048] The loading trajectory planning of the electric shovel comprises: combined with forward and inverse kinematics solving of the electric shovel, obtaining instructions corresponding to actual lifting distance and pushing distance, controlling a lifting motor and a pushing motor of the electric shovel to jointly complete a digging task according to the instructions, and using attitude information fed back by a sensor of the electric shovel to close-loop feedback digging action execution quality and timely correct deviation, realizing high-precision digging.

[0049] According to one embodiment of the present application, using a dynamic window algorithm for local path planning comprises:

[0050] S51, initializing state parameters of the device, the state parameters comprising a position (x, y), an orientation (θ), a speed (v), an angular speed (ω) of the device, and setting maximum speed (vmax) and maximum acceleration (amax) limits, and setting a time step Δt;

[0051] S52, generating a feasible speed window according to a current state, a linear speed window v∈[v min ,v max ] and an angular speed window ω∈[ω min ,ω max ], for each time step, the speed range is:

[0052] V d =[v current -a max ·Δt,v current +a max ·Δt]

[0053] Ω d =[ω current -a max ·Δt,ωcurrent +a max ·Δt]

[0054] S53, for each speed combination, predicting a trajectory in a future time range based on the current speed and angular velocity, calculating the position change in the time step;

[0055] S54, evaluating the predicted trajectory with the evaluation index, selecting the lowest cost trajectory as the motion plan of the current device, wherein the evaluation index includes at least one of the distance of the target, the distance from the obstacle and the trajectory smoothness;

[0056] S55, generating linear speed and angular velocity control commands according to the selected optimal trajectory to drive the motor of the device to operate;

[0057] S56, repeating steps S52 to S55 in each control cycle to continuously update the state and trajectory planning of the device to adapt to changes in the environment.

[0058] According to one embodiment of the application, in step S5, the moving trajectory of the unmanned aerial vehicle is planned according to the point cloud of the material to be excavated and the point cloud of the mine truck, comprising:

[0059] The unmanned aerial vehicle moving trajectory planning is synchronized with the electric shovel walking trajectory planning, and the unmanned aerial vehicle always stays above the electric shovel. When the electric shovel moves, the unmanned aerial vehicle moves synchronously to provide stable environment perception service for the electric shovel.

[0060] To achieve the above purpose, the second aspect of the present application proposes a mining shovel large scene mining and loading operation system, which comprises an unmanned aerial vehicle, an electric shovel and a plurality of mine trucks, the unmanned aerial vehicle is in communication connection with the electric shovel and the plurality of mine trucks, and the system comprises:

[0061] An environment perception module is used to control the unmanned aerial vehicle to rise to a preset height, and scan and shoot the mine environment through the laser radar and TOF camera carried on the unmanned aerial vehicle, so as to collect global point cloud data of the mine environment, which includes attitude information and position information of the electric shovel, attitude information and position information of the mine truck, and three-dimensional information of the material to be excavated.

[0062] A point cloud registration module is used to register the TOF camera data and the laser radar data;

[0063] A material and mine truck point cloud segmentation module is used to separate the material to be excavated point cloud and the mine truck point cloud from the global point cloud data, and to locate the cargo box of the mine truck and to surface fit the material to be excavated point cloud;

[0064] The truck box positioning and outer surrounding frame fitting module is configured to determine the position coordinates of the shovel bucket unloading according to the truck box position of the mine truck, and fit the truck box point cloud of the mine truck to fit the outer surrounding frame of the mine truck box.

[0065] The trajectory planning module is configured to plan the walking trajectory of the mine truck, the digging and loading trajectory of the electric shovel and the moving trajectory of the unmanned aerial vehicle according to the to-be-excavated material point cloud and the mine truck point cloud, respectively, so that the electric shovel and the plurality of mine trucks and the unmanned aerial vehicle cooperatively perform the mining and loading operation.

[0066] Compared with the prior art, the present application is based on the tethered unmanned aerial vehicle environment perception system, and is guaranteed by the multi-machine cooperation technology. The environment perception capability of the unmanned aerial vehicle, the autonomous operation capability of the electric shovel and the adaptive scheduling capability of the mine truck are effectively combined to form an efficient and intelligent cooperative system, and the efficient, energy-saving and safe goal in the open-pit mining process in a complex environment is achieved.

[0067] Additional aspects and advantages of the present application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0068] Figure 1 The flowchart of the mining electric shovel large-scene mining and loading operation method according to the embodiment of the present application;

[0069] Figure 2 The schematic diagram of the mining electric shovel large scene according to one embodiment of the present application;

[0070] Figure 3 The structural schematic diagram of the unmanned aerial vehicle according to one embodiment of the present application;

[0071] Figure 4 The structural schematic diagram of the electric shovel according to one embodiment of the present application;

[0072] Figure 5 The schematic diagram of the positioning of the electric shovel and the mine truck according to one embodiment of the present application;

[0073] Figure 6 The block schematic diagram of the mining electric shovel large-scene mining and loading operation system according to the embodiment of the present application. DETAILED DESCRIPTION

[0074] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the drawings, wherein the same or similar reference signs represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the drawings are exemplary and are intended to explain the present application, and cannot be understood as a limitation of the present application.

[0075] The mining shovel large scene mining and loading operation method and a mining shovel large scene mining and loading operation system are described below with reference to the accompanying drawings.

[0076] In the embodiments of the present application, as shown in Figure 2 The mining shovel large scene includes a drone 1, a shovel 2 and a plurality of trucks 3, the drone 1 is respectively connected with the shovel 2 and the plurality of trucks 3 in communication, and the shovel 2 and the plurality of trucks 3 perform cooperative mining and loading operation on the material 4. The drone 1 provides real-time environmental perception and data transmission services; the shovel 2 performs autonomous path planning and material excavation operations; and the plurality of trucks 3 are responsible for material transportation and cooperative operation with the shovel 2.

[0077] As shown in Figure 1 The mining shovel large scene mining and loading operation method of the embodiments of the present application can include the following steps:

[0078] S1, control the drone to rise to a preset height, and scan and shoot the mine site environment through the laser radar and TOF camera mounted on the drone to collect global point cloud data of the mine site environment, which includes attitude information and position information of the shovel, attitude information and position information of the plurality of trucks, and three-dimensional information of the material to be excavated. The preset height can be calibrated according to the range of the mining area, the specific performance of the drone, etc.

[0079] According to one embodiment of the present application, in step S1, the drone is a tethered drone, and the laser radar is a multi-line laser radar. The multi-line laser radar scans the environment around the shovel to obtain point cloud data of the environment around the shovel, and the TOF camera obtains a depth map of the mine site environment.

[0080] Specifically, as shown in Figure 3 The drone is configured with a laser radar 1-1, a TOF camera 1-2 and an edge computing device 1-3, which can obtain real-time three-dimensional environmental data of the mining area and transmit the data to the shovel and the truck through a wireless communication module 1-4. The laser radar 1-1 is used to generate a high-precision terrain model, the TOF camera 1-2 is responsible for providing depth information of the material, and the edge computing device 1-3 processes data locally to reduce communication delay. The drone communicates with the shovel 2 and the truck 3 through a wireless network to realize sharing of environmental information.

[0081] Specifically, a tethered drone mounted with a 360° laser radar and a TOF camera is controlled to rise to a preset height, the mine site is scanned and shot, the attitude and position of the shovel and the truck are obtained, and the three-dimensional information of the material to be excavated is obtained. The collected global point cloud data is transmitted to the industrial computer of the shovel through a wireless network.

[0082] S2, point cloud registration is performed on the TOF camera data and the laser radar data.

[0083] S3, separating the to-be-excavated material point cloud and the mine card point cloud from the global point cloud data, positioning the cargo box of the mine card, and performing surface fitting on the to-be-excavated material point cloud.

[0084] S4, determining the position coordinates of the unloading of the shovel bucket according to the cargo box position of the mine card, and fitting the cargo box point cloud of the mine card to fit the outer bounding box of the cargo box of the mine card.

[0085] S5, planning the walking track of the mine card, the walking, excavating and loading track of the electric shovel, and the moving track of the unmanned aerial vehicle according to the to-be-excavated material point cloud and the mine card point cloud respectively, so that the electric shovel and the multiple mine cards and the unmanned aerial vehicle cooperatively perform the mining and loading operation.

[0086] Specifically, after receiving the global point cloud data transmitted by the unmanned aerial vehicle, the industrial computer on the electric shovel pre-processes the global point cloud data, first fuses the TOF camera data and the laser radar data for point cloud registration, then separates the to-be-excavated material and the mine card and the like from the global point cloud, and further positions the cargo box of the mine card and performs surface fitting on the material point cloud, then positions the cargo box of the mine card to provide the position coordinates of the shovel bucket for unloading of the electric shovel, and fits the outer bounding box to provide the basis for obstacle avoidance of the electric shovel.

[0087] Further, the industrial computer on the electric shovel determines the stopping position of the mine card beside the electric shovel according to the point cloud information provided by the laser radar, and determines the walking track of the mine card; the electric shovel plans the excavating track according to the fitted heap surface point cloud; determines the unloading track according to the starting point of the excavating track and the position of the mine card; when there is no excavatable material in the working range of the electric shovel, the electric shovel is planned to walk to the remaining material, and the excavating and unloading track planning is continued to be executed; the tethered unmanned aerial vehicle is always stayed above the electric shovel, and when the electric shovel moves, the tethered unmanned aerial vehicle moves synchronously. The electric shovel is installed with a push gear shaft torque sensor, a steel wire rope tension sensor, a dipper boom inclination sensor, a steel wire rope drum shaft encoder and a push gear shaft encoder, a dipper boom pull wire displacement sensor and the like, for real-time monitoring of the pose and load information of the electric shovel.

[0088] It should be noted that, as Figure 4As shown, the electric shovel can be divided into a mechanical part and a control part in composition, the mechanical part is composed of a working device, a rotating device and a walking device. The working device of the electric shovel is the main device for undertaking the excavation task, which is composed of a pushing mechanism and a lifting mechanism. The pushing mechanism can be divided into a bucket rod, a bucket, a pushing gear, a pushing rack, a transmission belt, an encoder 2-1, an inclination sensor 2-2, a wire displacement sensor 2-3, a 0.75kw three-phase asynchronous motor and the like, and the core components of the lifting mechanism include a roller, a steel wire rope, a head sheave, a movable arm, an encoder, a tension sensor 2-4 and two 1.1kw three-phase asynchronous motors. In the process of excavation, the power provided by the driving motor is transmitted to the bucket through the transmission mechanism, and the inclination sensor, the wire displacement sensor, the encoder, the torque sensor 2-5 and the tension sensor installed on the working device are used to detect the running parameters of the excavation movement, and the pushing mechanism and the lifting mechanism work together to drive the bucket to complete the excavation work. The core components of the rotating device include a rotating motor, a walking motor, a rotating support, a rotating platform, a gear, a lower base, a track and an encoder. When the rotating device is running, the rotating motor provides power, the power is transmitted to the rotating platform through the rotating support and the pinion, and then the upper vehicle body is driven to realize the rotating function. When the walking device is running, the power provided by the two walking motors is transmitted to the track through the speed reducer, so as to realize the walking and turning functions of the electric shovel test bed.

[0089] According to one embodiment of the present application, step S2 comprises determining the TOF camera x and y direction focal length f x 、f y , the optical center c x 、c y in x and y directions by camera calibration, converting the depth image pixel coordinates (u, v) into points (X, Y, Z) in 3D space by the following formula:

[0090]

[0091] Z = D(u, v)

[0092] Where D(u, v) is the depth value corresponding to the pixel point (u, v) in the depth image;

[0093] There is a rotation and translation relationship between the TOF camera and the laser radar coordinate system. Through external parameter calibration, the transformation matrix T determined by the rotation matrix R and the translation vector t can be calculated:

[0094]

[0095] Using the transformation matrix T obtained by calibration, the point cloud generated by the TOF camera is converted from the camera coordinate system to the laser radar coordinate system:

[0096] P LiDAR = T·PTOF

[0097] wherein P LiDAR is a laser radar coordinate system, P TOF is a camera coordinate system;

[0098] After the coordinate conversion is completed, the point cloud generated by the TOF camera and the point cloud generated by the laser radar are both in the same coordinate system, so as to merge the TOF camera data and the laser radar data for point cloud registration.

[0099] According to one embodiment of the present application, the step S3 comprises: removing the noise points in the point cloud data through a median filter;

[0100] The point cloud density is reduced through a voxel grid method, the main features are kept, and the subsequent processing speed is accelerated; the normal vector of the point cloud is calculated through principal component analysis of the neighborhood points, so as to perform feature extraction, wherein the normal vector of the point cloud is calculated through the following formula:

[0101]

[0102] wherein N i is a neighborhood of the point P i , and V i is the normal vector;

[0103] The random sample consensus algorithm is used to segment the material point cloud to be excavated, the mine card point cloud and the obstacle point cloud from the global point cloud data, and the surface fitting is performed on the material pile surface and the ground.

[0104] According to one embodiment of the present application, the step S4 comprises: identifying the boundary points by calculating the normal vector and the curvature of the point cloud, so as to perform boundary extraction on the mine card box, wherein the point cloud curvature calculation formula is as follows:

[0105]

[0106] wherein N(p) is the domain point set of the point p, n(p) is the normal vector of the point p, and K(p) is the curvature of the point p;

[0107] The Harris corner point detection algorithm is used to detect the corner points of the mine card box by using the aggregation characteristics of the boundary points, the plane of the mine card box is fitted by using the random sample consensus algorithm; the geometric center and the relative coordinates of the mine card box are located by using the boundary points and the surface characteristics of the mine card box; the principal axis direction of the mine card box is obtained through the result of the principal component analysis method, and the minimum circumscribed rectangle of the mine card box is calculated, and the minimum circumscribed rectangle is the outer enclosing frame of the mine card box.

[0108] Specifically, according to the relatively obvious plane, corner and geometric features of the container, boundary extraction is first performed, boundary points are identified by calculating the normal vector and curvature of the point cloud, and the boundary points usually appear in places where the point cloud density changes significantly. By using the clustering feature of the boundary points, the Harris corner point detection is used to detect the corner points of the container. The random sample consensus (RANSAC) algorithm is used to fit the plane of the container, and by using the boundary points and the face features, the geometric center and the relative coordinates of the container can be accurately located. In order to facilitate subsequent operations (such as loading planning), it is necessary to fit an outer bounding box to the container, and the fitting method of the minimum circumscribed rectangle is to perform the most compact rectangular box surrounding the container point cloud. The principal component analysis (PCA) result can be used to obtain the principal axis direction of the container, and then the minimum circumscribed rectangle can be calculated. By using the fitted outer bounding box, the minimum distance error Error is used to verify the fitting accuracy:

[0109]

[0110] wherein N is the number of points in the point set, P i is the i-th point in the point set, P fit is the i-th point in the point set, P i is the corresponding point on the fitted model

[0111] According to one embodiment of the present application, in step S5, the walking track of the electric shovel is planned according to the point cloud of the material to be excavated and the point cloud of the container, including: obtaining the position coordinates of the electric shovel according to the point cloud data provided by the unmanned aerial vehicle, and determining the parking position of the container; using a hybrid A* algorithm, selecting key path points according to the map provided by the unmanned aerial vehicle, and planning a globally optimal route; when the container approaches the excavation area, entering a dynamic environment, and enabling a dynamic window algorithm to plan a local path for the container; using a dynamic scheduling algorithm to adjust the task priority of multiple containers in real time.

[0112] Specifically, according to the point cloud data provided by the tethered unmanned aerial vehicle, the position coordinates of the electric shovel are obtained, and the parking position of the container is determined; a hybrid A* algorithm is used to select key path points according to the map provided by the unmanned aerial vehicle, and a globally optimal route is planned. When the container approaches the excavation area, it enters a dynamic environment, and a DWA (dynamic window algorithm) is enabled for local path planning. When the container travels along the planned route to the excavation area, the DWA local algorithm will ensure that the container is parked directly below the electric shovel bucket, and the heading error is controlled within 3°, achieving precise parking. The system uses a dynamic scheduling algorithm to adjust the task priority of the container in real time. When a container is close to full load, it is preferentially arranged for unloading, and other containers continue to load operation. By intelligently allocating resources and optimizing the task sequence, the system maximizes the reduction of waiting time and conflicts, and improves the overall operation efficiency.

[0113] According to one embodiment of the present application, in step S5, the walking and digging and loading trajectory of the electric shovel is planned according to the material point cloud to be excavated and the truck point cloud, including: walking trajectory planning of the electric shovel, digging trajectory planning of the electric shovel and loading trajectory planning of the electric shovel, wherein the walking trajectory planning of the electric shovel includes: obtaining the remaining amount of material to be excavated around the electric shovel, when the remaining amount of material within the maximum working radius of the electric shovel is less than a preset threshold, a hybrid A* algorithm is used to plan the moving path of the electric shovel and navigate to the digging position with more material in the environment; when the electric shovel approaches the target position, a dynamic window algorithm is used for local path planning of the electric shovel, and the target point positioning error is controlled within 0.1m to realize accurate positioning of the electric shovel; the digging trajectory planning of the electric shovel includes: based on the point cloud information of the material to be excavated, combined with the self-attitude information of the electric shovel fed back by the self-sensor data, taking the unit volume digging energy consumption as the target, optimizing the digging trajectory, and selecting the best digging path in the trajectory planning by using the particle swarm algorithm, and through fitting the polynomial curve of the digging angle and the arm length, the trajectory is adjusted smoothly, wherein compared with the traditional algorithm, the particle swarm algorithm has the advantages of fast convergence speed and strong global optimization ability, ensuring efficient operation of the electric shovel under different terrain and load conditions; if the planned path does not meet the set boundary conditions, the trajectory will be automatically regenerated to ensure the continuity of the operation, wherein the set boundary conditions include: maximum digging depth and maximum lifting force; the loading trajectory planning of the electric shovel includes: combined with the forward and inverse kinematics solution of the electric shovel, the instructions corresponding to the actual lifting distance and the pushing distance are obtained, the lifting motor and the pushing motor of the electric shovel are controlled according to the instructions to complete the digging task together, and the attitude information fed back by the sensor of the electric shovel is used to close-loop feedback the digging action execution quality and timely correct the deviation, realizing high-precision digging. In the collaborative operation, the electric shovel dynamically adjusts the bucket angle and speed according to the real-time position of the truck, ensuring the accurate docking of the bucket and the truck. The electric shovel also maintains real-time communication with the truck, reasonably allocates tasks through the scheduling algorithm, avoids conflicts and resource waste among multiple devices, and improves the overall operation efficiency.

[0114] According to one embodiment of the present application, the local path planning using the dynamic window algorithm includes the following steps:

[0115] S51, initialize the state parameters of the device, the state parameters including the position (x, y) of the device, the orientation (θ), the speed (v), the angular velocity (ω), and set the maximum speed (vmax) and the maximum acceleration (amax) limits, and set the time step Δt;

[0116] S52, generate a feasible speed window according to the current state, the linear speed window v∈[v min ,v max ] and the angular velocity window ω∈[ω min ,ω max ], for each time step, the speed range is:

[0117] V d = [v current -a max · Δt, v current + a max · Δt]

[0118] Ω d = [ω current -a max · Δt, ω current + a max · Δt]

[0119] S53, for each speed combination, a trajectory in a future time range is predicted based on the current speed and angular velocity, and the position change in the time step is calculated;

[0120] S54, the predicted trajectory is evaluated with the evaluation index, and the trajectory with the lowest cost is selected as the motion plan of the current device, wherein the evaluation index includes at least one of the distance of the target, the distance from the obstacle and the trajectory smoothness;

[0121] S55, the linear speed and angular velocity control command is generated according to the selected optimal trajectory, and the motor of the device is driven to operate;

[0122] S56, steps S52 to S55 are repeatedly executed in each control period, and the state and trajectory planning of the device are continuously updated to adapt to the changes of the environment.

[0123] According to one embodiment of the present application, in step S5, the moving trajectory of the unmanned aerial vehicle is planned according to the material point cloud to be excavated and the mine truck point cloud, including: the unmanned aerial vehicle moving trajectory planning is carried out synchronously with the electric shovel walking trajectory planning, the unmanned aerial vehicle always stays above the electric shovel, when the electric shovel moves, the unmanned aerial vehicle moves synchronously, and provides stable environment sensing service for the electric shovel.

[0124] Further, as Figure 5 shown, the unmanned aerial vehicle 1 is provided with a warning device, when the electric shovel 2 or the mine truck 3 moves to the boundary of the positioning area, the warning device sends a warning signal to the electric shovel 2 or the mine truck 3, and drives the electric shovel 2 or the mine truck 3 to move into the positioning area, preventing the positioning of the electric shovel 2 or the mine truck 3 from being missed. The unmanned aerial vehicle 1 is provided with a communication relay device for communicating information with the electric shovel 2 or the mine truck 3.

[0125] Corresponding to the above embodiment, the present application also provides a mining electric shovel large scene mining and loading operation system.

[0126] Figure 6 A block diagram of the mining electric shovel large scene mining and loading operation system according to the embodiment of the present application.

[0127] As Figure 6As shown, the mining shovel large scene mining and loading operation system 100 of the embodiment of the present application includes a UAV, a shovel and a plurality of mine trucks, the UAV is respectively in communication connection with the shovel and the plurality of mine trucks, the system 100 can include: an environment perception module 110, a point cloud registration module 120, a material and mine truck point cloud segmentation module 130, a mine truck box positioning and outer enclosing frame fitting module 140 and a trajectory planning module 150.

[0128] The environment perception module 110 is configured to control the UAV to ascend to a preset height, and scan and shoot the mine site environment through the laser radar and the TOF camera carried on the UAV, so as to collect global point cloud data of the mine site environment, the global point cloud data including attitude information and position information of the shovel, attitude information and position information of the mine truck, and three-dimensional information of the material to be excavated. The point cloud registration module 120 is configured to perform point cloud registration on the TOF camera data and the laser radar data. The material and mine truck point cloud segmentation module 130 is configured to separate the material to be excavated point cloud and the mine truck point cloud from the global point cloud data, and to position the box of the mine truck and to perform surface fitting on the material to be excavated point cloud. The mine truck box positioning and outer enclosing frame fitting module 140 is configured to determine the position coordinates of the shovel bucket unloading according to the box position of the mine truck, and to fit the mine truck box point cloud to fit the outer enclosing frame of the mine truck box. The trajectory planning module 150 is configured to plan the walking trajectory of the mine truck, the excavating and loading trajectory of the shovel and the moving trajectory of the UAV according to the material to be excavated point cloud and the mine truck point cloud, so that the shovel and the plurality of mine trucks and the UAV cooperatively perform the mining and loading operation.

[0129] The mining electric shovel large scene mining and loading operation system of the embodiment mainly comprises a tethered unmanned aerial vehicle, an electric shovel and a mining truck, the tethered unmanned aerial vehicle specifically comprises a TOF camera, a laser radar, a communication relay and an edge detection device, the TOF camera can provide fast and real-time depth information by measuring the flight time of light pulses, and can capture high-resolution depth images, the laser radar can provide high-precision distance measurement in a longer distance, and the depth measurement accuracy is higher under various environmental conditions, and is especially suitable for large-scale scanning and laser radar, and can realize more comprehensive environmental perception, the high-resolution image provided by the TOF camera is combined with the long-distance measurement of the laser radar, and a more detailed and accurate three-dimensional map can be formed. The depth map generated by the TOF camera is converted into point cloud data, and the point cloud obtained by the laser radar is merged. The blank area of the laser radar data is filled by an interpolation method to form a more complete environmental model. The tethered unmanned aerial vehicle generates a three-dimensional map of the mining area through the laser radar and the TOF camera, transmits the point cloud data to the mining truck and the electric shovel, and pre-processes locally to extract the terrain change, obstacle position and real-time state information of the electric shovel. The unmanned aerial vehicle also acts as a communication relay to enhance signal coverage and reduce interference, ensuring stable and low-latency data transmission between the mining truck, the electric shovel and the control center. Through the relay device of the unmanned aerial vehicle, the signal can avoid interference or attenuation of direct transmission, reduce data packet loss and communication delay, and ensure accurate transmission of real-time information.

[0130] As the core execution device in mining operations, the electric shovel not only receives the three-dimensional map information transmitted by the unmanned aerial vehicle, but also real-time perceives its own working state (such as the position of the dipper arm, the load condition) and the distribution change of the material pile surface. The electric shovel calculates the total work of the lifting steel wire rope and the dipper arm push through the dynamics model, and optimizes the excavation trajectory with the minimum energy consumption of unit volume of material as the target. The system selects the best excavation path in the trajectory planning by using the particle swarm algorithm, and realizes the smooth adjustment of the trajectory by fitting the polynomial curve of the excavation angle and the dipper arm length. Compared with the traditional algorithm, the particle swarm algorithm has the advantages of fast convergence speed and strong global optimization ability, ensuring efficient operation of the electric shovel under different terrain and load conditions. If the planned path does not meet the set boundary conditions (such as maximum excavation depth, lifting force), the system will automatically regenerate the trajectory to ensure the continuity of the operation. In the cooperative operation, the electric shovel dynamically adjusts the dipper angle and speed according to the real-time position of the mining truck to ensure the accurate docking of the dipper and the mining truck. The electric shovel also maintains real-time communication with the mining truck, reasonably allocates tasks through the scheduling algorithm, avoids conflicts and resource waste among multiple devices, and improves the overall operation efficiency.

[0131] Through the wireless communication network, real-time information exchange is maintained between the unmanned aerial vehicle, the electric shovel and the mine truck. The mine truck receives the environment perception data (such as a three-dimensional map, obstacle position) from the unmanned aerial vehicle and the operation state information (such as the excavation progress, position) of the electric shovel. The mine truck combines the environment perception information provided by the unmanned aerial vehicle with its own sensor data by using a multi-sensor data fusion technology. The mine truck combines the map data of the unmanned aerial vehicle with its own motion state, updates the position and obstacle information in real time, and improves the accuracy of path planning. The mine truck first selects key path points according to the map provided by the unmanned aerial vehicle by using a hybrid A* algorithm, and plans a globally optimal route. When the mine truck approaches the excavation area, it enters a dynamic environment and enables a DWA (dynamic window algorithm) for local path planning. When the mine truck travels along the planned route to the excavation area, the DWA local algorithm ensures that the mine truck is parked in front of the electric shovel bucket, and the heading error is controlled within 3°, so as to realize accurate parking. The system uses a dynamic scheduling algorithm to adjust the task priority of the mine truck in real time. When a mine truck approaches full load, it is preferentially arranged for unloading, and other mine trucks continue to load operation. Through intelligent allocation of resources and optimization of task sequence, the system minimizes waiting time and conflicts, and improves overall operation efficiency.

[0132] It should be noted that the details of the mining electric shovel large scene mining and loading operation system of the embodiments of the present application are not disclosed. Please refer to the details disclosed in the mining electric shovel large scene mining and loading operation method of the embodiments of the present application. Here, specific details will not be described again.

[0133] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0134] In addition, the terms "first", "second" are only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one feature. In the description of the present application, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise specifically limited.

[0135] In this application, unless otherwise clearly indicated and limited, the terms "mounting", "connection", "connecting", "fixing" and the like should be interpreted broadly, for example, can be fixed connection, can also be detachable connection, or integral; can be mechanical connection, can also be electrical connection; can be directly connected, can also be indirectly connected through an intermediate medium, can be the internal communication of two elements or the interaction relationship of two elements, unless otherwise clearly limited. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0136] Although the embodiments of the present application have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present application.

Claims

1. A method for mining shovel large scene mining and loading operation, characterized in that, The mining shovel large scene includes a UAV, a shovel and a plurality of mine trucks, the UAV is respectively connected with the shovel and the plurality of mine trucks, and the method comprises: S1, control the UAV to rise to a preset height, and scan and shoot the mine site environment through the laser radar and TOF camera carried on the UAV to collect global point cloud data of the mine site environment, the global point cloud data comprising attitude information and position information of the shovel, attitude information and position information of the plurality of mine trucks, and three-dimensional information of the material to be excavated; S2, TOF camera data and laser radar data are point cloud registered; S3, the material to be excavated point cloud and the mine truck point cloud are separated from the global point cloud data, and the cargo box of the mine truck is positioned and the surface of the material to be excavated point cloud is fitted; S4, the position coordinates of the shovel bucket unloading are determined according to the cargo box position of the mine truck, and the cargo box point cloud of the mine truck is fitted to fit the outer enclosing frame of the mine truck cargo box; S5, the walking track of the mine truck, the walking excavation loading track of the shovel and the moving track of the UAV are respectively planned according to the material to be excavated point cloud and the mine truck point cloud, so that the shovel and the plurality of mine trucks and the UAV cooperatively perform the mining and loading operation; The walking track of the shovel is planned according to the material to be excavated point cloud and the mine truck point cloud, comprising: According to the point cloud data provided by the UAV, the position coordinates of the shovel are obtained, and the parking position of the mine truck is determined; A hybrid A* algorithm is used to select key path points according to the map provided by the UAV to plan a globally optimal route; When the mine truck approaches the excavation area, a dynamic environment is entered, and a dynamic window algorithm is enabled to plan a local path for the mine truck; A dynamic scheduling algorithm is used to adjust the task priority of the plurality of mine trucks in real time; The walking excavation loading track of the shovel is planned according to the material to be excavated point cloud and the mine truck point cloud, comprising the walking track planning of the shovel, the excavation track planning of the shovel and the loading track planning of the shovel, wherein, The walking track planning of the shovel comprises: obtaining the remaining amount of the material to be excavated around the shovel, when the remaining amount of the material in the maximum working radius of the shovel is less than a preset threshold, a hybrid A* algorithm is used to plan the moving path of the shovel to navigate to the excavation position with more material in the environment; when the shovel approaches the target position, a dynamic window algorithm is used to plan a local path for the shovel to achieve accurate positioning of the shovel; The excavation track planning of the shovel comprises: based on the material to be excavated point cloud information, combining the self attitude information fed back by the self sensor data of the shovel, taking the unit volume excavation energy consumption as the target, optimizing the excavation track, and using a particle swarm algorithm to select the best excavation path in the track planning, and through fitting the polynomial curve of the excavation angle and the boom length, the track is smoothly adjusted; if the planned path does not meet the set boundary condition, the track is automatically regenerated to ensure the continuity of the operation, wherein the set boundary condition comprises the maximum excavation depth and the maximum lifting force. The loading trajectory planning of the electric shovel comprises: obtaining instructions corresponding to actual lifting distance and pushing distance in combination with forward and inverse kinematics solution of the electric shovel, controlling the lifting motor and the pushing motor of the electric shovel to jointly complete the excavation task according to the instructions, and using posture information returned by a sensor of the electric shovel to close-loop feedback excavation action execution quality and timely correct deviation, thereby realizing high-precision excavation; The local path planning using a dynamic window algorithm comprises: S51, initialize state parameters of the device, including position (x, y), orientation θ, velocity v, angular velocity ω of the device, and set maximum velocity v max and maximum acceleration a max limit, set time step Δt; S52, generate feasible speed window according to current state, linear speed window and angular speed window For each time step Δt, the linear speed feasible range Vd and the angular speed feasible range Ωd are: ， ， where v current is the current linear velocity, ω current is the current angular velocity; S53, for each speed combination, predicting a trajectory in a future time range based on the current speed and angular velocity, and calculating the position change in the time step; S54, evaluating the predicted trajectory against evaluation indexes, and selecting a trajectory with the lowest cost as the motion plan of the current device, wherein the evaluation indexes comprise at least one of distance to the target, distance to the obstacle and trajectory smoothness; S55, generating linear velocity and angular velocity control commands according to the selected optimal trajectory, and driving the motor of the device to operate; S56, repeatedly executing steps S52 to S55 in each control cycle to continuously update the state and trajectory planning of the device to adapt to changes in the environment.

2. The mining shovel large scene mining operation method according to claim 1, characterized by, In step S1, the unmanned aerial vehicle is a tethered unmanned aerial vehicle, and the laser radar is a multi-line laser radar, which scans the surrounding environment of the electric shovel to obtain point cloud data of the surrounding environment of the electric shovel, and the TOF camera obtains a mine site environment depth map.

3. The mining shovel large scene mining operation method according to claim 2, characterized by, Step S2 comprises: Determine the TOF camera x and y direction focal length by camera calibration , , x and y direction optical center , , convert depth image pixel coordinates (u, v) to a point in 3D space (X, Y, Z) by the following equations: ， ， ， wherein, is the depth value corresponding to the pixel point (u, v) in the depth image; Through external parameter calibration, a transformation matrix T determined by a rotation matrix R and a translation vector t is calculated: ， Using the transformation matrix T obtained by calibration, the point cloud generated by the TOF camera is converted from the camera coordinate system to the laser radar coordinate system: ， wherein, is a laser radar coordinate system, is a camera coordinate system; After completing the coordinate conversion, the point cloud generated by the TOF camera and the point cloud generated by the laser radar are both in the same coordinate system, so that the TOF camera data and the laser radar data are merged for point cloud registration.

4. The mining shovel large scene mining operation method according to claim 1, characterized by, Step S3 comprises: Noise points in the point cloud data are removed through a median filter; The point cloud density is reduced through a voxel grid method to maintain main features and speed up subsequent processing; The normal vector of the point cloud is calculated through principal component analysis of neighborhood points for feature extraction, wherein the normal vector of the point cloud is calculated through the following formula: ， wherein is a point of the neighborhood, is a normal vector, p is a point within the neighborhood; The random sample consensus algorithm is used to segment the material point cloud to be excavated, the mine truck point cloud and the obstacle point cloud from the global point cloud data, and the material pile surface and the ground are curvedly fitted.

5. The mining shovel large scene mining operation method according to claim 1, characterized by, Step S4 comprises: Boundary points are identified by calculating the normal vector and curvature of the point cloud to extract the boundary of the mine truck container, wherein the point cloud curvature calculation formula is as follows: ， wherein, is a set of field points for point p, is a normal vector for point p, is a curvature of point p, is a normal vector for point q in the neighborhood; The Harris corner detection algorithm is used to detect the corner points of the mine truck container by using the aggregation feature of the boundary points, and the plane of the mine truck container is fitted using the random sample consensus algorithm; The geometric center and relative coordinates of the mine truck container are located by the boundary points and surface features of the mine truck container; The principal axis direction of the mine truck container is obtained from the result of the principal component analysis method, and the minimum circumscribed rectangle of the mine truck container is calculated, which is the outer bounding box of the mine truck container.

6. The mining shovel large scene mining operation method according to claim 1, characterized by, In step S5, the movement trajectory of the unmanned aerial vehicle is planned according to the point cloud of the material to be excavated and the point cloud of the mine truck, including: The movement trajectory planning of the unmanned aerial vehicle is synchronized with the walking trajectory planning of the electric shovel, and the unmanned aerial vehicle always stays above the electric shovel. When the electric shovel moves, the unmanned aerial vehicle moves synchronously to provide stable environment perception services for the electric shovel.

7. A large scene mining shovel mining and loading operation system, using the large scene mining shovel mining and loading operation method according to claim 1, characterized in that, The mine electric shovel large scene includes an unmanned aerial vehicle, an electric shovel and multiple mine trucks. The unmanned aerial vehicle is respectively connected with the electric shovel and multiple mine trucks in communication. The system comprises: An environment perception module is configured to control the unmanned aerial vehicle to rise to a preset height, and scan and shoot the mine site environment through the laser radar and TOF camera carried on the unmanned aerial vehicle to collect global point cloud data of the mine site environment. The global point cloud data includes attitude information and position information of the electric shovel, attitude information and position information of the mine truck, and three-dimensional information of the material to be excavated. A point cloud registration module is configured to register the TOF camera data and the laser radar data. A material and mine truck point cloud segmentation module is configured to separate the point cloud of the material to be excavated and the point cloud of the mine truck from the global point cloud data, and to locate the cargo box of the mine truck and perform surface fitting on the point cloud of the material to be excavated. A mine truck cargo box positioning and outer enclosing frame fitting module is configured to determine the position coordinates of the electric shovel bucket unloading according to the cargo box position of the mine truck, and to fit the cargo box point cloud of the mine truck to fit the outer enclosing frame of the cargo box of the mine truck. A trajectory planning module is configured to plan the walking trajectory of the mine truck, the excavation and loading trajectory of the electric shovel and the movement trajectory of the unmanned aerial vehicle according to the point cloud of the material to be excavated and the point cloud of the mine truck, so that the electric shovel and multiple mine trucks and the unmanned aerial vehicle cooperatively perform the mining and loading operation.

Images