Autonomous loading and unloading method of unmanned loader for bulk material transportation in mining areas

Abstract

The present invention discloses an autonomous loading and unloading method for an unmanned loader applicable to bulk material transportation in a mining area, and relates to the field of unmanned loading and unloading and transportation control technology for mining, including: S1, functional division of an open-pit mining area to obtain corresponding bulk material storage areas, unmanned loader waiting areas, and truck waiting areas; S2, integrating an autonomous loading and unloading system on the unmanned loader, so as to obtain mining area environmental information through the autonomous loading and unloading system, construct a global map of the operating area and plan the operating path, and complete the loading and unloading operation, wherein, in the autonomous loading and unloading system, the loading and unloading operation logic module is used as the main process. The present invention provides an autonomous loading and unloading method for an unmanned loader applicable to bulk material transportation in a mining area, so that all scenes in the bulk material loading and unloading process in the mining area are independently realized by the unmanned loader, that is, the unmanned loader relies on advanced sensors and intelligent algorithms to accurately construct an operating environment map and realize efficient autonomous loading and unloading.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned loading and unloading and transportation control for mines. More specifically, the present invention relates to an autonomous loading and unloading method of an unmanned loader applicable to bulk material transportation in mining areas. Background Art

[0002] Compared with traditional manned loaders, unmanned operation of unmanned loaders ensures the safety of workers and can operate stably in dangerous and harsh environments. In addition, unmanned operation replaces people to perform repetitive work, reducing labor costs and improving the economic benefits of enterprises. Moreover, unmanned loaders can achieve continuous operation 24 hours a day, which can greatly improve production efficiency. With the continuous development of unmanned automation technology in the mining industry, the intelligent control and detection of unmanned mining equipment has become a research hotspot. In the prior art, there are many related technical solutions, but each faces some problems and challenges. Unmanned loaders are generally excavators that travel between the loading area and unloading area of ​​the mine, used for excavating, transporting, and unloading ore to the transport vehicle. The transport vehicle is filled with ore by an excavator in the loading area. After the ore is full, it is transported to the designated location for unloading as needed.

[0003] However, for mining areas, the intelligent loading and unloading methods and equipment involved in the prior art are too complicated and inflexible, such as the control method, device, medium and electronic equipment for unmanned mining transport vehicles involved in the announcement number CN114326738B. This patent determines the transportation route based on the operating position of the excavation equipment, but the path planning is not flexible enough. Once the road conditions in the mining area suddenly change, such as landslides or new obstacles, the vehicle may not be able to adjust the path in time. At the same time, there is insufficient consideration for the collaborative operation of multiple vehicles. In actual mining operations, there may be a risk of collision when multiple vehicles work at the same time, and the overall transportation efficiency needs to be improved.

[0004] Another example is the bulk material automatic loading method and system based on multi-sensor fusion with announcement number CN118560956A. The main problems are as follows: 1. The equipment integration is difficult and the maintenance cost is high. 2. The performance is degraded in complex environments (such as strong light, bad weather, special terrain, etc.). 3. It is aimed at the content of the bulk material automatic loading method and system, and does not involve autonomous unloading.

[0005] Another example is the perception-based unmanned transport vehicle pointing and docking method with announcement number CN116578081A. The problem is that it is mainly targeted at specific loading area docking scenarios, has poor adaptability and limited functional expansion. There may be problems such as the human-computer interaction is not intuitive and efficient, and the work efficiency is affected.

[0006] Therefore, it can be seen that the above methods cannot be applied in an integrated manner to bulk material loading and unloading scenarios in mining areas. Summary of the invention

[0007] An object of the present invention is to solve at least the above problems and / or disadvantages and to provide at least the advantages which will be described hereinafter.

[0008] In order to achieve these purposes and other advantages of the present invention, a method for autonomous loading and unloading of bulk materials by an unmanned loader applicable to bulk material transportation in a mining area is provided, comprising:

[0009] S1. Functionally divide the open-pit mining area to obtain the corresponding bulk material storage area, unmanned loading vehicle waiting area, and truck waiting area;

[0010] S2. Integrate an autonomous loading and unloading system on the unmanned loader to obtain mining area environmental information through the autonomous loading and unloading system, build a global map of the operation area and plan the operation path to complete the loading and unloading operation;

[0011] Among them, in the autonomous loading and unloading system, the loading and unloading operation logic module is used as the main process, and its working method is:

[0012] After constructing the global map M of the operation area or dynamically updating the global map M, it is determined in real time whether a loading order from the dispatch center is received. If so, a loading station path to the bulk material storage area is planned and the loading station is reached. Otherwise, the unmanned loader goes to the unmanned loader waiting area.

[0013] After the unmanned loader arrives at the loading station, it determines in real time whether it has reached the bulk material excavation station. If not, it continues to go there. If it has, it executes the autonomous excavation process and enters the global map M update process after completing the excavation.

[0014] The unmanned loader plans a path to the truck waiting area and goes to the truck waiting area to execute the autonomous unloading process. If the vehicle is not full, it continues to plan the path to the loading station. If the vehicle is full, it determines whether there is the next vehicle to load. If so, it repeats the autonomous unloading process or loading process, otherwise it goes to the unmanned loader waiting area.

[0015] Preferably, the autonomous loading and unloading system comprises:

[0016] Provides a human-machine interaction interface to support the loading and unloading interaction layer for the operation configuration and control of unmanned loaders;

[0017] Provides core perception and control algorithms for unmanned loaders and supports the intelligent computing layer for all operations;

[0018] A functional device layer that interacts with the intelligent computing layer to perform corresponding actions;

[0019] The environment perception module, the action planning module, and the device driving module are connected to form a loading and unloading operation logic module of the business operation flow and are set on the intelligent computing layer. The intelligent computing layer also includes:

[0020] The environment perception module is based on the visual equipment and radar on the unmanned loader to obtain environmental and operating target information to complete map construction, autonomous positioning, vehicle posture calculation, bulk material recognition, truck recognition, excavator arm posture calculation, bulk material position calculation, unloading position calculation, and compartment material level calculation;

[0021] Motion planning module for planning loading path, unloading path, digging arm motion, and autonomous collision avoidance motion to achieve autonomous control;

[0022] A device driver module that corresponds one-to-one with the underlying device to provide underlying hardware connection and / or driver information.

[0023] Preferably, the process of dynamically updating the global map M includes:

[0024] Determine whether a map update command is received, and if so, adjust the posture of the unmanned loader to enter the map update mode;

[0025] The unmanned loader scans the current position to obtain the local point cloud C1, estimates the current position P in the global map M, and searches the global map M for a point cloud with the C1 feature at point P when the map update conditions are met. The point cloud in the global map M that satisfies the following formula is Recorded as :

[0026]

[0027] in, , u is the number of points found that meet the conditions, is the jth point in the point cloud subset that meets the conditions, are the coordinates of a point on the map, is the center coordinate of the local point cloud C1, a is the half-side length in the x-axis direction, b is the half-side length in the y-axis direction, and c is the half-side length in the z-axis direction;

[0028] Execute M = Map(M, C1) to update the global map M.

[0029] Preferably, the process of executing M = Map(M, C1) includes:

[0030] Will As the newly acquired point cloud data, in the global map M and Select the initial corresponding point pair set from ,and ,in , K is the number of initial corresponding point pairs, is the distance threshold for feature matching;

[0031] Based on the initial corresponding point pair set P k , the point cloud subset can be calculated by the following formula The transformation matrix T that matches the points in with the corresponding points in the global map M is:

[0032]

[0033] Apply the transformation matrix T to the point cloud subset All points of the transformed point cloud subset are obtained ,and , n represents the number of points in the point cloud after transformation;

[0034] Re-search in the global map M The nearest point of each point in is used to update the corresponding point pair set. Let the updated corresponding point pair set ,and ,in , L is the number of corresponding point pairs after update;

[0035] The current matching quality is evaluated by the root mean square error RMSE as follows: , then proceed to the next update operation; otherwise, continue to iterate until the upper limit of the number of iterations is reached :

[0036]

[0037] When the matching quality meets the requirements, All the points are added to M to update the global map M.

[0038] Preferably, the mining area environmental information is obtained by a method of joint calibration of dual laser radar and camera, and the process of joint calibration of dual laser radar and camera includes:

[0039] The point cloud after hybrid laser radar scanning calibration is obtained by calibrating the dual laser radars ;

[0040] Jointly calibrate the hybrid lidar and camera;

[0041] When calibrating dual laser radars, one of the laser radars, Lidar1, is used as the fixed coordinate system, and the coordinates of the other laser radar, Lidar2, are converted to Lidar1. After obtaining the homogeneous transformation matrix between Lidar1 and Lidar2, the point cloud Obtained by the following formula:

[0042]

[0043] In the above formula, is the unit matrix;

[0044] The method of joint calibration of hybrid lidar and camera is:

[0045] Hybrid LiDAR The conversion relationship from the coordinate system to the camera coordinate system is:

[0046]

[0047] In the above formula, is the hybrid lidar coordinate system, is the camera coordinate system, They are the rotation matrix and translation matrix from the hybrid lidar coordinate system to the camera coordinate system;

[0048] Collecting images observed by both the camera and the hybrid lidar n t feature points, and n t The pixel coordinates of the feature points in the camera image are , is the u-axis coordinate of the pixel point on the camera image plane, and the u-axis is horizontal to the right. is the v-axis coordinate of the pixel point on the camera image plane, the v-axis is vertically downward, and the point cloud coordinate in the hybrid lidar is ;

[0049] From the pixel coordinates, Reverse the coordinates in the camera coordinate system :

[0050]

[0051] The conversion relationship between the hybrid laser radar and the camera can be expressed by the following formula to calculate and The value of , completes the joint calibration:

[0052]

[0053] In the above formula, is the rotation matrix The values ​​of i and j are 1, 2, and 3 respectively.

[0054] Preferably, the autonomous excavation process includes:

[0055] Use the visual equipment and radar on the unmanned loader to search for bulk material piles in the environment;

[0056] Measure the bulk material position Q closest to the unmanned loading vehicle;

[0057] Plan the best path for the unmanned loader to the bulk material location Q, and adjust the excavation posture to perform the excavation action after the unmanned loader arrives at Q;

[0058] The process of searching for bulk material piles includes:

[0059] Prepare a stockpile dataset and train a convolutional neural network segmentation model;

[0060] The convolutional neural network segmentation model is used to segment the two-dimensional images obtained by the unmanned loading vehicle;

[0061] Filter the segmentation results to obtain the filtered point cloud set P f ;

[0062] For a set observation point, the point cloud set is calculated by the following formula: P f The point cloud point closest to the observation point p closest :

[0063]

[0064] In the above formula, O g are the observation point coordinates, P j Point cloud collection P f Every point in .

[0065] Preferably, in the process of training the convolutional neural network segmentation model, the total loss function of the model is defined by the following formula: L tolal :

[0066]

[0067] In the above formula, L class is the classification loss, L mask is the mask segmentation loss, L edge is the mask edge loss, λ is the weight hyperparameter of the mask edge loss;

[0068] Among them, the classification loss function L class Obtained by the following formula:

[0069]

[0070] In the above formula, t i is the true category label, S i is the probability predicted by the model;

[0071] Mask segmentation loss L mask Obtained by the following formula:

[0072]

[0073] In the above formula, m j is the true mask, The mask value predicted by the model;

[0074] Mask edge loss L edge Obtained by the following formula:

[0075]

[0076] In the above formula, and They are the gradient values ​​of the real mask and the predicted mask at the edge respectively.

[0077] Preferably, the autonomous unloading process includes:

[0078] The unmanned loading vehicle searches to identify whether there is a truck in the parking space ahead. If there is, the material level in the truck compartment is observed to determine whether the compartment is full. If not, unloading is performed;

[0079] Among them, when unloading operation is carried out, the unmanned loading vehicle scans the carriage in real time to update the unloading position through material level observation, and then determines whether to stop the unloading operation;

[0080] The process of searching and identifying whether there is a truck in the parking space ahead includes:

[0081] The images collected by the camera and the point cloud data collected by the lidar are preprocessed to obtain the corresponding image sets. I processed , Point Cloud Collection P filtered ;

[0082] Image collection based on timestamp I processed , Point Cloud Collection P filtered Perform data synchronization to obtain the filtered image collection I selected , Point Cloud CollectionP selected ;

[0083] Use the segmentation model to filter the image set I selected Segment the truck in the image and get the segmented mask image M y ;

[0084] Based on the filtered point cloud collection P selected Each point in In the mask picture M y Is there a corresponding pixel value in the point cloud to complete the point cloud filtering and obtain the filtered point cloud set P filtered2 ;

[0085] Use clustering algorithm to filter the point cloud collection P filtered2 Clustering is performed and the point cloud collection after clustering is P truck A maximum plane is fitted in the image, and then the point cloud at 1 / 2 of the maximum plane is returned to calculate the distance and direction of the truck from the radar.

[0086] Preferably, the material level observation process includes:

[0087] The origin is the bottom center of the empty truck compartment. O c , establish a three-dimensional rectangular coordinate system along the length, width and height directions of the carriage;

[0088] Assume that the laser radar is at a fixed position above the carriage with the bucket of the unmanned loader. After each unloading, the laser radar scans and measures the material in the carriage and obtains the coordinate information of a series of discrete points on the surface of the material, which is recorded as ,in, r =1,2…, n r , n r is the number of measurement points, then the height of each point on the material surface relative to the center of the bottom of the carriage h i It can be expressed by the following formula:

[0089] ;

[0090] The process of determining whether the carriage is full is:

[0091] Assign a weight to each measurement point in the cabin collected by the LiDAR w i, to calculate the weighted average material height by the following formula h wavg :

[0092]

[0093] The fullness of the carriage can be determined by the following formula h wavg Make a judgment:

[0094]

[0095] In the above formula, k is the filling coefficient and H is the height of the carriage.

[0096] Preferably, the process of updating the discharge level through material level observation is:

[0097] Assume that the initial position O of the unloading position of the carriage is ;

[0098] Find the point with the minimum material height among the scanned points in the truck compartment, recorded as , through the following formula The average value of all x-coordinates and y-coordinates in are used as the x-coordinate and y-coordinate of the updated unloading position:

[0099] ,

[0100] In the above formula, N min is the number of scan points with minimum material height, x new To update the x coordinate of the discharge position, y new To update the y coordinate of the discharge position, h min is the minimum material height, x ij For all x-coordinates with the minimum material height, y ij for all y coordinates with minimum material height;

[0101] The new unloading position for .

[0102] The present invention includes at least the following beneficial effects: the present invention provides an unmanned loader autonomous loading and unloading system method, which adopts an unmanned loader autonomous loading and unloading system set on a single unmanned loader, so that all scenes in the bulk material loading and unloading process in the mining area are independently realized by the unmanned loader, that is, the unmanned loader relies on advanced sensors and intelligent algorithms to accurately construct an operating environment map and realize efficient and autonomous loading and unloading. Compared with the existing technology, there is no need to arrange other equipment and perform complex algorithm design, and it is more feasible and better adaptable.

[0103] Specifically, the present invention has the following advantages over the prior art:

[0104] First, the present invention integrates multiple functions into an unmanned loader by constructing an autonomous loading and unloading system with a layered architecture, thereby reducing compatibility issues between different devices and lowering the difficulty of integration; and the system is relatively independent, which is convenient for maintenance to a certain extent, reducing the risk of operation interruption due to equipment failure, thereby alleviating the defects of the prior art in this regard to a certain extent.

[0105] Second, the dual laser radar and camera joint calibration technology and the perfect map update mechanism in the present invention enhance the perception and adaptability of unmanned loaders in complex mining environments, enabling them to better cope with environmental changes and reduce performance degradation caused by environmental factors. Compared with the existing technology, it can improve adaptability to complex environments, and the dual laser radar and camera joint calibration has the advantages of low cost and high resolution.

[0106] Third, the advantage of the present invention is that, in terms of unmanned forklift operations, the entire process from searching for bulk material piles to completing loading is planned in detail, including path planning, posture adjustment and state judgment, etc., making the automatic loading process of the unmanned forklift more intelligent and efficient, and improving the autonomy and accuracy of the overall operation.

[0107] Fourth, the present invention designs a universal autonomous loading and unloading process, which can be flexibly adjusted according to different mine layouts and operational requirements and is not limited to specific scenarios; at the same time, its system architecture provides a basis for functional expansion, facilitates the integration of more functions, and solves the shortcomings of CN116578081A in this regard.

[0108] Fifth, the loading and unloading interaction layer of the present invention provides rich human-computer interaction functions, and the entire operation process improves efficiency through intelligent control. Compared with the existing technology, there are obvious improvements in human-computer interaction and operation efficiency.

[0109] Other advantages, objectives and features of the present invention will be embodied in part through the following description, and in part will be understood by those skilled in the art through study and practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0110] Figure 1 This is a schematic diagram of the architecture of the autonomous loading and unloading system of the unmanned loader of the present invention;

[0111] Figure 2 This is the main flow chart of autonomous loading and unloading of the unmanned loader of the present invention;

[0112] Figure 3 It is a global map M update flow chart of the present invention;

[0113] Figure 4 It is a diagram of the system calibration process of the present invention;

[0114] Figure 5 This is a calibration model diagram of the vertical dual laser radar of the present invention;

[0115] Figure 6 It is a flow chart of autonomous bulk excavation of the present invention;

[0116] Figure 7 The loss function composition and its functional diagram of the present invention;

[0117] Figure 8 It is a schematic diagram of the autonomous unloading process of the present invention. DETAILED DESCRIPTION

[0118] The present invention is further described in detail below in conjunction with the accompanying drawings so that those skilled in the art can implement the invention with reference to the description.

[0119] 1. Autonomous loading and unloading system for unmanned loader

[0120] like Figure 1 As shown in the figure, the autonomous loading and unloading system of the unmanned loader is divided into: loading and unloading interaction layer, intelligent computing layer and functional equipment layer. The loading and unloading interaction layer provides a human-machine interaction interface and supports the operation configuration and control of the unmanned loader, including: scene selection, function configuration, autonomous loading and unloading, and remote loading and unloading. The intelligent computing layer provides the core perception and control algorithm of the unmanned loader and supports all operations, including: loading and unloading operation logic, environmental perception, action planning, and device driver. Among them, the loading and unloading operation logic is the main process, connecting the functional modules into a business operation flow; environmental perception supports the unmanned loader to detect the environment and operation targets; action planning supports autonomous control; and the device driver provides information connection to the underlying hardware. Environmental perception includes: map construction, autonomous positioning, vehicle posture calculation, bulk material recognition, truck recognition, excavator arm posture calculation, bulk material position calculation, unloading position calculation, and carriage material position calculation. Action planning includes: loading path planning, unloading path planning, excavator arm action planning, and autonomous collision avoidance. The device drivers correspond one-to-one to the underlying devices, including: chassis wire control, excavator arm wire control, lidar drive, vision drive, inertial sensor drive, satellite positioning drive, observation gimbal drive, and wireless communication drive.

[0121] The main process of autonomous loading and unloading by unmanned loader is as follows Figure 2 As shown in the figure, after building the global map of the operation area, go to the waiting station to determine whether the loading order from the dispatch center has been received. If the order is received, plan the path to the loading station and go to the loading station; if the order is not received, go to the waiting station (i.e., the waiting area for unmanned loading vehicles). After arriving at the loading station (i.e., the bulk material stacking area), determine whether the bulk material excavation station has been reached. If not, continue to go there. If reached, execute the autonomous excavation process. If the loading is not completed, continue to execute the excavation process. If the loading is completed, execute the global map M update process, then plan the path to the unloading observation station and go to the unloading observation station. After reaching the unloading observation station (i.e., the truck waiting area), execute the autonomous unloading process. If the car is not full, continue to plan the loading path. If the car is full, determine whether there is a next car to load. If so, plan the loading path as well. If not, go to the waiting station.

[0122] 2. Autonomous loading and unloading process

[0123] The autonomous loading and unloading process mainly includes: global map M update - autonomous bulk material excavation process - autonomous unloading process.

[0124] It should be noted that the global map M update process is as follows Figure 3 As shown in the figure, first obtain the global map M, then determine whether the map update command is received, and wait if not; after receiving the command, adjust the posture of the unmanned forklift to enter the map update mode. Then build a local map, scan the current position to obtain the local point cloud C1, and estimate the current position P in the global map M. Then determine whether the map update conditions are met. If not, return to the step of rebuilding the local map; if yes, find the point cloud with C1 feature at point P in the global map M, and finally execute the operation to update the global map M, thus completing the entire map update process.

[0125] The process of finding a point cloud with C1 feature at point P in the M map includes:

[0126] Furthermore, the process of finding the point cloud with C1 feature at point P in the M map is as follows:

[0127] 1. Assume that the global map M can be represented as a point cloud set ,in are the coordinates of a point on the map.

[0128] 2. For the rectangular space C 1， Let the center coordinates be , the half-side length in the x-axis direction is a, the half-side length in the y-axis direction is b, and the half-side length in the z-axis direction is c.

[0129] 3. Find the point cloud with rectangular space C1 feature at point P in map M, that is, find the point that satisfies the following conditions The point clouds found are recorded as ,Right now , where u is the number of points found that meet the conditions, is the jth point in the point cloud subset that meets the conditions.

[0130] Execute M=Map(M,C1), and the process of updating the global map M includes:

[0131] Furthermore, executing M=Map(M,C1), the process of dynamically updating the global map M is:

[0132] 1. Subset the found point cloud As the newly acquired point cloud data, it is used to update the global map M. Set the parameters required for feature matching execution and the matching distance threshold , the upper limit of the number of iterations .

[0133] 2. In the global map M and point cloud subset Select the initial corresponding point pairs from the original point cloud and the target point cloud. Select based on the geometric feature preliminary matching method. Suppose the selected initial corresponding point pair set is ,in, , K is the number of initial corresponding point pairs.

[0134] 3. Based on the selected initial corresponding point pair set P k , calculate a point cloud subset by the least squares method The transformation matrix T that matches the points in as much as possible with the corresponding points in the global map M. The goal is to minimize the following objective function:

[0135]

[0136] By solving the minimum value of the above objective function, the transformation matrix T is obtained.

[0137] 4. Apply the calculated transformation matrix T to the point cloud subset All points of the transformed point cloud subset are obtained . Then search again in the global map M The nearest point of each point in is used to update the corresponding point pair set. Let the updated corresponding point pair set be ,in , L is the number of corresponding point pairs after update.

[0138] 5. Use the root mean square error (RMSE) to evaluate the current matching quality. The calculation formula is:

[0139]

[0140] if , it is considered that the matching quality meets the requirements and the next update operation can be performed; otherwise, it is necessary to continue iterating until the upper limit of the number of iterations is reached. .

[0141] 6. When the matching quality meets the requirements, the transformed point cloud subset Directly merge with the original global map M, All the points of are added to M, that is, , to update the global map M.

[0142] Dual LiDAR + Camera Joint Calibration Method: Figure 4 As shown in the figure, two laser radars are first integrated into one laser radar, and then the integrated laser radar and camera are jointly calibrated to obtain a laser radar with a larger viewing angle. The specific process includes:

[0143] Step 1: Dual LiDAR Calibration

[0144] 1. Establishment of point cloud correspondence

[0145] Use the KD-tree point cloud data structure to find the point closest to each point. For the point clouds of the two laser radars, they are set as and .

[0146] 2. Iterative optimization based on least squares method

[0147] Due to measurement errors and other reasons, the point set cannot correspond one to one. Based on the least squares method, the Euclidean norm is minimized, and the objective function is:

[0148]

[0149] Among them, R is the rotation matrix and t is the translation vector.

[0150] Solving for the optimal translation matrix :

[0151] Calculation point set , The centroid , , respectively and , let the partial derivative of the objective function with respect to t be 0, and get the optimal translation matrix .

[0152] Solving for the optimal rotation matrix :

[0153] Will Substituting into the objective function, we can simplify it to . To process F(R), use matrix singular value decomposition and set , perform singular value decomposition on H , where U and V are orthogonal matrices, is a diagonal matrix, then the optimal rotation matrix .

[0154] 3. Iteration termination condition

[0155] By calculating the optimal , After that, the transformed point cloud is used as the new point cloud input of the algorithm to iteratively calculate the new point set , and its corresponding optimal , , until the change in the loss function is less than a certain threshold or the iteration number reaches the maximum, the iteration is terminated, and two lidar calibration results are obtained.

[0156] 4. Point cloud transformation

[0157] After obtaining the homogeneous transformation matrix between the Lidars, let the number of point clouds of Lidar1 be N1, which is defined as , the point cloud of Lidar2 is defined as The point clouds after the two radar calibrations are , as shown below:

[0158]

[0159] It should be noted that during the transformation process, the points in the point cloud should be represented as .

[0160] In this paper, Lidar1 is used as a fixed reference system, that is, After the calibration is completed, the relative position and posture of the two radars are fixed, that is, the parameters are fixed, and the real-time calibrated point cloud can be obtained. At this time, the point cloud scanned by the hybrid laser radar can be obtained. .

[0161] After the calibration is completed, the relative position and posture of the two radars are fixed, that is, the parameters are fixed, and the real-time calibrated point cloud can be obtained. At this time, a hybrid lidar composed of two radars can be obtained. The point cloud obtained by scanning. Assume that the final field of view is ,like Figure 5 As shown, since the vertical scanning range of each laser radar is , When the two radars work separately, their coverage areas are , , assuming that the scanning range of Lidar1 is from , the scanning range of Lidar2 is from Now consider the case of two radars. Assume that after the rotation matrix of the extrinsic matrix is ​​converted into Euler angles, the scanning angles of the two radars partially overlap and can achieve a wider range of coverage. Assume the overlapping angle , then the total coverage angle is the non-overlapping part of Lidar1 plus the overlapping part plus the non-overlapping part of Lidar2, that is, . Finally, the vertical field of view can be achieved Then, the hybrid lidar and camera are calibrated to achieve the final system calibration.

[0162] Step 2: Hybrid LiDAR and Camera Joint Calibration

[0163] 1. Coordinate system and transformation relationship

[0164] Define the world coordinate system , Camera coordinate system , Hybrid LiDAR Coordinate System The transformation relationship from the world coordinate system to the camera coordinate system can be expressed as:

[0165]

[0166] in, , The rotation and translation matrices from the world coordinate system to the camera coordinate system are respectively.

[0167] From world coordinate system to hybrid lidar coordinate system is converted to:

[0168] in, , They are the rotation and translation matrices from the world coordinate system to the hybrid lidar coordinate system.

[0169] The intrinsic parameter matrix of the camera is:

[0170]

[0171] in: f x and f y The camera is x Axis and y The focal length along the axis, in pixels. cx and c y is the center coordinate of the image plane, that is, the pixel coordinate of the intersection of the optical axis and the image plane in the image. Usually, the center of the image plane is not necessarily the center pixel of the image, which depends on the installation and calibration of the camera.

[0172] According to the pinhole camera model, the point in the world coordinate system , the pixel coordinates projected onto the camera image plane satisfy:

[0173]

[0174] 2. Mathematical expression of joint calibration target

[0175] The main goal of joint calibration is to find the coordinate system from the hybrid lidar To the camera coordinate system. Assume that from the hybrid laser radar The conversion relationship from the coordinate system to the camera coordinate system is:

[0176]

[0177] in, They are the rotation and translation matrices from the hybrid lidar coordinate system to the camera coordinate system.

[0178] 3. Data collection and expression based on feature points

[0179] Collecting images observed by both the camera and the hybrid lidar n t feature points. Let the pixel coordinates of these feature points in the camera image be , the point cloud coordinates in the hybrid lidar are . Through the camera model, from pixel coordinates Reverse its coordinates in the camera coordinate system , according to the inverse process of the camera projection model (assuming the camera intrinsic parameters are known):

[0180]

[0181] 4. Mathematical process of solving transformation relationship

[0182] For hybrid lidar, according to its conversion relationship with the camera

[0183] , and expand it to get:

[0184]

[0185] in, is the rotation matrix elements.

[0186] for n t feature points, we can get 3 n t equation, written in matrix form AX = B , where A is and the identity matrix, X is the coefficient matrix containing and The unknown vector of elements, B is given by The vector composed of . Solve X by the least squares method, that is , get the conversion relationship from hybrid laser radar to camera, and find and .

[0187] Although there are also proposals for automatic external parameter calibration of cameras and laser radars in the prior art, such as the automatic external parameter calibration method and device of cameras and laser radars based on plane matching in announcement number CN118411424A, the problems it has are: 1. The calibration accuracy and reliability are affected in specific scenarios (such as lack of plane features and complex environment), while the joint calibration technology of the present invention and the multiple perception and planning functions in the intelligent loading and unloading system enable it to obtain environmental information and perform operations more accurately in different scenarios (including complex mining scenarios), and have stronger adaptability to various scenarios, improving the problem of insufficient optimization of CN118411424A in specific scenarios. 2. The performance is reduced under complex lighting and weather conditions, while the system design of the present invention can overcome the impact of complex environments on operations to a certain extent through multi-sensor fusion and effective data processing and operation process control, ensuring the stability and accuracy of operations in complex environments, which is better than the performance of CN118411424A in complex environments.

[0188] It should be noted that the autonomous bulk excavation process is as follows Figure 6 As shown in the figure, first search and identify the bulk pile in the environment, and then measure the bulk position Q closest to the unmanned forklift. Then plan the best path for the unmanned forklift to position Q, and adjust the unmanned forklift to approach the position. During the approach process, determine whether the unmanned forklift has reached position Q. If not, continue to adjust the approach. If it has reached, the unmanned forklift adjusts the excavation posture according to the bulk position and performs the excavation action. During the excavation process, determine whether the current loading is completed. If not, continue to excavate. If completed, notify the end of the current loading.

[0189] Specifically, the autonomous bulk excavation process includes the following steps:

[0190] Step 1: Search and identify bulk material piles in the environment

[0191] Step 1: Dataset preparation and model training

[0192] 1. Prepare the "pile" dataset:

[0193] Collect a large number of 2D images containing stockpiles to form an image collection ,and . Label each image, label the material stacking area, and obtain the labeled image set ,and ,in, L N Representing images Use data enhancement technology to expand the data set and obtain an enhanced image set ,and .

[0194] 2. Train the convolutional neural network segmentation model:

[0195] The Yolov8 segmentation model for the stockpile dataset is trained to identify the location and contour of the bulk stockpile. In the process of training the segmentation model, the total loss function of the model is defined by the following formula L tolal :

[0196]

[0197] in, L class is the classification loss, L mask is the mask segmentation loss, L edge It is the mask edge loss, which is used to improve the accuracy of edge prediction.

[0198] Classification Loss L class Used to correctly distinguish the stockpile area from the background, the definition is as follows:

[0199]

[0200] in, t i is the true category label (pile or background), S i is the probability predicted by the model.

[0201] Mask segmentation loss L mask Used to measure the accuracy of the model in pixel-level segmentation, binary cross-entropy loss is used:

[0202]

[0203] in, m j is the real mask (1 represents the stockpile area, 0 represents the background), The mask value predicted by the model;

[0204] Mask edge loss L edge In order to ensure the accuracy of the edge of the stockpile segmentation, a mask edge loss is defined to optimize the clarity and continuity of the segmentation edge. This loss can be achieved through the gradient difference loss:

[0205]

[0206] in, and They are the gradient values ​​of the real mask and the predicted mask at the edge respectively.

[0207] like Figure 7 As shown, by integrating the above partial losses, the total loss function of the model is redefined:

[0208]

[0209] Among them, λ is the weight hyperparameter of the mask edge loss, which is used to balance the impact of various losses.

[0210] Step 2: 2D image segmentation and point cloud filtering

[0211] 1. 2D Image Segmentation:

[0212] For a given two-dimensional image, the trained model is used for segmentation to obtain a segmentation result S, in which each pixel is marked as belonging to the pile or not belonging to the pile.

[0213] 2. Point cloud filtering:

[0214] Assume that the original point cloud set is , filter the point cloud according to the 2D image segmentation result S. For each point in the point cloud , find the corresponding pixel position in the image. If the pixel is marked as not belonging to the stockpile in the segmentation result, remove the point from the point cloud set to obtain a filtered point cloud set. .

[0215] Step 3: Distance calculation

[0216] Calculate distance and set observation points The coordinates are expressed as For the filtered point cloud set Each point in ,and , calculated by the following formula With observation point Distance :

[0217] .

[0218] Then find the point cloud point closest to the observation point ,Right now:

[0219]

[0220] It should be noted that the autonomous unloading process of the present invention is as follows Figure 8 As shown, the unloading operation is performed at the beginning of the process. First, search and identify whether there is a truck in the parking space ahead. If there is no truck, continue searching and identifying; if there is a truck, calculate the unloading position O (select the midpoint of the length of the carriage for the first time), then plan the best path for the unmanned forklift to this position, and then the unmanned vehicle goes to the unloading position O. During the process of going, determine whether the unmanned forklift has reached the unloading position O. If not, continue to go. If it has arrived, the unmanned forklift adjusts the body posture according to the height of the carriage and performs the unloading action. After that, the material level is observed. If the material level reaches the maximum value, determine whether the carriage is full. If not, the material level observation station scans the carriage to update the unloading unloading position O and notifies to continue loading; if the carriage is full, notify that the carriage is full.

[0221] Specifically, searching and identifying whether there is a truck in the parking space ahead includes the following steps:

[0222] Step 1: Data collection and preprocessing

[0223] 1. Assume that the image data set collected by the camera is , where h represents different image frames, and h=1,2,…N. Point cloud data set collected by LiDAR Expressed as , where m represents the index of a point in the point cloud.

[0224] 2. Preprocess the image, including denoising, resizing, etc., to obtain a set of preprocessed images Expressed as . Filter the point cloud to remove noise points and obtain the processed point cloud set Expressed as .

[0225] Step 2: Data Synchronization

[0226] Assume the timestamp of the point cloud is , the timestamp of the image is , the threshold is ,if , it is considered to be the same frame of point cloud and image, and enters the subsequent processing. The filtered point cloud collection Expressed as , the filtered image collection Expressed as .

[0227] Step 3: Image Segmentation

[0228] Use the segmentation model to filter the images Segment the truck in the image and get the segmented mask image M y Assume that the value of each pixel in the mask image indicates whether the pixel belongs to the truck area, a value of 1 indicates that it belongs, and a value of 0 indicates that it does not belong.

[0229] Step 4: Point cloud filtering

[0230] For the filtered point cloud collection Each point in , projecting it onto the image plane to obtain the projected point Expressed as .

[0231] Check Projection Points In the mask picture M y The corresponding pixel value in . If the pixel value is 1, the point cloud point is retained; if it is 0, the point cloud point is filtered out. After filtering, a new point cloud set is obtained It is expressed as: .

[0232] Step 5: Clustering and plane fitting

[0233] 1. Use clustering algorithm to filter the point cloud set Clustering is performed to remove redundant points and only retain the point cloud belonging to the truck. Suppose the truck point cloud set obtained after clustering is .

[0234] 2. Collecting the truck point cloud P truck A largest plane is fitted in the image, and the point cloud at 1 / 2 of the plane is returned. The distance and direction of the truck from the radar are calculated through this point.

[0235] The process of material level observation includes:

[0236] 1. The bottom center of the empty truck compartment is taken as the origin O c , along the length of the carriagex Axis, with the front direction of the vehicle defined as x the positive direction of the axis; establish along the width direction of the carriage y axis, with the left side of the carriage defined as y the positive direction of the axis; establish vertically upward from the bottom surface of the carriage z axis, forming a three-dimensional rectangular coordinate system. Suppose the lidar is at a certain fixed position above the carriage along with the bucket of the unmanned loading vehicle. The lidar obtains the information of each point on the surface of the material in the carriage by emitting laser beams and receiving the reflected signals.

[0237] 2. After unloading once, the lidar scans and measures the material in the carriage, and obtains the coordinate information of a series of discrete points on the surface of the material according to a certain scanning frequency and angular resolution , denoted as , where r = 1, 2…, n r , n r is the number of measurement points. Since we take the center of the carriage bottom as the origin, and the coordinate of the center of the carriage bottom on the z axis is 0, so the height h i of each point on the surface of the material relative to the center of the carriage bottom z can be directly determined by its coordinate value, that is

[0238] The process of judging whether the carriage is full includes:

[0239] 1. Suppose the length, width, and height of the carriage are L , W , H , respectively, then the volume of the carriage is expressed as .

[0240] 2. Suppose the volume of the material when the carriage is full is expressed as , 0 < k ≤ 1, and k is the fullness coefficient.

[0241] 3. In order to more accurately describe the overall height situation of the material in the carriage after reaching the maximum unloading position, considering that the measurement points at different positions are affected by the distance from the lidar and affect the measurement accuracy, we can assign a weight w i to each measurement point, and then calculate the weighted average material height. The points closer to the lidar have larger weights because their measurement accuracy is relatively higher. The calculation formula for the weighted average material height is .

[0242] 4. When the material volume V is calculated based on the material level height, and ,Right now , simplified to , it can be judged that the carriage is full.

[0243] The process of updating the unloading level O by scanning the carriage by the laser radar at the material level observation station includes:

[0244] 1. Find the point with the minimum material height among the scanning points in the truck compartment, and set the initial position of the unloading position O as ,

[0245] 2. The scanning point with the minimum material height in the carriage is recorded as , the number is N min The average value of all their x coordinates is used to update the x coordinate of the unloading position. x new ,Right now ,in, N min is the number of scanning points with the minimum material height. Similarly, update the y coordinate of the discharge position y new ,and The new unloading position for .

[0246] The above solution is only an illustration of a preferred embodiment, but is not limited thereto. When implementing the present invention, appropriate replacement and / or modification can be performed according to user needs.

[0247] Although the embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the specification and the embodiments. It can be fully applied to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily realized. Therefore, without departing from the general concept defined by the claims and equivalent scope, the present invention is not limited to the specific details and the illustrations shown and described here.

Claims

1. An autonomous loading and unloading method for unmanned loading vehicles suitable for bulk material transportation in mining areas, characterized in that: include: S1. Functionally divide the open-pit mining area to obtain the corresponding bulk material storage area, unmanned loading vehicle waiting area, and truck waiting area; S2. Integrate an autonomous loading and unloading system on the unmanned loader to obtain mining area environmental information through the autonomous loading and unloading system, build a global map of the operation area and plan the operation path to complete the loading and unloading operation; Among them, in the autonomous loading and unloading system, the loading and unloading operation logic module is used as the main process, and its working method is: After constructing the global map M of the operation area or dynamically updating the global map M, it is determined in real time whether a loading order from the dispatch center is received. If so, a loading station path to the bulk material storage area is planned and the loading station is reached. Otherwise, the unmanned loader goes to the unmanned loader waiting area. After the unmanned loader arrives at the loading station, it determines in real time whether it has reached the bulk material excavation station. If not, it continues to go there. If it has, it executes the autonomous excavation process and enters the global map M update process after completing the excavation. The unmanned loader plans the path to the truck waiting area and goes to the truck waiting area to perform the autonomous unloading process. If the vehicle is not full, it continues to plan the path to the loading station. If the vehicle is full, it determines whether there is a next vehicle to load materials. If so, it repeats the autonomous unloading process or loading process, otherwise it goes to the unmanned loader waiting area; The mining area environmental information is obtained by a method of joint calibration of dual laser radar and camera, and the process of joint calibration of dual laser radar and camera includes: The point cloud after hybrid laser radar scanning calibration is obtained by calibrating the dual laser radars ; Jointly calibrate the hybrid lidar and camera; When calibrating dual laser radars, one of the laser radars, Lidar1, is used as the fixed coordinate system, and the coordinates of the other laser radar, Lidar2, are converted to Lidar1. After obtaining the homogeneous transformation matrix between Lidar1 and Lidar2, the point cloud Obtained by the following formula: In the above formula, is the unit matrix; The method of joint calibration of hybrid lidar and camera is: Hybrid LiDAR The conversion relationship from the coordinate system to the camera coordinate system is: In the above formula, is the hybrid lidar coordinate system, is the camera coordinate system, They are the rotation matrix and translation matrix from the hybrid lidar coordinate system to the camera coordinate system; Collecting images observed by both the camera and the hybrid lidar n t feature points, and n t The pixel coordinates of the feature points in the camera image are , is the u-axis coordinate of the pixel point on the camera image plane, and the u-axis is horizontal to the right. is the v-axis coordinate of the pixel point on the camera image plane, the v-axis is vertically downward, and the point cloud coordinate in the hybrid lidar is ; From the pixel coordinates, Reverse the coordinates in the camera coordinate system : The conversion relationship between the hybrid laser radar and the camera can be expressed by the following formula to calculate and The value of , completes the joint calibration: In the above formula, is the rotation matrix The values ​​of i and j are 1, 2, and 3 respectively.

2. The autonomous loading and unloading method of an unmanned loader applicable to bulk material transportation in a mining area as claimed in claim 1, characterized in that: The autonomous loading and unloading system comprises: Provides a human-machine interaction interface to support the loading and unloading interaction layer for the operation configuration and control of unmanned loaders; Provides core perception and control algorithms for unmanned loaders and supports the intelligent computing layer for all operations; A functional device layer that interacts with the intelligent computing layer to perform corresponding actions; The environment perception module, the action planning module, and the device driving module are connected to form a loading and unloading operation logic module of the business operation flow and are set on the intelligent computing layer. The intelligent computing layer also includes: The environment perception module is based on the visual equipment and radar on the unmanned loader to obtain environmental and operating target information to complete map construction, autonomous positioning, vehicle posture calculation, bulk material recognition, truck recognition, excavator arm posture calculation, bulk material position calculation, unloading position calculation, and compartment material level calculation; Motion planning module for planning loading path, unloading path, digging arm motion, and autonomous collision avoidance motion to achieve autonomous control; A device driver module that corresponds one-to-one with the underlying device to provide underlying hardware connection and / or driver information.

3. The autonomous loading and unloading method of an unmanned loader applicable to bulk material transportation in a mining area as claimed in claim 1, characterized in that: The process of dynamically updating the global map M includes: Determine whether a map update command is received, and if so, adjust the posture of the unmanned loader to enter the map update mode; The unmanned loader scans the current position to obtain the local point cloud C1, estimates the current position P in the global map M, and searches the global map M for a point cloud with the C1 feature at point P when the map update conditions are met. The point cloud in the global map M that satisfies the following formula is Recorded as : in, , u is the number of points found that meet the conditions, is the jth point in the point cloud subset that meets the conditions, are the coordinates of a point on the map, is the center coordinate of the local point cloud C1, a is the half-side length in the x-axis direction, b is the half-side length in the y-axis direction, and c is the half-side length in the z-axis direction; Execute M = Map(M, C1) to update the global map M.

4. The autonomous loading and unloading method of an unmanned loader applicable to bulk material transportation in a mining area as claimed in claim 3 is characterized in that: The process of executing M = Map(M, C1) includes: Will As the newly acquired point cloud data, in the global map M and Select the initial corresponding point pair set from ,and ,in , K is the number of initial corresponding point pairs, is the distance threshold for feature matching; Based on the initial corresponding point pair set P k , the point cloud subset can be calculated by the following formula The transformation matrix T that matches the points in with the corresponding points in the global map M is: Apply the transformation matrix T to the point cloud subset All points of the transformed point cloud subset are obtained ,and , n represents the number of points in the point cloud after transformation; Re-search in the global map M The nearest point of each point in is used to update the corresponding point pair set. Let the updated corresponding point pair set ,and ,in , L is the number of corresponding point pairs after update; The current matching quality is evaluated by the root mean square error RMSE as follows: , then proceed to the next update operation; otherwise, continue to iterate until the upper limit of the number of iterations is reached : When the matching quality meets the requirements, All the points are added to M to update the global map M.

5. The autonomous loading and unloading method of an unmanned loader applicable to bulk material transportation in a mining area as claimed in claim 1, characterized in that: The process of self-excavation includes: Use the visual equipment and radar on the unmanned loader to search for bulk material piles in the environment; Measure the bulk material position Q closest to the unmanned loading vehicle; Plan the best path for the unmanned loader to the bulk material location Q, and adjust the excavation posture to perform the excavation action after the unmanned loader arrives at Q; The process of searching for bulk material piles includes: Prepare a stockpile dataset and train a convolutional neural network segmentation model; The convolutional neural network segmentation model is used to segment the two-dimensional images obtained by the unmanned loading vehicle; Filter the segmentation results to obtain the filtered point cloud set P f ; For a set observation point, the point cloud set is calculated by the following formula: P f The point cloud point closest to the observation point p closest : In the above formula, O g are the observation point coordinates, P j Point cloud collection P f Every point in .

6. The autonomous loading and unloading method of an unmanned loader applicable to bulk material transportation in a mining area as claimed in claim 5, characterized in that: In the process of training the convolutional neural network segmentation model, the total loss function of the model is defined by the following formula L tolal : In the above formula, L class is the classification loss, L mask is the mask segmentation loss, L edge is the mask edge loss, λ is the weight hyperparameter of the mask edge loss; Among them, the classification loss function L class Obtained by the following formula: In the above formula, t i is the true category label, S i is the probability predicted by the model; Mask segmentation loss L mask Obtained by the following formula: In the above formula, m j is the true mask, The mask value predicted by the model; Mask edge loss L edge Obtained by the following formula: In the above formula, and They are the gradient values ​​of the real mask and the predicted mask at the edge respectively.

7. The autonomous loading and unloading method of an unmanned loader applicable to bulk material transportation in a mining area as claimed in claim 2, characterized in that: The autonomous unloading process includes: The unmanned loading vehicle searches to identify whether there is a truck in the parking space ahead. If there is, the material level in the truck compartment is observed to determine whether the compartment is full. If not, unloading is performed; Among them, when unloading operation is carried out, the unmanned loading vehicle scans the carriage in real time to update the unloading position through material level observation, and then determines whether to stop the unloading operation; The process of searching and identifying whether there is a truck in the parking space ahead includes: The images collected by the camera and the point cloud data collected by the lidar are preprocessed to obtain the corresponding image sets. I processed , Point Cloud Collection P filtered ; Image collection based on timestamp I processed , Point Cloud Collection P filtered Perform data synchronization to obtain the filtered image collection I selected , Point Cloud Collection P selected ; Use the segmentation model to filter the image set I selected Segment the truck in the image and get the segmented mask image M y ; Based on the filtered point cloud collection P selected Each point in In the mask picture M y Is there a corresponding pixel value in the point cloud to complete the point cloud filtering and obtain the filtered point cloud set P filtered2 ; Use clustering algorithm to filter the point cloud collection P filtered2 Clustering is performed and the point cloud collection after clustering is P truck A maximum plane is fitted in the image, and then the point cloud at 1 / 2 of the maximum plane is returned to calculate the distance and direction of the truck from the radar.

8. The autonomous loading and unloading method of an unmanned loader applicable to bulk material transportation in a mining area as claimed in claim 7, characterized in that: The process of material level observation includes: The origin is the bottom center of the empty truck compartment. O c , establish a three-dimensional rectangular coordinate system along the length, width and height directions of the carriage; Assume that the laser radar is at a fixed position above the carriage with the bucket of the unmanned loader. After each unloading, the laser radar scans and measures the material in the carriage and obtains the coordinate information of a series of discrete points on the surface of the material, which is recorded as ,in, r =1,2…, n r , n r is the number of measurement points, then the height of each point on the material surface relative to the center of the bottom of the carriage h i It can be expressed by the following formula: ； The process of determining whether the carriage is full is: Assign a weight to each measurement point in the cabin collected by the LiDAR w i , to calculate the weighted average material height by the following formula h wavg : The fullness of the carriage can be determined by the following formula h wavg Make a judgment: In the above formula, k is the filling coefficient and H is the height of the carriage.

9. The autonomous loading and unloading method of an unmanned loader applicable to bulk material transportation in a mining area as claimed in claim 7, characterized in that: The process of updating the discharge level through material level observation is as follows: Assume that the initial position O of the unloading position of the carriage is expressed as ; Find the point with the minimum material height among the scanned points in the truck compartment, recorded as , through the following formula The average value of all x-coordinates and y-coordinates in are used as the x-coordinate and y-coordinate of the updated unloading position: ， In the above formula, N min is the number of scan points with minimum material height, x new To update the x coordinate of the discharge position, y new To update the y coordinate of the discharge position, h min is the minimum material height, x ij For all x-coordinates with the minimum material height, y ij for all y coordinates with minimum material height; The new unloading position for .

Images