Unmanned mine card online scheduling method based on hierarchical reinforcement learning

Abstract

The invention discloses an unmanned mine card online scheduling method based on hierarchical reinforcement learning, and the method comprises the steps: determining a destination through employing a lower-layer strategy according to the lower-layer state of an unmanned mine card; determining an upper-layer state of the unmanned mine card based on the destination, and determining a driving path by using an upper-layer strategy according to the upper-layer state of the unmanned mine card; and carrying out online scheduling on the unmanned mine cards according to the driving path. The scheduling decision is carried out in combination with hierarchical reinforcement learning, the method adapts to complex and dynamic operation environment characteristics of the strip mine, greenhouse gas emission is reduced, and production efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of mining truck scheduling, and more specifically to an unmanned mining truck online scheduling method based on hierarchical reinforcement learning. Background Art

[0002] The extremely high demand for fuel and electricity during the mining, processing and transportation of minerals has led to related greenhouse gas emissions accounting for about 10% of the total global emissions. In open-pit mining operations, a major source of greenhouse gas emissions is the fuel consumption of open-pit mining trucks (hereinafter referred to as mining trucks), which accounts for about 30% of the total energy consumption of open-pit mines. At the same time, the operating costs of mining trucks account for nearly 50% of the total production costs of open-pit mines. Reasonable mining truck scheduling methods will improve transportation efficiency, reduce mining truck operating costs, and reduce production emissions. Therefore, research on mining truck scheduling methods to optimize mining truck scheduling is of great significance to reducing costs, increasing efficiency and reducing emissions in open-pit mine production.

[0003] Open-pit mining operations are transforming towards unmanned and intelligent operations. Unmanned mining trucks are being used in open-pit mines, and unmanned transportation is becoming the future development model for open-pit mine transportation systems.

[0004] Considering the limitations of current driverless technology, for safety reasons, only one unmanned mining truck is allowed to pass through each intersection at the same time. This may cause the unmanned mining trucks to queue up at the intersection, resulting in reduced production efficiency and additional fuel consumption emissions.

[0005] At the same time, the mining environment is complex and abnormal events occur frequently. The traditional static scheduling method of determining the scheduling plan before the shift is prone to failure in the dynamically changing operating environment, and the commonly used mathematical programming method faces a bottleneck in the solution rate under the ever-expanding scale of open-pit mining operations.

[0006] At present, the research on mine truck scheduling is still mainly focused on manned mine truck scheduling. The research on unmanned mine truck scheduling has failed to fully combine its operating characteristics, and often ignores the impact of the highly dynamic operating environment of open-pit mines on the unmanned mine trucks' execution of transportation tasks. Moreover, most of the research focuses on single-path scheduling, and there is little research on online scheduling of unmanned mine trucks under multi-paths.

[0007] Therefore, how to fully tap the massive operating data generated by the intelligent and unmanned development of open-pit mines and implement scheduling methods that conform to the operating characteristics of unmanned mine trucks and production realities has become a key issue that technical personnel in this field urgently need to solve. Summary of the invention

[0008] In view of this, in order to at least partially solve the above technical problems, the present invention provides an online scheduling method for unmanned mining trucks based on hierarchical reinforcement learning.

[0009] In order to achieve the above object, the present invention adopts the following technical solution:

[0010] An online scheduling method for unmanned mining trucks based on hierarchical reinforcement learning, comprising:

[0011] Use the lower-level strategy to determine the destination based on the lower-level status of the unmanned mining truck;

[0012] Determine the upper state of the unmanned mining truck based on the destination, and use the upper strategy to determine the driving path according to the upper state of the unmanned mining truck;

[0013] Online dispatch of unmanned mining trucks based on driving routes.

[0014] As a preference, the lower-level status of the unmanned mining truck includes:

[0015] Estimated travel time from current location to destination ij , including driving time and waiting time;

[0016] Estimated service time m i ;

[0017] Destination target output completion rate c it ;

[0018] The expected number of vehicles receiving destination service q it .

[0019] Preferably, determining a lower-level action space and a lower-level reward function for reinforcement learning of the lower-level strategy;

[0020] The lower action space includes all loading and unloading points in the target area of ​​the open-pit mine;

[0021] The lower-level reward function includes waiting time reward and transportation volume reward, where the waiting time reward is given after completing each transportation task, and the transportation volume reward is given after the complete operation cycle. The expression is:

[0022]

[0023]

[0024] In the formula, represents the waiting time of the unmanned mining truck on the way to destination i, δ represents whether destination i is the destination with the largest number of service vehicles, if so, the value is 1, otherwise it is 0, M is an infinitesimal negative number, α is an adjustment parameter, which is determined according to the actual production situation of the mining area to ensure The value is between 0.5 and 1. Using model training, λ represents the correlation coefficient with the target production completion of each destination, p f Indicates the transportation volume of unmanned mining trucks.

[0025] Preferably, according to the operation requirements of the unmanned mining vehicle, the Q value of the inactive action is set to an infinitesimal negative number through the action mask so that the decision ignores the inactive action.

[0026] As a preference, the upper-level status of the unmanned mining truck includes:

[0027] Estimated travel time d for each path from the current location of the unmanned mining truck to the destination ijk ;

[0028] The number of unmanned mining trucks on each path q jtk ;

[0029] Number of unmanned mining trucks at the intersection n jtk .

[0030] Preferably, determining an upper-level action space and an upper-level reward function for reinforcement learning of the upper-level strategy;

[0031] The upper action space includes all schedulable paths from the current position of the unmanned mining truck to the destination;

[0032] The upper reward function takes into account the fuel consumption cost setting, and the expression is:

[0033]

[0034] In the formula, pow avg It indicates the average fuel consumption of the unmanned mining truck to complete the journey at the preset speed without congestion. real Indicates the actual fuel consumption for the journey completed.

[0035] Preferably, when the unmanned mining truck to be scheduled sends a scheduling request at its current location, the lower state of the unmanned mining truck is obtained through a real-time database, and the upper state is determined based on the destination; the real-time database includes:

[0036] Open pit road network data: road information between loading and unloading points in the target area of ​​the open pit, as well as the topology, distance and accessibility of the actual road network including intersections;

[0037] Open-pit mine operating equipment data: the number of fixed and mobile operating equipment, the location of fixed operating equipment, the maximum service capacity of fixed operating equipment, the operating time of unmanned mining trucks at loading and unloading points, the speed limit of unmanned mining trucks on various sections of the road, and the fuel consumption rate of unmanned mining trucks under different working conditions;

[0038] Open-pit mine unmanned mining truck operation data: the current operation information of the unmanned mining truck, including the operating position, working status and operating speed.

[0039] As a preferred method, after the unmanned mining truck is scheduled online, the completed quantity at the destination is updated.

[0040] Preferably, during online scheduling, a first-come, first-served rule is used to resolve conflicts.

[0041] As a preferred method, the upper-level strategy and the lower-level strategy are trained by strategy sharing and experience sharing, and the indicator output is used as the judgment indicator of convergence.

[0042] It can be seen from the above technical solutions that the present invention discloses an unmanned mining truck online scheduling method based on hierarchical reinforcement learning. Compared with the prior art,

[0043] The present invention constructs a real-time data-driven scheduling decision framework based on a hierarchical reinforcement learning model: the lower-level strategy is responsible for selecting the next destination for the unmanned mining truck currently requesting scheduling, and the upper-level strategy is responsible for specific path planning based on the destination selection result.

[0044] By combining the operating characteristics of unmanned mining trucks and accurately depicting the queues at intersections, multi-path online scheduling of unmanned mining trucks can be achieved, reducing the efficiency reduction and additional emissions caused by queuing at intersections, which not only reduces the greenhouse gas emissions of unmanned mining truck operations, but also improves the production efficiency of unmanned mining trucks; at the same time, it reduces the fuel consumption of mining trucks and reduces operating costs, providing technical support and engineering guidance for the safe, continuous, efficient and low-carbon operation of unmanned transportation systems in open-pit mines. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0046] Figure 1 A schematic flow chart of an online scheduling method for unmanned mining trucks based on hierarchical reinforcement learning provided in an embodiment of the present invention.

[0047] Figure 2 A schematic diagram of an online scheduling decision framework for unmanned mining trucks based on hierarchical reinforcement learning provided in an embodiment of the present invention.

[0048] Figure 3 This is a decision block diagram for online scheduling of unmanned mining trucks provided in an embodiment of the present invention.

[0049] Figure 4 Provides a training curve diagram of a multi-agent hierarchical reinforcement learning model in different production scales for an embodiment of the present invention.

[0050] Figure 5A schematic diagram of an electronic map of an open-pit mine provided in an embodiment of the present invention.

[0051] Figure 6 A schematic diagram comparing output and greenhouse gas emissions per unit output under different scheduling strategies provided in an embodiment of the present invention.

[0052] Figure 7 A schematic diagram comparing additional greenhouse gas emissions caused by start-stop and idle fuel consumption under different scheduling strategies provided in an embodiment of the present invention.

[0053] Figure 8 A schematic diagram of the robustness of the proposed online scheduling method under fleet size fluctuations in different production scales provided in an embodiment of the present invention.

[0054] Fig. 9 A Gantt chart of mining truck tasks without considering intersection congestion and considering intersection congestion is provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0055] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] See also Figure 1 As shown, an embodiment of the present invention provides a flowchart of an online scheduling method for unmanned mining trucks based on hierarchical reinforcement learning, which specifically includes the following steps:

[0057] Use the lower-level strategy to determine the destination based on the lower-level status of the unmanned mining truck;

[0058] Determine the upper state of the unmanned mining truck based on the destination, and use the upper strategy to determine the driving path according to the upper state of the unmanned mining truck;

[0059] Online dispatch of unmanned mining trucks based on driving routes.

[0060] The goal of the present invention is to realize an online scheduling method for unmanned mining trucks that can take into account dynamic operation requirements. In order to make the scheduling decision closer to the actual production of unmanned mining truck transportation operations, the present invention considers the queuing waiting at the intersection and the fluctuation of operation parameters and the change of fleet size during the production process.

[0061] Specifically for the situation where there are multiple paths between a pair of loading points and unloading points in open-pit mines, the present invention designs a scheduling decision framework based on hierarchical reinforcement learning, which can make destination and specific path scheduling decisions at the same time. The developed scheduling algorithm can meet the time requirements of online scheduling and can improve the output within the operation cycle and reduce greenhouse gas emissions.

[0062] In one embodiment:

[0063] The lower state of the unmanned mining card is S l express, Indicates the lower-level status of the unmanned mining card when it requests scheduling at time t, including:

[0064] (1) The current time, i.e., the time t when the unmanned mining truck requests scheduling during online scheduling;

[0065] (2) Estimated travel time d from current location to destination ij , including driving time and waiting time;

[0066] The estimated travel time of an unmanned mining truck from its current location i to the destination j for scheduling selection consists of two parts: one is the driving time consumed to the destination under the speed limit of the mining area road section, and the other is the waiting time consumed at each intersection on the way and waiting for service after arriving at the destination. Since the waiting time fluctuates due to the congestion of the road network, the travel time is estimated based on real-time data, specifically taking the average travel time of all previous mining trucks to complete the trip.

[0067] (3) Estimated service time m i ;

[0068] The time that the unmanned mining truck receives service at the destination i for scheduling selection refers to the time of loading materials at the loading point or the time of unloading materials at the unloading point. Since the operating parameters of fixed operating equipment have certain fluctuations, the service time in this implementation is read based on the real-time status data of the operating equipment.

[0069] (4) Destination target output completion rate c it ; Refers to the target production completion rate of all destinations i selected for dispatch when the unmanned mining truck requests dispatch. The calculation method is:

[0070]

[0071] Among them, the destination target output is set in advance by the daily plan of the open-pit mine, and the completed output at the destination is continuously updated by the real-time status data of the operating equipment.

[0072] (5) The expected number of vehicles receiving destination service q it; Refers to the number of unmanned mining trucks that have been assigned to receive services at destination i when the unmanned mining truck requests dispatch, including those that are heading to the destination but have not yet arrived, and those that have arrived at the destination and are waiting in line to receive services.

[0073] In one embodiment:

[0074] Determine the lower-level action space and the lower-level reward function for reinforcement learning of the lower-level strategy;

[0075] Furthermore, the lower action space includes all loading and unloading points in the target area of ​​the open-pit mine;

[0076] In this embodiment, the lower action space is represented by A l Indicates that it contains all possible actions of the unmanned mining truck. The definition is as follows: The unmanned mining truck goes to destination i. The unmanned mining truck transports materials between loading and unloading points, so all loading and unloading points constitute its action space.

[0077] In a preferred embodiment, the specific value of the state space of the unmanned mining truck is related to the position at which it requests scheduling, that is, in a specific state, certain actions are not allowed. Therefore, in order to deal with the variable action space brought about by the operation requirements of the unmanned mining truck, the present application uses an action mask to set the Q value of the inaction action to an infinitesimal negative number according to the operation requirements of the unmanned mining truck, so that the decision ignores the inaction action and only selects from the actionable actions.

[0078] In a specific embodiment, when the unmanned mining truck is at a loading point, since its next destination must be the unloading point, the information related to the loading point will be masked to 0, and vice versa.

[0079] Furthermore, the lower layer reward function is composed of R l Indicates that rewards are distributed in a decentralized manner, including waiting time rewards and transportation volume rewards.

[0080] When the unmanned mining truck f arrives at the destination i specified by the scheduling decision, it will receive a basic waiting time reward:

[0081]

[0082] In the formula, It represents the waiting time of the unmanned mining truck on the way to destination i, including the waiting time in the intersection and the destination. It is normalized by exponential function to limit the reward value between 0 and 1, which is convenient for model training. δ represents whether destination i is the destination with the largest number of expected service vehicles. If so, it is 1, otherwise it is 0. M is an infinitesimal negative number. α is an adjustment parameter, which is set according to the actual production situation of the mining area to ensure The value is between 0.5 and 1. Using model training, δM jointly indicates that if the unmanned mining truck goes to the destination that is expected to serve the largest number of vehicles at the decision time, it will be subject to additional penalties. This is because the excessive concentration of unmanned mining trucks at a single destination will cause road network congestion.

[0083] After a full work shift, the unmanned mining trucks will be assigned a material transport volume p f Give transportation volume rewards

[0084]

[0085] In the formula, λ represents the correlation coefficient with the target production completion of each destination. When all destinations reach the target production, the value is 1, and p f Indicates the transportation volume of unmanned mining trucks.

[0086] All unmanned mining trucks share a unified λ value, which encourages collaboration among unmanned mining trucks during scheduling decisions to avoid congestion and maximize output while meeting production targets.

[0087] In one embodiment:

[0088] The upper-level strategy is responsible for further selecting the specific path to the destination based on the destination selection result of the lower-level strategy. In this embodiment, the upper-level state of the unmanned mining truck is represented by S u Indicates, including:

[0089] (1) Current location and next destination: The current location i and the next destination j determined by the lower-level strategy.

[0090] (2) Estimated travel time d for each path from the current location of the unmanned mining truck to the destination ijk That is, the time required for the unmanned mining truck to travel from the current position i to the destination j determined by the lower-level strategy through each scheduling path k is the same as the estimated travel time in the lower-level strategy, specifically, the average travel time of all previous mining trucks completing the journey through this path.

[0091] (3) The number of unmanned mining trucks on the path to be dispatched q jtk ; refers to the number of unmanned mining trucks currently traveling on each path k leading to the destination j determined by the lower-level strategy when the unmanned mining truck requests scheduling.

[0092] (4) Number of unmanned mining trucks at the intersection n jtk ; refers to the sum of the number of all unmanned mining trucks related to the intersection on the path k leading to the destination j determined by the lower-level strategy when the unmanned mining truck requests scheduling. An unmanned mining truck related to an intersection means that it is currently waiting to pass through the intersection or the next travel node is the intersection.

[0093] In one embodiment:

[0094] Determine the upper-level action space and upper-level reward function for reinforcement learning of the upper-level strategy;

[0095] In this embodiment, the upper action space is represented by A u Indicates all schedulable paths from the current location of the unmanned mining truck to the destination;

[0096] The upper layer reward function is composed of R u It means that in a decentralized manner, when an unmanned mining truck reaches the destination via the decided path, it will receive the following rewards:

[0097]

[0098] In the formula, pow avg It indicates the average fuel consumption required for the unmanned mining truck to complete the trip at the preset speed without congestion. real Indicates the actual fuel consumption of completing the trip. The specific fuel consumption can be obtained through the data of the unmanned mining truck. Compared with the lower-level reward function, the upper-level reward function focuses on the fuel consumption cost, which is conducive to selecting a path with low fuel consumption cost in actual production.

[0099] In one embodiment,

[0100] The lower state of the unmanned mining truck is obtained through a real-time database, and the upper state is determined based on the destination; the real-time database includes:

[0101] Open pit road network data: road information between loading and unloading points in the target area of ​​the open pit, as well as the topology, distance and accessibility of the actual road network including intersections;

[0102] Open-pit mine operating equipment data: the number of fixed and mobile operating equipment, the location of fixed operating equipment, the maximum service capacity of fixed operating equipment, the operating time of unmanned mining trucks at loading and unloading points, the speed limit of unmanned mining trucks on various sections of the road, and the fuel consumption rate of unmanned mining trucks under different working conditions;

[0103] Open-pit mine unmanned mining truck operation data: the current operation information of the unmanned mining truck, including the operating position, working status and operating speed.

[0104] In an exemplary embodiment:

[0105] For the online dispatching method of unmanned mining trucks, refer to Figure 2 and Figure 3 ,include:

[0106] S1. The unmanned mining truck to be dispatched sends a dispatch request at its current location to obtain the location information of the unmanned mining truck to be dispatched, the current fuel consumption data and the locations of all unmanned mining trucks in the road network in the real-time database;

[0107] S2. Calculate the number of vehicles waiting in line at each loading point, unloading point, and intersection based on the road network topology and the locations of each loading point, unloading point, and intersection, and confirm the locations available for dispatch;

[0108] S3. Calculate the upper state space of the unmanned mining truck to be scheduled based on the destination and other operational data obtained by the lower-level strategy decision, and call the trained upper-level strategy decision to obtain the path;

[0109] S4. Send the "destination + path" obtained by the decision to the unmanned mining card to be dispatched. When the unmanned mining card completes this dispatch task, the destination has been updated and the completed production, and the relevant data is stored in the database;

[0110] Furthermore, it also includes S5, within a work shift, the above process is continuously repeated; after the work shift is completed, the lower-level strategy and the upper-level strategy are further trained according to all scheduling data in combination with actual production needs.

[0111] In one embodiment, the upper-layer strategy and lower-layer strategy are trained by strategy sharing and experience sharing. During training, all unmanned mining trucks share the same scheduling strategy to reduce instability during training; all unmanned mining trucks share the experience of interacting with the environment, and each unmanned mining truck can learn from the experience of other unmanned mining trucks to accelerate the training process. In order to intuitively show whether the training has converged, the output indicator that is often concerned in actual production is used as a judgment indicator of convergence.

[0112] In one embodiment, the first-come, first-served rule is used to resolve conflicts, that is, the mining truck that arrives at the intersection first will have priority to pass through the intersection; combining the FIFO rule to resolve conflicts means that when the system processes mining trucks at the intersection, it will determine their passing order based on the time sequence of the mining trucks arriving at the intersection. If multiple mining trucks arrive at the same time, the system will arrange their passing order according to the first-come, first-served principle. This method is simple and fair, can ensure traffic order at the intersection, reduce waiting time, avoid congestion, and also improve the efficiency and safety of the entire mining truck operation system.

[0113] The model training curves under different production scale instances are as follows: Figure 4As shown, it can be seen that all production scales eventually converge. Among them, different production scales are distinguished by "number of loading points × number of unloading points × number of unmanned mining trucks". In this embodiment, the production scale in (a) is 8×4×60, the production scale in (b) is 10×5×75, the production scale in (c) is 12×6×90, and the production scale in (d) is 14×7×105; As can be seen from the figure, for larger-scale instances, longer training time is required, which can be solved in practical applications by loading pre-trained models.

[0114] The method of the present invention is applicable to open-pit mines using unmanned mining trucks; a specific introduction is given below using an open-pit mine in China as an example.

[0115] The embodiment of the present invention takes the actual production of an open-pit mine in Northeast China as an example. The electronic map of the open-pit mine is as follows: Figure 5 As shown, an example is given to verify the dynamic scheduling of unmanned mining trucks.

[0116] The open-pit mine has 1 parking lot, 19 loading points (12 rock stripping shovels and 7 coal mining shovels), 7 unloading points (4 spoil dumps, 1 rock crushing station, 1 coal crushing station), and 47 intersections. The data of operating facilities in the target area of ​​the open-pit mine are shown in Table 1.

[0117] Table 1 Data of operating facilities in the target area of ​​the open-pit mine

[0118] equipment Loading / unloading time (minutes) Maximum number of vehicles allowed to operate Excavation area 1-2 (quarrying shovel) 4 1 Excavation area 3-12 (electric shovel for mining) 7 1 Excavation area 13-19 (coal mining shovel) 5.6 1 Unloading area 1 (rock field) 1.5 10 Unloading area 2-5 (dumping area) 1.5 1 Unloading area 6-7 (coal yard) 1.5 1

[0119] At the same time, there are 105 unmanned mining trucks in the open-pit mine that can be used for scheduling. The operating parameters of the unmanned mining trucks are shown in Table 2, including vehicle speed, driving fuel consumption, idling fuel consumption, start-stop additional fuel consumption, and loading capacity. In this embodiment, mining trucks within a shift (8 hours) are scheduled online.

[0120] Table 2 Open-pit mine autonomous driving truck operation data

[0121]

[0122]

[0123] The target output of each loading / unloading point in the open-pit mine, the speed limit and travel time of each section in the road network, etc. are derived from actual production data. Due to the huge number and the fact that they vary from open-pit mine to open-pit mine, the present invention will not list them again.

[0124] In order to verify the applicability of the proposed scheduling method under different operation scales in open-pit mines, four production scales were constructed based on the above data in combination with actual production for example verification.

[0125] First, in the embodiment of the present invention, Python is used to write a scheduling optimization program. The online scheduling method proposed in the embodiment is compared with the output and greenhouse gas emissions per unit output of different scheduling strategies. Figure 6 The benchmark comparison strategies are two heuristic scheduling strategies currently commonly used in the industry: minimum idle time and minimum queue time. The proposed scheduling method increases output by 8% and reduces greenhouse gas emissions per unit output by 5%.

[0126] For a comparison of the additional greenhouse gas emissions caused by the online scheduling method proposed in the embodiment and the start-stop and idle fuel consumption of different scheduling strategies, see Figure 7 , the proposed scheduling method reduces the additional greenhouse gas emissions caused by start-stop and idle fuel consumption by 7.7% on average.

[0127] This is because the proposed scheduling method can consider the scheduling of unmanned mining trucks from a global perspective, coordinate the operation of unmanned mining trucks and reduce the occurrence of queue congestion, thereby improving the utilization rate of the fleet and reducing unnecessary greenhouse gas emissions. The performance advantage of the proposed scheduling method is more obvious under large-scale operations. This is because the road network structure under large-scale operations is more complex, and the uncertainty caused by queue congestion at intersections is aggravated. The proposed scheduling method is based on the characteristics of reinforcement learning and strong interaction with the environment, and can better cope with dynamic operation needs.

[0128] In actual production of open-pit mines, there are often fluctuations in the size of the fleet available for dispatch due to maintenance of unmanned mining trucks and the introduction of new equipment. The embodiment of the present invention analyzes the robustness of the proposed dispatching method under fluctuations in the size of the fleet. Figure 8 As shown, the vertical axis represents the improvement of the proposed scheduling method over the baseline strategy. The proposed scheduling method is trained at a fixed fleet size corresponding to the gray line, and tested at different fleet sizes (fluctuating within a 10% range). Among them, the scale in (a) is 8×4×60, the scale in (b) is 10×5×75, the scale in (c) is 12×6×90, and the scale in (d) is 14×7×105;

[0129] The results show that the proposed scheduling method is still generally better than the baseline strategy under the fleet size and has good robustness. Compared with the small-scale open-pit mine operation scale which is more sensitive to the fleet size, the proposed scheduling method has better performance under large-scale fleet operation, which shows that the proposed scheduling method can not only adapt to the dynamically changing operating environment of open-pit mines, but also provide stable and reliable online scheduling for large-scale operations.

[0130] Further analysis is conducted to determine whether the impact of intersection congestion on unmanned mining truck scheduling is considered. The queue waiting events in the scheduling results under different scheduling schemes are displayed in the form of a Gantt chart, such as Fig. 9As shown in Figure 2, (a) does not consider intersection congestion, and (b) considers intersection congestion. It is easy to observe that if the scheduling strategy does not consider intersection congestion, the unmanned mine will spend a lot of waiting time at the intersection, such as Fig. 9 The black block in (a). On the contrary, if the intersection congestion is taken into consideration when dispatching the unmanned mining truck, the waiting time of the unmanned mining truck at the intersection will be reduced. This is because ignoring the intersection congestion will cause the dispatch decision to deviate from the actual situation. Fig. 9 Taking the beginning of the middle work shift (specifically between 8:00 and 9:00) as an example, regardless of whether the scheduling results take into account intersection congestion, significant queuing phenomena appear at the intersection. This congestion phenomenon occurs because the unmanned mining trucks all depart from the same parking lot at the beginning of the work shift. Although the destinations of the unmanned mining trucks are different, they must all pass through the intersection connecting the parking lot, resulting in queuing and waiting at the intersection. Similar queuing phenomena also occur at key intersections with multiple intersections, and it is necessary to take this into consideration during scheduling. The present invention combines the first-come-first-served rule to resolve conflicts, fully considering the actual situation of congestion at open-pit mine intersections, and making scheduling more in line with production reality.

[0131] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0132] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An online scheduling method for unmanned mining trucks based on hierarchical reinforcement learning, characterized in that: Use the lower-level strategy to determine the destination based on the lower-level status of the unmanned mining truck; Determine the upper state of the unmanned mining truck based on the destination, and use the upper strategy to determine the driving path according to the upper state of the unmanned mining truck; Online dispatch of unmanned mining trucks based on driving routes.

2. The unmanned mining truck online scheduling method according to claim 1 is characterized in that: The lower-level status of the unmanned mining card includes: The estimated driving time from the current location to the destination, including driving time and waiting time; Estimated service time; Destination target production completion rate; The estimated number of vehicles to be served at the destination.

3. The unmanned mining truck online scheduling method according to claim 1 is characterized in that: Determine the lower-level action space and the lower-level reward function for reinforcement learning of the lower-level strategy; The lower action space includes all loading and unloading points in the target area of ​​the open-pit mine; The lower layer reward function includes waiting time reward and transportation volume reward, and the expression is: In the formula, represents the waiting time of the unmanned mining truck on the way to destination i, δ represents whether destination i is the destination with the largest number of expected service vehicles, if so, the value is 1, otherwise it is 0, M is an infinitesimal negative number, α is the adjustment parameter, λ represents the correlation coefficient with the target production completion of each destination, p f Indicates the transportation volume of unmanned mining trucks.

4. The unmanned mining truck online scheduling method according to claim 3 is characterized in that: According to the operation requirements of the unmanned mining vehicle, the Q value of the inactive action is set to an infinitesimal negative number through the action mask so that the decision-making ignores the inactive action.

5. The unmanned mining truck online scheduling method according to claim 1 is characterized in that: The upper-level status of the unmanned mining card includes: The estimated travel time for each route from the current location of the unmanned mining truck to the destination; The number of unmanned mining trucks on each path; Number of unmanned mining trucks at the intersection.

6. The unmanned mining truck online scheduling method according to claim 1 is characterized in that: Determine the upper-level action space and upper-level reward function for reinforcement learning of the upper-level strategy; The upper action space includes all schedulable paths from the current position of the unmanned mining truck to the destination; The upper reward function takes into account the fuel consumption cost setting, and the expression is: In the formula, powr avg It indicates the average fuel consumption of an unmanned mining truck when completing a trip at a preset speed without congestion. eal Indicates the actual fuel consumption for the journey completed.

7. The unmanned mining truck online scheduling method according to claim 1 is characterized in that: When the unmanned mining truck to be dispatched sends a dispatch request at its current location, the lower state of the unmanned mining truck is obtained through the real-time database, and the upper state is determined based on the destination; the real-time database includes: Open pit road network data: road information between loading and unloading points in the target area of ​​the open pit, as well as the topology, distance and accessibility of the actual road network including intersections; Open-pit mine operating equipment data: the number of fixed and mobile operating equipment, the location of fixed operating equipment, the maximum service capacity of fixed operating equipment, the operating time of unmanned mining trucks at loading and unloading points, the speed limit of unmanned mining trucks on various sections of the road, and the fuel consumption rate of unmanned mining trucks under different working conditions; Open-pit mine unmanned mining truck operation data: the current operation information of the unmanned mining truck, including the operating position, working status and operating speed.

8. The unmanned mining truck online dispatching method according to claim 7 is characterized in that: After the unmanned mining truck is scheduled online, the completed quantity at the destination is updated.

9. The unmanned mining truck online scheduling method according to claim 1 is characterized in that: During online scheduling, the first-come-first-served rule is used to resolve conflicts.

10. The unmanned mining truck online scheduling method according to claim 1, characterized in that: The upper-level and lower-level strategies are trained by strategy sharing and experience sharing, and the indicator output is used as the convergence criterion.

Images