Ai-based control system integrated with a haul truck that optimizes energy efficiency for mining routes

Abstract

Systems and methods for smart integrated energy management of a haul truck fleet include obtaining operational data at a base station (125), wherein the operational data comprises one or more of haul truck telemetry data from a fleet of haul trucks (105, 107), route image data, environment data, haul load efficiency data, and payload data. The methods further include obtaining a mine plan of a mine, and determining, using an Al and Autonomy System comprised of Al models and connected to the base station (125), operational parameters selected from the following group: a CO2 emission, a productivity, an energy use, a haul truck velocity, and a haul truck acceleration, given the operational data. Furthermore, the methods include adjusting the operational parameters of at least one haul truck in the fleet of haul trucks (105, 107) and one or more haul trucks in operation.

Description

AI-BASED CONTROL SYSTEM INTEGRATED WITH A HAUL TRUCK THAT OPTIMIZES ENERGY EFFICIENCY FOR MINING ROUTESBACKGROUND

[0001] The future of energy management in mining operations relies heavily on integrating Autonomy and Artificial Intelligence (Al). While autonomy has already proven beneficial in certain mining processes, its capabilities are limited in the broader mineral logistics space. Al is prepared to enhance machine control systems significantly, potentially increasing equipment efficiency when paired with autonomous technologies. The operational complexities of haul trucks in open-pit mines, such as managing energy availability, road conditions, dynamic environmental changes, and variations in load and grade, demand sophisticated Al solutions. These Al systems are expected to utilize a hybrid model approach, incorporating neural networks essential for adapting to the ever-changing mining environment. This adaptation includes optimizing truck speeds, payloads, routes, and the number of active trucks to minimize energy per tonne moved while achieving target production rates.

[0002] Accordingly, there exists a need for the integration of these Al systems with current mine planning and management software to enhance productivity, reduce CO2 emissions, and implement a centralized, efficient energy management system.SUMMARY

[0003] This summary is provided to introduce a selection of concepts that are further described below in the detailed description. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used as an aid in limiting the scope of the claimed subject matter.

[0004] In general, in one aspect, embodiments are disclosed related to methods for Al & smart systems with smart integrated energy management for a haul truck fleet. The methods include obtaining operational data at a base station, wherein the operational data comprises one or more selected from the following group: haul truck telemetry data from a fleet of haul trucks, route image data, environment data, haul load efficiency data, and payload data. The methods further include obtaining a mine plan of a mine,the mine plan comprising mine plan targets and route constraints describing a set of haul routes at the mine, and determining, using an Al and Autonomy System comprised of Al models and connected to a base station, operational parameters selected from the following group: a CO2 emission, a productivity, an energy use, a haul truck velocity, and a haul truck acceleration, given the operational data. Furthermore, the methods include adjusting, automatically and according to the mine plan and the mine plan targets, the operational parameters of at least one haul truck in the fleet of haul trucks and one or more haul trucks in operation.

[0005] In general, in one aspect, embodiments are disclosed related to a system for Al & smart systems with smart integrated energy management for a haul truck fleet. The system includes a base station, configured to be connected to an Al and Autonomy System and to receive operational data from sensors; the sensors, configured to record operational data comprising one or more selected from the following group: haul truck telemetry data from a fleet of haul trucks, route image data, environment data, haul load efficiency data, and payload data; a fleet of haul trucks, configured to record telemetry data; and the Al and Autonomy System, comprised of Al models, configured to: determine operational parameters selected from the following group: a CO2 emission, a productivity, an energy use, a haul truck velocity, and a haul truck acceleration, given the operational data, and adjust, automatically and according to a mine plan and mine plan targets, the operational parameters of at least one haul truck in the fleet of haul trucks and a number of haul trucks in operation.

[0006] Other aspects and advantages of the claimed subject matter will be apparent from the following description and the appended claims.BRIEF DESCRIPTION OF DRAWINGS

[0007] FIG. 1A depicts a mine environment in accordance with one or more embodiments.

[0008] FIG. IB depicts the operation of a system in accordance with one or more embodiments.

[0009] FIG. 1C depicts the architecture of a system in accordance with one or more embodiments.

[0010] FIG. ID depicts the architecture of a model in accordance with one or more embodiments.

[0011] FIG. IE depicts the architecture of a model in accordance with one or more embodiments.

[0012] FIG. IF depicts the architecture of a model in accordance with one or more embodiments.

[0013] FIG. 1G depicts the architecture of a model in accordance with one or more embodiments.

[0014] FIG. 1H depicts the architecture of a model in accordance with one or more embodiments.

[0015] FIG. II depicts the architecture of a model in accordance with one or more embodiments.

[0016] FIG. 1J depicts the architecture of a model in accordance with one or more embodiments.

[0017] FIG. IK depicts the architecture of a model in accordance with one or more embodiments.

[0018] FIG. IL depicts the architecture of a model in accordance with one or more embodiments.

[0019] FIG. IM depicts the architecture of a model in accordance with one or more embodiments.

[0020] FIG. IN depicts the architecture of a model in accordance with one or more embodiments.

[0021] FIG. IP depicts the architecture of a model in accordance with one or more embodiments.

[0022] FIG. 2 depicts a list in accordance with one or more embodiments.

[0023] FIG. 2A depicts a list in accordance with one or more embodiments.

[0024] FIG. 2B depicts a list in accordance with one or more embodiments.

[0025] FIG. 2C depicts a list in accordance with one or more embodiments.

[0026] FIG. 2D depicts a list in accordance with one or more embodiments.

[0027] FIG. 2E depicts a list in accordance with one or more embodiments.

[0028] FIG. 2F depicts a list in accordance with one or more embodiments.

[0029] FIG. 2G depicts a list in accordance with one or more embodiments.

[0030] FIG. 2H depicts a list in accordance with one or more embodiments.

[0031] FIG. 3 A depicts a diagram in accordance with one or more embodiments.

[0032] FIG. 3B shows a list in accordance with one or more embodiments.

[0033] FIG. 3C presents a list in accordance with one or more embodiments.

[0034] FIG. 4 depicts a neural network in accordance with one or more embodiments.

[0035] FIG. 5 depicts a flowchart in accordance with one or more embodiments.

[0036] FIG. 6A depicts a recurrent neural network in accordance with one or more embodiments.

[0037] FIG. 6B depicts an unrolled recurrent neural network in accordance with one or more embodiments.

[0038] FIG. 7 depicts a long short-term memory network in accordance with one or more embodiments.

[0039] FIG. 8A depicts a reinforcement learning scenario in accordance with one or more embodiments.

[0040] FIG. 8B depicts a reinforcement learning scenario in accordance with one or more embodiments.

[0041] FIG. 9 depicts a flowchart in accordance with one or more embodiments.

[0042] FIG. 10 depicts a system in accordance with one or more embodiments.DETAILED DESCRIPTION

[0043] In the following detailed description of embodiments of the disclosure, numerous specific details are set forth in order to provide a more thorough understanding of the disclosure. However, it will be apparent to one of ordinary skillin the art that the disclosure may be practiced without these specific details. In other instances, well-known features have not been described in detail to avoid unnecessarily complicating the description.

[0044] Throughout the application, ordinal numbers (e.g., first, second, third, etc.) may be used as an adjective for an element (z.e., any noun in the application). The use of ordinal numbers is not to imply or create any particular ordering of the elements nor to limit any element to being only a single element unless expressly disclosed, such as using the terms “before,” “after,” “single,” and other such terminology. Rather, the use of ordinal numbers is to distinguish between the elements. By way of an example, a first element is distinct from a second element, and the first element may encompass more than one element and succeed (or precede) the second element in an ordering of elements.

[0045] It is to be understood that the singular forms “a,” “an,” and “the” include plural referents unless the context clearly dictates otherwise. For example, a “renewable energy facility,” may include any number of “renewable energy facilities,” without limitation.

[0046] Terms such as “approximately,” “substantially,” etc., mean that the recited characteristic, parameter, or value need not be achieved exactly, but that deviations or variations, including for example, tolerances, measurement error, measurement accuracy limitations and other factors known to those of skill in the art, may occur in amounts that do not preclude the effect the characteristic was intended to provide.

[0047] It is to be understood that one or more of the steps shown in the flowcharts may be omitted, repeated, and / or performed in a different order than the order shown. Accordingly, the scope disclosed herein should not be considered limited to the specific arrangement of steps shown in the flowcharts.

[0048] Although multiple dependent claims are not introduced, it would be apparent to one of ordinary skill that the subject matter of the dependent claims of one or more embodiments may be combined with other dependent claims.

[0049] In the following description of FIGs. 1-10, any component described with regard to a figure, in various embodiments disclosed herein, may be equivalent to one or more like-named components described with regard to any other figure. Forbrevity, descriptions of these components will not be repeated with regard to each figure. Thus, each and every embodiment of the components of each figure is incorporated by reference and assumed to be optionally present within every other figure having one or more like-named components. Additionally, in accordance with various embodiments disclosed herein, any description of the components of a figure is to be interpreted as an optional embodiment which may be implemented in addition to, in conjunction with, or in place of the embodiments described with regard to a corresponding like-named component in any other figure.

[0050] The future of energy management in mining operations may rely heavily on integrating autonomy and artificial intelligence (Al). Key objectives of such an Al and Autonomy System may include achieving CO2 reduction targets, increasing productivity, and implementing an efficient centralized energy management system. In general, embodiments of the disclosure include methods and systems for intelligent drive optimization for mining haul trucks to accomplish the key objectives. However, it would be understood by a person of ordinary skill in the art that the methods and systems of this disclosure apply equally to a single haul truck operating independently. A plurality of haul trucks operating together may be referred to as a fleet of haul trucks, and a fleet of haul trucks may include two or more haul trucks (e.g., including but not limited to ten or more). Haul trucks typically operate at a mine according to a mine plan. The mine plan of a mine may describe the layout and organization of routes throughout the mine, or the route constraints. The mine plan may include one or more objectives to be achieved at the mine.

[0051] The schematic shown in FIG. 1A illustrates a mine environment (100) of a pit mining operation. The mine plan may include a mining production target, or a goal in weight or volume for excavated and processed ore (116). In one or more embodiments, the mine plan may also include a carbon dioxide emission threshold such that the objective for the mine is to achieve the mining production target while not exceeding the carbon dioxide emission threshold. Additional objectives may be specified in the mine plan, for example, with respect to the amount of overburden or waste (115) moved.

[0052] Haul trucks drive along haul routes throughout the mine to transport both waste(115) and ore (116). Haul routes often vary in their characteristics, including in theircurvature, slope, and length, for example. In addition, haul trucks may vary in their characteristics, exhibiting different fuel efficiencies, carbon dioxide emission, load carrying ability, and power output, for example. Many performance characteristics of haul trucks may change depending on whether the haul truck is empty or is carrying ore (116) or waste (115). In addition, the characteristics of both haul routes at a mine and of haul trucks may be influenced by the status of the environment. For example, rain and wind may degrade the road conditions along one or more routes and consequently require more power output from a haul truck to navigate slopes and turns. Accordingly, it may be necessary to optimize various parameters related to the operation of a haul truck in order to meet the constraints described above. As an example, haul trucks in a haul truck fleet operating at a mine will typically be required to drive at different velocities or speeds depending on many factors. Advantageously, by using embodiments disclosed herein, the optimal velocity of one or several haul trucks operating together may be reliably determined and their velocities adjusted accordingly in real time.

[0053] Pit mining is a surface mining technique often used in the extraction of rock and minerals from the upper layers of the surface of the earth. However, pit mines may be used in conjunction with other forms of subsurface mines. Throughout the mine environment (100) are haul routes (101) providing pathways from the crater (131), marking the lower interior surface of the pit mine where ore (116) is typically excavated, to the outer surface (133) of the mine. Pit mines often have a specific haul route (101) or set of haul routes (101) for descending towards the crater (131), or entering the pit, and a separate specific haul route (101) or set of haul routes (101) for ascending to the outer surface (133) or exiting the pit. Haul routes (101) generally facilitate the transfer of ore (116) obtained from the pit mine crater (131) to the outer surface (133) by providing a safe driving area for haul trucks (105, 107) and other heavy vehicles. Haul routes (101) may be constructed after progressively stripping of the rock face towards a certain target depth while creating “benches” (e.g., bench (102)) or “berms,” which are relatively flat surfaces resulting from removing overburden in a horizontal layer. Depending on the goals of the mine and the type and distribution of ore (116) present, overburden from the rock face is removed in decreasing volume (z.e., initially digging a large volume and then progressively smaller volumes), in increasing volume (z.e., initially digging a small volume and thenprogressively larger volumes around and below the initial dig layer), or with constant volume. In each case, one layer is stripped and removed at a time, creating a series of benches (102). All benches (102) serve as a safety measure to prevent the downward progress of large rocks or other loose material, though benches (102) at strategic locations may be made intentionally wide to serve as areas for mining work. Wide benches (102) may further be graded and connected forming pathways utilized as haul routes (101) for haul trucks (105, 107).

[0054] The mining operations at the mine environment (100) may be overseen from a base station (125). The base station (125) may generally provide a workspace for monitoring mining progress, for managing personnel, and for managing a mine plan, or description of the layout and objectives for the mine. The base station (125) may include a computer that is similar or the same as the computer illustrated in FIG. 10 and its accompanying description. The base station (125) may be connected to a wireless communication system (113) that facilitates communication between the base station (125) and the personnel and systems involved in the mine environment (100), such as drivers and vehicles. Wireless communication may be facilitated by long-range radio networks as well as broadband cellular networks, or any other method known in the art. Changes in weather may affect productivity and safety in the mine environment (100). Accordingly, the wireless communication system (113) may further facilitate communication with weather equipment (129) used to monitor the weather in the proximity of the mine environment (100). If necessary, the base station (125) may also coordinate changing the number of haul trucks (105, 107) operating together at the mine. One or more empty haul trucks (105) may be removed from the fleet, for example, to prevent machinery failure, if an empty haul truck (105) overheats. An empty haul truck (105) that is removed from the fleet may become a standby haul truck (109) where it waits for further instruction. If an additional empty haul truck (105) is needed, one or more of the standby haul trucks (109) may be added to the operating fleet.

[0055] The mine plan, maintained at the base station (125), may be used to coordinate the operation of haul trucks (105, 107) throughout the mine environment (100). Typically, a production target is specified in the mine plan, which defines the goal of extracted and processed ore (116) for the mine (e.g., as a rate by volume or mass, ora total quantity). To achieve the production target, empty haul trucks (105) traverse haul routes (101) to reach the location of excavated ore (116) and waste (115). While ore (116) may be retrieved from any of the exposed rock faces, FIG. 1 depicts the majority of ore (116) as being dug near the crater (131) using a power shovel (117). Ore (116) that has been excavated is loaded onto empty haul trucks (105) using a loader (119). The loader (119) may also be used to load an empty haul truck (105) with waste (115). Loaded haul trucks (107) typically traverse different haul routes (101) than empty haul trucks (105), and haul trucks loaded with ore (116) may use different haul routes (101) than haul trucks (105) loaded with waste (115). Loaded haul trucks (107) carrying waste (115) drive towards dumping sites where the waste (115) is gathered for later use in the mine (e.g., for building roads) or for uses external to the mine. Loaded haul trucks (107) carrying ore (116) travel to a processing station (111) where the newly excavated ore (116) may undergo one or more processing methods such as crushing, grinding, separation, beneficiation, roasting, and smelting, to name a few examples. Loaded haul trucks (107) carrying ore (116) may also travel to a stockpile (not shown) where the ore (116) is gathered before being moved elsewhere.

[0056] Haul trucks (105, 107) typically are equipped with a number of sensors and devices used to monitor the status of the haul truck (105, 107) and improve its operation. For example, a haul truck (105, 107) may have an ultrasonic range detector or radar sensors to assist in detecting the presence of other vehicles or materials. Haul trucks (105, 107) may also have digital cameras for gathering visual information in their surroundings, further improving navigational abilities. Haul truck (105, 107) suspensions may include pressure sensors to monitor load distribution and total load weight. The pressure sensors may be associated with struts. In addition, haul trucks (105, 107) may include thermometers to measure outside temperature as well as tire temperature. LIDAR systems may also be used for monitoring the status of haul truck (105, 107) tires and measure deformation. To assist in navigation, a haul truck (105, 107) may include a system providing communication to a global navigation satellite system (GNSS; for example, GPS (1145), GLONASS, BeiDou, and Galileo). In one or more embodiments, a haul truck (105, 107) may be equipped with a computer that is similar or the same as the computer of FIG. 10 and its accompanying description (below). The onboard computer may be used as a hub for gathering data from thevarious sensors and devices disposed throughout the haul truck (105, 107). The onboard computer may be connected to the wireless communication system (113) thereby facilitating wireless communication between haul trucks (105, 107) and the base station (125) as well as each other.

[0057] As both empty and loaded haul trucks (105, 107) traverse haul routes (101), the conditions of the haul routes (101) may change. For example, changes in the weather, as monitored by weather equipment (129) may cause changes in the conditions of one or more haul routes (101). Alternatively, repeated driving may degrade a haul route (101) over time, or a mining operation conducted elsewhere on a bench (102) connected to a haul route (101) may impact the haul route (101). If one or more haul routes (101) is found to be uneven and in need of attention, a grader (121) may be used to remove bumps, potholes, and other irregularities across the route. Graders (121) assist by providing more uniform and stable driving areas for haul trucks (105, 107). In addition, if conditions are too dry among one or more haul routes (101), a route may become unstable and affect visibility. A water truck (123) may be used to dampen a haul route (101) to mitigate dust while also providing density to the route surface. Together, graders (121) and water trucks (123) may assist in maintaining the conditions of the haul routes (101).

[0058] Conditions may vary within a single haul route (101) due to changes in incline, curvature, surface material, and due to the dynamic nature of mines and their environments described above (e.g., due to changes in weather and from repeated driving). A single haul route (101) may be described as composed of many individual haul route segments (e.g., haul route segment X (103) or haul route segment Y (104)). In some cases, it is useful to define a haul route segment (e.g., haul route segment X (103)) as a portion of a haul route (101) with approximately uniform conditions. In other words, the conditions of a haul route segment (e.g., haul route segment X (103)) do not change across the haul route segment (e.g., haul route segment X (103)) but may change at another haul route segment (e.g., haul route segment Y (104)). However, alternative definitions of a haul route segment may be used without limitation. For example, a haul route segment may be defined such that all route segments have equal lengths.

[0059] To reiterate, a variety of conditions may be present at the mine environment(100), and the conditions may change both dynamically (e.g., changes in weather) and spatially (e.g., changes in route curvature and incline). Accordingly, a haul truck (105, 107) must overcome each of these diverse conditions throughout its operation at a mine environment (100). FIG. IB illustrates changes in conditions that a haul truck (105, 107) may experience during its operation. Here, the operation of a haul truck (105, 107) is broken down into six general stages. The first stage (201) is loading, where ore (116) or waste (115) is loaded onto an empty haul truck (105). In FIG. 1 A, the first stage (201) may take place within the crater (131) in proximity to the loader (119), although a person of ordinary skill in the art will appreciate that loading can take place anywhere so long as the necessary equipment and materials are present. An empty haul truck (105) may weigh between tens to several hundreds of metric tons, and this base weight directly influences the energy consumption and carbon dioxide emission (z.e., the energy efficiency), of the empty haul truck (105). After being loaded, a haul truck (105) proceeds to the second stage (203) where it drives on a substantially flat surface with a full load of either ore (116) or waste (115). A loaded haul truck (107) may transport up to several hundred metric tons of material, and this additional weight (on top of the base weight of the empty haul truck (105)) further affects fuel efficiency. That is, a loaded haul truck (107) has a significantly different fuel efficiency compared to an empty haul truck (105) due to the drastic increase in weight. In addition, the fuel efficiency will vary depending on whether the loaded haul truck (107) is carrying ore (116) or waste (115), as typically these materials differ in their density (z.e., mass / volume). In FIG. 1 A, the second stage (203) may take place as a loaded haul truck (107) drives from the crater (131) towards one of the haul routes(101).

[0060] Eventually a loaded haul truck (107) must traverse an incline to reach the outer surface (133) of the mine environment (100). The third stage (205) of haul truck (105, 107) operation includes driving the loaded haul truck (107) uphill. Depending on the grade of the incline, or the steepness of the slope, the third stage (205) may require substantially more power output from the loaded haul truck (107) compared to the second stage (203) of driving the loaded haul truck (107) on a flat surface. In FIG. 1 A, the third stage (205) may take place as a loaded haul truck (107) traverses a haulroute (101) from the crater (131) to the outer surface (133) and, if the loaded haul truck (107) is carrying ore (116), towards the processing station (111). After the third stage (205), the loaded haul truck (107) proceeds to the fourth stage (207) of stopping and unloading. Transitioning between the third stage (205) and the fourth stage (207) requires a significant change in velocity of the loaded haul truck (107) weighing up to several hundred metric tons and consequently a significant change in momentum and energy output. In FIG. 1, the fourth stage (207) may take place at the processing station (111) or a stockpile (not shown) if the loaded haul truck (107) is carrying ore (116), or at a dumping area (not shown) if the loaded haul truck (107) is carrying waste (115).

[0061] Once a loaded haul truck (107) has unloaded at the fourth stage (207), the now empty haul truck (105) progresses to the fifth stage (209) of driving on a substantially flat surface without carrying any material. In FIG. IB, the fifth stage (209) may take place as an empty haul truck (105) drives from the processing station (111) along a haul route (101) leading towards the crater (131) or another destination where the ore (116) or waste (115) is located, but before beginning the actual descent. As previously described, an empty haul truck (105) will weigh significantly less than a loaded haul truck (107) and thus exhibit greater fuel efficiency. After the fifth stage (209), the empty haul truck (105) proceeds to the sixth stage (211) where it drives downward along a haul route (101) and towards the location of ore (116) or waste (115). The sixth stage (211) is the most energy-efficient stage because the load is the lightest and descending may exploit the difference in gravitational potential energy. Although gravity will assist the empty haul truck (105) to roll downhill, the empty haul truck (105) is nonetheless a massive vehicle and will consume energy to maintain a safe speed and navigate potential turns. In FIG. 1 A, the sixth stage (211) may take place as an empty haul truck (105) proceeds to descend a sloped haul route (e.g., the haul route (101) containing haul route segment Y (104)). After the sixth stage (211), the empty haul truck (105) returns to the loading point, or the first stage (201), and the cycle repeats.

[0062] It is emphasized that a haul truck (105, 107) need not proceed from one stage to another in the exact order described and illustrated in FIG. IB. The progression of stages (201-211) of operating a haul truck (105, 107) are primarily for illustrativepurposes. In principle, a haul truck (105, 107) may transition from one stage to any other physically allowable stage depending on the organization and layout of the mine environment (100). For example, an empty haul truck (105) may drive repeatedly on flat, declining, and inclining surfaces as it moves from one location to another. Similarly, a loaded haul truck (107) may drive repeatedly on flat, declining, and inclining surfaces as it moves from one location to another. Obviously, a haul truck (105) cannot transition from the first stage (201) of loading directly to unloading at the fourth stage (207) without first driving some distance and moving through a driving stage (e.g., second stage (203)).

[0063] As described above, FIG. IB illustrates different stages (201-211) of operating a haul truck (105, 107). Recall that haul routes (101) may be composed of different haul route segments (e.g., haul route segment X (103) or haul route segment Y (104)), and, in some instances, a haul route segment may be defined such that the conditions across the segment are substantially uniform. Accordingly, a haul route segment may be described as a portion of a haul route (101) requiring only one stage of operating a haul truck (105). For example, returning to FIG. 1, a haul route (101) may include a haul route segment X (103) that is defined by requiring only the fifth stage (209) of driving on a substantially flat surface with no load. As another example, FIG. 1A shows a haul route (101) with a haul route segment Y (104) that is defined by requiring only the sixth stage (211) of driving on a declined surface with no load. In this way, it may not be entirely necessary to keep track of every individual haul route segment because haul route segments may be grouped according to their mutual properties (e.g., two haul route segments both requiring the third stage (205) of operating a haul truck (105) may require similar energy output). However, this need not be the only definition used for defining a haul route segment, and the above is provided for illustrative purposes. For example, it may be convenient to define all haul route segments as having a particular length, a particular increase or decrease in elevation, a predefined fixed number of turns, or according to another requirement.

[0064] FIG. IB further illustrates that various possible environment conditions (213) may be present as a haul truck (105) operates. More specifically, FIG. IB illustrates different possible environment conditions (213) being present across a week of operation at a mine. In the illustration, the first day may present sunny conditions(214), the second day may present windy conditions (215), the third day may present a dust storm (216), the fourth day may present rainy conditions (217), the fifth day may present storming conditions (218), the sixth day may present sunny conditions (214) again, and the seventh day may present cloudy conditions (219). Each different environment condition (213) will affect the haul truck (105, 107) operation stages (201-211).

[0065] Consider the impact that sunny conditions (214) on the first day may have on each of the stages (201-211) of operating a haul truck (105, 107) in comparison to the impact of storming conditions (217) on the fourth day. The solid arrows in FIG. IB indicate the impact of the sunny conditions (214) of the first day on each of the stages (201-211). On the first day, sunny conditions (214) may contribute to increasing tire heat. As a haul truck (105, 107) operates, increasing loads and power output of the haul truck (105, 107) lead to greater heating of the tires. To preserve the longevity of tires, the temperature of tires is typically monitored, and power demands of haul trucks (105, 107) must be adjusted to maintain the tire temperature within a safe limit. Consequently, driving a loaded haul truck (107) through both the second stage (203) on flat ground and the third stage (205) uphill may each require a reduction in velocity. In more extreme sun and heat conditions, velocities may need to be reduced across all stages of operating a haul truck (201-211). Further, sunny conditions (214) may dry out the driving surfaces, creating dust, which may decrease tire traction. As illustrated, the sunny conditions (214) of the first day are followed by windy conditions (215) on the second day and, consequently, a dust storm (216) on the third day. A dust storm may decrease visibility, further requiring a reduction in haul truck (105, 107) velocities or ceasing operation altogether.

[0066] Rainy conditions (217), for example on the fourth day, may have a different impact on each of the stages (201-211) of operating a haul truck (105, 107). The dashed arrows in FIG. IB indicate the impact of the rainy conditions (217) of the fourth day on each of the stages (201-211). Depending on the amount of rain, the rainy conditions (217) may either improve the driving conditions or make them worse. As discussed in relation to the water truck (123) in FIG. 1 A, providing an appropriate amount of water to the driving surfaces may improve the driving conditions by adding density to the material on the surface and preventing the formation of dust. However,too much rain can degrade the driving surfaces and negatively affect slope stability. Accordingly, if the rainy conditions (217) of the fourth day are mild, haul trucks (105, 107) may operate at greater velocities at each driving stage (203, 205, 209, 211) owing to the improved driving surfaces and improved visibility. However, if the rainy conditions (217) of the fourth day are extreme, haul trucks (105, 107) will be required to operate at slower velocities, especially during an uphill stage (205) and a downhill stage (211).

[0067] In one or more embodiments, intelligent drive optimization for mining haul trucks (105) may be determined using an Al and Autonomy System based on (or, accepting as inputs) real-time haul truck data obtained by a fleet of haul trucks operating together at a mine, environment data describing the environmental factors and weather at the mine, haul load efficiency data describing the performance of the haul trucks (105) depending on the load that is being carried, a mine plan that includes route constraints for the mine and a mining production target, and other inputs detailed below.

[0068] As an example, one part of the intelligent drive optimization for mining haul trucks (105) may be the determination of an optimal haul truck velocity. This parameter may be determined given, (or in relation to, or in view of), one or more targets or goals specified by the mine plan. As described above, the mine plan may include a mining production target. Therefore, the optimal haul truck velocity may be the haul truck velocity that, when operating the haul truck (105) at the optimal haul truck velocity, would achieve the mining production target with the least amount of resources consumed. For example, the objective of the mine plan may be to achieve the mining production target in the least amount of time which might result in a considerably high optimal haul truck velocity. Alternatively, the objective may be to achieve the mining production target while using the least amount of fuel, or while emitting the least amount of carbon dioxide, which might result in a relatively lower optimal haul truck velocity. Thus, the optimal haul truck velocity may vary depending on the goals of the mine plan.

[0069] The Al and Autonomy System may determine an optimal haul truck velocity for any number of haul trucks (105) in the fleet, including for one haul truck (105) up to and including every haul truck (105). In addition, the optimal haul truck velocitymay vary for different routes in the mine, or for segments (or portions) of a route. Accordingly, the Al and Autonomy System may further determine an optimal haul truck acceleration that describes the rate at which a haul truck velocity should be changed along different segments of one or more routes. In addition to determining the optimal haul truck velocity, the Al and Autonomy System may further determine a prediction of the mining production that will be achieved when operating one or more haul trucks (105) at the optimal haul truck velocity, as well as an estimate of the resources that will be consumed, for example, the amount of fuel consumed. Additional outputs of the composite Al model are not beyond the scope of the present disclosure. For example, the Al and Autonomy System may also determine a predicted carbon dioxide emission, or a predicted amount of waste (115) moved, when operating one or more haul trucks (105) at the optimal haul truck velocity.

[0070] Continuing with the example of determining an optimal haul truck velocity, in one or more embodiments, the Al and Autonomy System may include a plurality of individual Al models working in concert to determine the optimal haul truck velocity. The optimal haul truck velocity is dependent on many factors within the mine, and determining the optimal haul truck velocity may be achieved first by determining the conditions of at least one segment of a haul route and then determining consequent maximum safe hauling velocity and acceleration across the at least one segment. The Al and Autonomy System may contain an Al model that determines the conditions of at least one haul route (101) of the set of haul routes (101) based on the real-time haul truck data. For example, as a haul truck (105) drives along a route or route segment, it may gather, in the form of real-time haul truck data, information regarding the status of the route or route segment via sensors and cameras described below. The determined conditions may include an estimate of the rolling resistance or rugosity of the route or route segment. In addition, component Al models within the Al and Autonomy System may be configured to request a grader (121) or water truck (123) to improve the conditions of the haul route (101) or route segment. An Al model may determine the conditions of only one haul route segment, a plurality of haul route segments, an entire haul route (101), and up to and including all of the routes throughout the mine. However, the minimum determination of the Al model is the conditions of at least one segment of a haul route (101).

[0071] The effective implementation of an Al and Autonomy System may depend on various innovative strategies that are incorporated within it, including the following.

[0072] Intelligent PLC-Driven Electric Drive Optimization: May optimize power usage across varying torque demands and enhances regenerative braking capabilities, improving both efficiency and safety.

[0073] EV Automation and Autonomous Operation: May implement advanced automation for autonomous vehicle routing and operation. This includes automatic alerts for efficient route recommendations and managing overall site speed limits to ensure compliance with operational and safety standards.

[0074] Emission Analysis and Optimal Speed Regulation: May dynamically adjust vehicle speeds to minimize CO2 emissions, substantially contributing to environmental sustainability.

[0075] Customized Generative Al: May develops adaptive operational strategies and facilitates transparent communication.

[0076] Automated Efficient Route Recommendation and Fleet Size Adjustment Alerts: May utilize real-time data to optimize routing and adjust fleet size as necessary to meet demand without sacrificing efficiency.

[0077] Optimal Energy Management System Determination: May focus on maximizing energy efficiency throughout all operations.

[0078] Decision Support System UI: May provide a centralized dashboard for real-time decision-making, enhancing fleet management and operational oversight.

[0079] By leveraging these strategies, in conjunction with automation capabilities, the system may achieve peak operational efficiency, substantially reduce environmental impacts, and pave the way for future advancements in autonomous and sustainable mining operations. With predictive maintenance and optimized routing, the Al and Autonomy System may ensure optimal operation of each haul truck (105). This comprehensive approach may not only lower operational costs and reduce the environmental footprint but also may significantly improve safety, presenting a forward-thinking solution to the challenges of modem mining.

[0080] There may be multiple components in the Al and Autonomy System including, but no limited to, the following layers, shown in FIG. 1C:

[0081] Input Layer Multi-Modal (180)

[0082] Data transmission Layer (Communication Network Data Storage) (182)

[0083] Control and Communication Interface Layer (184)

[0084] Hybrid Multi-Modal Al Fusion Layer (Al Models, Data Processing and Analytics), i.e., Mobile Al Layer (186)

[0085] Inter-Vehicle Communication Layer (188)

[0086] UI Layer (190)

[0087] Report for Decision Support System UI i.e., Centralized Dashboard (191))

[0088] Productivity Dashboard (192)

[0089] Energy Management System Dashboard (193)

[0090] Emissions Dashboard (194)

[0091] Maintenance and Diagnostics Layer (195)

[0092] Safety and Compliance Monitoring Layer (196)

[0093] FIG. 1C presents a possible embodiment of the architecture of the Al and Autonomy System and illustrates its organization and flow of information. For instance, data may enter the system through the Input Layer Multi-Modal (180), pass through several layers that process the data, and then be reported to a user in the Centralized Dashboard (191). Each of the above layers of the Al and Autonomy System will be addressed individually below.

[0094] The Input Layer Multi-Modal (180) may serve as the foundational entry point for diverse data streams into the Al and Autonomy System. This layer may be specifically designed to handle and normalize data from a variety of sources to ensure consistency and usability across the system. It may interface directly with on-truck sensors (capturing real-time data from haul trucks (105), including internal telemetry, tire pressure, load weight, and more); the electric drive systems (gathering specific metrics related to the performance data (1036) and status of electric drivetrains); roadcondition monitoring systems (utilizing sensors deployed across the mining site to provide up-to-date information on road conditions such as grade, curvature, surface type, and potential hazards); mine site operational data (integrating data from overall mine operations, including mine planning outputs, fleet management systems, and environmental data (1035)); and fleet and mine software applications and databases (tapping into existing mine fleet software and databases to pull historical and operational data (1030), enhancing the system’s decision-making capabilities with a rich historical context).

[0095] Key features of the Input Layer Multi-Modal (180) may include data aggregation (collecting and aggregating data from multiple sources, ensuring a comprehensive dataset is available for processing); data normalization (implementing normalization processes to standardize data formats, units, and scales, making them compatible for analysis and processing in subsequent layers); real-time data handling (equipped to handle real-time data streams, enabling the system to react promptly to dynamic changes in mine operations); and data integration (1037) (seamlessly integrating with existing IT infrastructure, minimizing disruptions and leveraging current technological investments).

[0096] Key objectives of the Input Layer Multi-Modal (180) may include data accuracy and reliability (ensuring high data quality and reliability, which are critical for the accuracy of Al-driven decisions); timeliness (providing timely data inputs to downstream Al processing layers, crucial for real-time decision-making and operational adjustments); and scalability (allows for scaling with increasing data volumes or new types of data sources as mining operations expand or evolve).

[0097] The Data Transmission Layer (182) may serve as the critical infrastructure for the secure and efficient movement and storage of data within the Al and Autonomy System. It may employ a hybrid architecture that leverages both edge and cloud computing to optimize performance and scalability.

[0098] Key components of the Data Transmission Layer (182) may include edge computing devices, which can be deployed directly at the mining site, to perform realtime data processing near the source of data. Edge computers may be crucial for immediate responses required by autonomous vehicles and machinery, local decision-making processes where latency is a critical factor, and handling sensitive data that requires quick processing and should not leave the site for security reasons.

[0099] Key components of the Data Transmission Layer (182) may also include cloud computing resources, which may be utilized for storing vast amounts of data that exceed the local storage capacities, running complex analytics and machine learning models that are not time-sensitive but require significant computational power, and aggregating data from multiple sites for centralized management and analysis.

[0100] Key features of the Data Transmission Layer (182) may include data synchronization (ensuring seamless data synchronization between edge and cloud components, maintaining data consistency across the system); scalability (allowing for easy scaling of resources as data demands grow, without the need for significant upfront investment in local infrastructure); security protocols (implementing robust security measures, including data encryption and secure data transfer channels, to protect sensitive information against unauthorized access); and data redundancy and backup (providing data redundancy and robust backup solutions to ensure data is preserved and recoverable in the event of a hardware failure or other disruptions).

[0101] Objectives of the Data Transmission Layer (182) may include low latency (achieving minimal latency in data processing and transmission to support real-time operational needs of autonomous mining vehicles and systems); high availability (ensuring high availability of the system and data, crucial for continuous mining operations and safety); and cost efficiency (balancing between local processing and cloud services to optimize costs associated with data storage and processing).

[0102] The Control and Communication Interface Layer (184) may serve as the nexus between the physical components of the mining vehicles and the Al-driven analytics and control systems. This layer may facilitate robust, real-time interactions between the vehicle's PLC (Programmable Logic Controller) and the Al system, enhancing the efficiency and responsiveness of operations.

[0103] Key components of the Control and Communication Interface Layer (184) may include PLC connections (interacting with various vehicle components such as the battery, electric motors, gearbox, and inverters). PLC connections may collect signals from these components to monitor their status and control operations. Another keycomponent of the Control and Communication Interface Layer (184) may be an Al System Interface. This may provide a communication bridge between the Al models and the vehicle's PLCs. This interface may also be critical for transmitting AL generated commands to the PLCs for optimized control over vehicle mechanics, such as energy use and drive optimization.

[0104] Key features of the Control and Communication Interface Layer (184) may include real-time data exchange (ensuring high-speed data exchange between the Al systems and the vehicle components, allowing for immediate adjustments and controls based on Al insights); protocol standardization (implementing standardized communication protocols to ensure compatibility and reliability across different types of equipment and software systems); and security and safety measures (including robust security measures to prevent unauthorized access and ensure the integrity of the control signals, crucial for maintaining operational safety).

[0105] Objectives of the Control and Communication Interface Layer (184) may include enhanced operational efficiency (enabling effective communication between Al and PLC systems, this layer may optimize energy use and operational performance of the mining vehicles); scalability and flexibility (accommodating additional components or upgrades without significant changes to the existing infrastructure); and reliability and low latency (maintaining a reliable communication channel with minimal latency, essential for the seamless operation of autonomous and semi- autonomous vehicles).

[0106] The Control and Communication Interface Layer (184) may integrate with Al Models through intelligent PLC-driven electric drive optimization. Although this function is primarily hosted within the Mobile Al Layer (186), the Al and Autonomy System may rely on the Control and Communication Interface Layer (184) to execute its algorithms effectively. The model may send optimized commands back to the PLC to adjust electric drive parameters for improved energy efficiency and vehicle performance.

[0107] The Mobile Al Layer (186) is the central processing unit of the Al system and uses a hybrid multi -model fusion approach. The Mobile Al Layer (186) may take energy availability, road conditions, the mine plan, dynamic occurrences, and load / grade variations to create solutions that may be directly fed into the vehiclesthemselves. The Al and Autonomy System may follow a hybrid model approach where a neural network serves a vital role for learning and adapting to the changing needs of the mining, adjusting speeds, payloads, routes, and the number of trucks in the system to ensure the lowest energy use per tonne moved whilst maintaining specified production rates.

[0108] Key components of the Mobile Al Layer (186) may include data integration (aggregating and integrating data from multiple sensors and systems, ensuring a comprehensive dataset for analysis); model orchestration (managing various Al models, including machine learning algorithms, neural networks, and customized generative models). These models may process the integrated data to generate insights and operational commands.

[0109] Key components of the Mobile Al Layer (186) may also include a fusion mechanism (FM) (187) that acts as the core processor of the Mobile Al Layer (186), fusing outputs from individual models to determine optimal operational parameters such as speed, route, and energy use, etc. Key functions of the FM (187) may include sending data to the Centralized Dashboard (191) or HMI for operator interaction, issuing alerts and recommendations for optimal speed, routing (1124), and fuel consumption, etc., and communicating with other vehicles in the fleet for coordinated actions. The FM (187) ensures that decisions are made based on a consensus approach; a confidence threshold may be set at over 90%, although other values may be used as well. If consensus is not achieved, the system may escalate the decisionmaking process to a Master Al model. This agent may act as a higher-order analytical model, using a broader dataset from various fleet software applications to validate and refine the decisions and model outputs.

[0110] Key components of the Mobile Al Layer (186) may further include a Master Al Model that functions as an overarching decision agent, analyzing data discrepancies or conflicts among model outputs and providing a higher-order analysis to refine decisions and model thresholds. The Master Al model may interface with cloud APIs (1021) (possibly via a third-party database manager) to access and process data for advanced analytics. An intelligent feedback mechanism may interlink the Master Al Model with the fusion mechanism, thus enabling continuous refinement of the operational algorithms. This interlinking process may dynamically adjust decision- 1making thresholds and enhance data collection protocols based on real-time performance feedback. Such iterative updates to the FM (187) may ensure that the system’s adaptive intelligence remains at the cutting edge, consistently informed by the most recent and relevant operational insights.

[0111] Key components of the Mobile Al Layer (186) may further include a generative Al, which may be a key innovation. Model 6 (1006) is engineered to serve as an intelligent advisor for haul operations. This model may leverage advanced generative algorithms to synthesize and extrapolate from vast datasets, producing novel patterns, predictions, and operational strategies.

[0112] Key features of the Mobile Al Layer (186) may include advanced data processing, which may utilize state-of-the-art data processing and analytics techniques to ensure real-time responsiveness and accuracy. Key features may further include dynamic decision-making, which may employ a dynamic decision-making process that adapts to real-time data and model feedback, optimizing operational decisions continuously. Key features further may include an interface with control systems. This may directly interface with vehicle drive controls and other operational systems via edge computing devices and cloud APIs (1021), allowing seamless execution of Al-derived commands.

[0113] Objectives of the Mobile Al Layer (186) may include CO2 reduction, productivity enhancement, and energy management. CO2 reduction may target significant reductions in CO2 emissions by optimizing operational parameters based on environmental impact assessments. Productivity Enhancement may aim to enhance productivity by intelligently managing fleet operations based on real-time data and predictive analytics. Energy management may implement sophisticated energy management strategies to minimize energy consumption while maintaining optimal operational output.

[0114] The Mobile Al Layer (186) may act as a fusion mechanism. Data fusion may be performed by a weighted sum or averaging method that combines model outputs based on their reliability and accuracy. Data fusion may also be performed with a voting system that employs majority voting to decide on the most frequent output from models for critical decisions. Data fusion may also be performed with concatenation and feature fusion; this method may integrate features or outputs frommultiple models to form a comprehensive input vector for further analysis. Other methods of data fusion may include model stacking and hybrid models, which utilize a hierarchical approach where outputs of initial models serve as inputs to subsequent models to refine decisions. Another data fusion method that may be used is neural network fusion, which leverages neural networks to learn how to best combine outputs from various models for optimal decision-making. These data fusion methods are not limiting; other data fusion methods may be used by the Mobile Al Layer (186).

[0115] The architecture of the Mobile Al Layer (186) may include a Hybrid Integrative Architecture, which is a blending edge and cloud computing resource to balance computational load and optimize response times across the mining fleet's operations. The Hybrid Integrative Architecture may ensure the Mobile Al Layer (186) may serve as the technological linchpin within the fleet management ecosystem, enabling a proactive and intelligent mining operation.

[0116] The Inter-Vehicle Communication Layer (188) may be essential for the decentralized coordination of mining trucks, enabling them to operate semi- autonomously with minimal reliance on centralized control. This layer incorporates advanced communication technologies and protocols to facilitate direct and efficient interaction among vehicles, enhancing both operational efficiency and safety.

[0117] Key technologies and functions of the Inter-Vehicle Communication Layer (188) may include vehicle-to-vehicle (V2V) communication; trucks equipped with V2V technology may share vital information such as location, speed, direction, and planned maneuvers directly with each other. This capability may be useful for coordinating movements to optimize routing, prevent collisions, and efficiently manage loading and unloading procedures.

[0118] Key technologies and functions of the Inter-Vehicle Communication Layer (188) may further include autonomous and semi-autonomous systems that may integrate with autonomous driving technologies and allow trucks to execute complex navigation and operational tasks on haul roads autonomously. These systems may leverage data from V2V communications and other sensors to make informed decisions on the spot.

[0119] Another key technology of the Inter-Vehicle Communication Layer (188) may be decentralized decision-making algorithms. These may utilize multi-agent systems (MAS) and distributed Al to empower each truck to act as an independent agent. These agents may assess their immediate environment, communicate with peer vehicles, and make autonomous decisions, enhancing the system's scalability and reducing bottlenecks associated with central control.

[0120] Yet another key technology of the Inter-Vehicle Communication Layer (188) may be edge computing (1022), where data may be processed locally on or near the vehicles, minimizing latency and reliance on distant servers. This setup may support real-time decision-making critical in dynamic and potentially hazardous mining environments.

[0121] Another key technology of the Inter-Vehicle Communication Layer (188) may be safety and efficiency protocols, which may implement protocols designed to maximize safety and operational efficiency without centralized oversight. These may include dynamic rerouting based on immediate traffic conditions, operational delays, or emergencies.

[0122] A final key technology of the Inter-Vehicle Communication Layer (188) may be robust message passing. This may ensure clear, reliable, and timely communication among vehicles. This system may be designed to be resilient to disruptions and capable of handling a range of data types, from simple notifications to complex sensor data.

[0123] Objectives of the Inter-Vehicle Communication Layer (188) may include increased scalability (facilitating scaling operations more effectively by reducing dependency on centralized control systems); enhanced robustness (improving system robustness by eliminating single points of failure, critical in expansive and hazardous mining environments); and immediate responsiveness (increasing the system’s responsiveness to changes and events in the environment by enabling local, real-time decision-making).

[0124] Potential enhancements from the Inter-Vehicle Communication Layer (188) may include integration with global communication networks (1146). This may further enhance communication reliability and range, integrating satellitecommunication or other advanced wireless technologies could be considered. Potential enhancements of the Inter-Vehicle Communication Layer (188) may also include advanced encryption and security protocols. Given the critical nature of operational data, implementing advanced encryption methods and security protocols may safeguard communications.

[0125] The UI Layer (190) may serve as the human-facing interface of the Al and Autonomy System and may be designed to present complex data analysis (1094) in an accessible and actionable format for fleet managers and operators. It may consist of specialized dashboards that provide real-time insights and management capabilities, enabling prompt and informed decision-making for the optimization of haul truck operations.

[0126] In the scenario of a fleet of haul trucks operating in a large mining operation, the Centralized Dashboard (191) may monitor each truck in real-time, displaying its status on a screen accessible by the fleet manager. When the system detects that a particular truck is consuming fuel at an unsustainable rate, it may trigger an alert and suggest a more efficient route (e.g., one that avoids uphill paths where possible). Simultaneously, it may identify that this truck is due for predictive maintenance on its fuel system, scheduling a maintenance task automatically. The fleet manager may review these recommendations on the dashboard and approve the route change and maintenance schedule with a few clicks, optimizing operations without needing to deep dive into the raw data or manually coordinate these tasks.

[0127] The Centralized Dashboard (191) Layer may ensure that the vast amounts of data and insights generated by Al and machine learning models are translated into practical, actionable information, facilitating smart decision-making and efficient haul truck operations. These benefits may include efficiently identifying trucks with unsustainable fuel consumption, automatically triggering an alert and recommends a more efficient route (1124), automatically scheduling predictive maintenance for the truck's fuel system, on-click approval of routes and maintenance tasks by a fleet manager, and, in general, the enablement of smarter decisions and more efficient haul truck operations.

[0128] Serving as a critical part of the Decision Support System UI (user interface), theProductivity Dashboard (192) may be used for assessing operational efficiency. TheProductivity Dashboard (192) may be fed by the Mobile Al Layer (186) and, within it, the Master Al Model (1007). The Productivity Dashboard (192) may reflect the real-time productivity status and historical trends, aiding in the identification of opportunities for operational improvements.

[0129] A central component of the Centralized Dashboard (191) is its display capability. This display may convey information related to optimal energy use and fuel consumption, predicted productivity compared to actual productivity, productivity metrics (1117), and production metrics hybrid trucks.

[0130] Another critical part of the Centralized Dashboard (191) may be the Energy Management System Dashboard (193). This dashboard may monitor and manage the fleet's energy consumption. By leveraging the data processed by the Mobile Al Layer (186) and Master Al Model (1007), the Energy Management System Dashboard (193) may present a clear view of energy usage patterns, providing insights for energy optimization and cost savings.

[0131] As part of the display of the Energy Management System Dashboard (193), optimal energy use and fuel consumption may be shown (to ensure the lowest energy use per tonne moved whilst maintaining or increasing specified production rates). This may facilitate determining the best energy management system. Regeneration of energy and emissions may be displayed as well.

[0132] Another part of the Centralized Dashboard (191) is the Emissions Dashboard (194), which may play an important part in environmental stewardship and compliance management. It may utilize outputs from the Mobile Al Layer (186) and Master Al Model (1007) to monitor emissions metrics, support regulatory reporting, and drive initiatives for reducing the environmental impact. Its display may show predicted versus actual carbon emissions per type of fleet: z.e., diesel, hybrid, electric, etc. .

[0133] The Maintenance and Diagnostics Layer (195) may be dedicated to the proactive and ongoing health monitoring of mining fleet equipment. This layer may employ predictive analytics and machine learning algorithms to process data from vehicle sensors and operational systems, anticipating maintenance needs before they become critical.

[0134] The key functions of the Maintenance and Diagnostics Layer (195) may be predictive maintenance, diagnostics, and health monitoring. Predictive maintenance may include the utilization of patterns and trends from historical and real-time data to predict equipment failures and schedule maintenance activities; it may also minimize downtime. Diagnostics may provide tools to rapidly identify and troubleshoot issues, supporting swift resolution and reducing the impact on operations. Health monitoring may include continuous monitoring of the health and performance of vehicle components, ensuring they are operating within optimal parameters.

[0135] The Maintenance and Diagnostics Layer (195) may receive inputs from the Mobile Al Layer (186), including detailed operational data and sensor readings, and may accurately assess the condition of each vehicle component. It may also contribute to the overall efficiency of the fleet by ensuring that vehicles are maintained in a state that maximizes energy efficiency and reduces the likelihood of energy wastage through malfunctioning components.

[0136] The Maintenance and Diagnostics Layer (195) may minimize unscheduled downtime by predicting maintenance needs; the layer aims to minimize unscheduled downtime and extend the lifespan of fleet assets. The Maintenance and Diagnostics Layer (195) may also enhance overall operational efficiency by ensuring that all vehicles operate at peak performance, thus contributing to the productivity and energy efficiency of the mining operation.

[0137] The Safety and Compliance Monitoring Layer (196) may ensure that all aspects of the mining operation adhere to established safety standards and regulatory compliance requirements. This layer may be critical in managing risks and protecting assets, including personnel, equipment, and the environment. Its key functions may include safety monitoring, compliance tracking, incident analysis and reporting. Safety monitoring may leverage real-time data to monitor safety-critical parameters and detect potential safety breaches, triggering immediate alerts and corrective actions. Compliance tracking may keep track of all compliance requirements, including emissions, noise levels, and operational limits, to ensure that the fleet operations do not violate regulatory stipulations. Incident analysis and reporting may, in the event of an incident, facilitate detailed analysis to understand causality and to report findings for regulatory compliance and continuous safety improvement.

[0138] The Safety and Compliance Monitoring Layer (196) may collaborate closely with the Mobile Al Layer (186) to receive real-time operational data that is crucial for monitoring compliance and identifying safety-related anomalies. The Safety and Compliance Monitoring Layer (196) may utilize Al-driven insights to enhance decision-making related to safety procedures and compliance measures, incorporating a proactive approach to risk management.

[0139] The objectives of the Safety and Compliance Monitoring Layer (196) may include reducing operational risks by ensuring that safety protocols are followed and that vehicles operate within safe parameters. The objectives may also include maintaining strict adherence to regulatory standards, thus protecting the mining operation from potential fines, shutdowns, or legal action.

[0140] The embodiment presented in FIG. 1C shows the 12 models in the Mobile Al Layer (186), each performing a different ML-related task. This is not a limitation of the methods presented in this document. There may be more or less models in the actual implementation of the Al and Autonomy System. The 12 models are as follows:

[0141] Model 1 (1001): Truck model, Optimal energy use per tons moved

[0142] Model 2 (1002): Overall Site Speed and Acceleration Optimisation for production and CO2

[0143] Model 3 (1003): Estimate the maximum speed / accel per segment of a route or Optimal speed regulation

[0144] Model 4 (1004): Estimate the road conditions of haul road segments from vision

[0145] Model 5 (1005): Intelligent PLC-Driven Electric Drive Optimization

[0146] Model 6 (1006): Generative Al

[0147] Model 7 (1007): Master Al

[0148] Model 8 (1008): Predictive production

[0149] Model 9 (1009): Vote to reduce or increase truck numbers

[0150] Model 10 (1010): Analysis comparison of predicted and actual data

[0151] Model 11 (1011): Hybrid fuel consumption per truck and fleet

[0152] Model 12 (1012): Autonomous Operation

[0153] FIG. ID shows the details of Model 1 (1001), known as the “Truck” model. The diagram in FIG. ID visualizes how various inputs may flow into the Truck Model, how the model may process these inputs, and how outputs may be utilised within the mining operation system. This diagram may include data flow from sensors to edge devices, from edge devices to cloud storage, and from the model to operational systems for real-time adjustments. This model may leverage a comprehensive dataset sourced from multiple inputs, including client historical data (1030), loT devices (1031), sensors (1032), various predictive models, and mine software systems (1033). This model may integrate these diverse data streams to optimise operational parameters across mining operations. One objective may be to minimize energy consumption per tonne moved while ensuring that production rates are maintained or increased. This may involve dynamically adjusting payloads, truck numbers, and routing, as well as providing strategic stockpile recommendations.

[0154] The inputs to Model 1 (1001) may be separated into different categories. First are vehicle and route parameters (1034), which may include truck telematics (1119) (e.g., gradeability, tire performance), route parameters (e.g., gradient, total distance, speed limits), and payload parameters (e.g., type and weight of the payload). Second are environmental data (1035), which may include weather conditions (e.g., ambient temperature, rainfall, wind speed) and operational data from mine software (e.g., energy management system, fleet management system). Next are performance and compliance data (1036), which may include compliance data (e.g., emissions data, regulatory speed limits) and performance data (e.g., cycle times, fuel burn, vehicle wear and tear). Finally, are the data integration (1037), which may include calculated route scenarios (e.g., segment distance, elevation changes) and data from other models (e.g., data from Model 3 (1003) and Model 4 (1004)).

[0155] The outputs from the Truck Model (1001) may also be separated into different categories. First are operational efficiency parameters (1038), which may include optimal energy use (e.g., energy per tonne moved), payload management (e.g., payload for optimal performance), fleet management (e.g., truck numbers based onneed), and route optimization: (e.g., best routes based on various criteria). Second are reporting and compliance parameters (1039), which may include carry back reporting (e.g., residual cargo on trucks), emissions optimization (e.g., emissions within operational limits), and volume optimization (e.g., maximum volume handled efficiently). Third are strategic decisions output parameters (1040), which may include optimal stockpile locations (stockpile locations for ore (116)), optimal dump locations (e.g., efficient dump locations for waste (115)), and comparison analysis (e.g., comparing efficiency metrics between operational zones).

[0156] The architecture for the Truck Model may combine predictive analytics and optimization algorithms. Techniques such as regression analysis, machine learning models (like random forests or gradient boosting), and optimization methods (e.g., linear programming) may be employed to process inputs and compute the most efficient operational parameters.

[0157] The Truck Model may integrate with loT devices (1031), mine software systems (1033), and client databases via APIs (1021) to gather real-time and historical data. The Truck Model may also utilize existing mine communication infrastructure for real-time data transmission. Furthermore, the Truck Model may employ cloud computing (1020) for data aggregation and heavy computational tasks, and edge computing (1022) for real-time decision-making at the site.

[0158] FIG. IE shows Model 2 (1002), the “Overall Site Speed and Acceleration Optimization for Production and CO2” model, which may utilize data from various operational and environmental sources to optimize speed and acceleration parameters across the mining site. The data sources may include Model 1 (1001), Model 3 (1003), loT devices and sensors (s), and mine software (1044). Its main goal may be to balance production efficiency and CO2 emission targets. This may involve dynamically setting speed and acceleration limits for different segments of the mining site based on truck loads, emission data, and compliance requirements.

[0159] The inputs to Model 2 (1002) may include maximum speed and acceleration (1045), and velocity from optimal estimates per route segment. They may also include CO2 emissions and compliance data (1046) from mine engineering software. Additionally, they may include truck load and payload telemetry (1047). Furthermore, they may include mine operational plans and regulatory speed rules.

[0160] The outputs from Model 2 (1002) may include the site optimal speed limit and acceleration limit (1048) (adjusted per route segment to meet CO2 and production target; customized per route segment for optimal performance and compliance) and suggested optimal braking (1049) and payload configurations (1050) for efficiency and safety.

[0161] The architecture of Model 2 (1002) may use a combination of predictive modelling and constraint optimization algorithms to analyse input data and determine optimal speed and acceleration settings. It may incorporate feedback mechanisms to adjust recommendations based on real-time operational data and compliance updates.

[0162] Model 2 (1002) may integrate directly with loT sensors (1043) for real-time data, and mine software systems (1044) for operational planning and compliance data. Model 2 (1002) also may use cloud computing (1020) for historical data analysis (1094) and aggregation, with edge computing (1022) handling real-time decisionmaking and updates.

[0163] FIG. IF shows Model 3 (1003), which may utilize data from various environmental sensors, operational systems, and road condition assessments to calculate the optimal speed and acceleration for different segments of the mining haul roads. Possible inputs for Model 3 (1003) are listed by categories, as shown in FIG. IF. This model may aim to maximize safety and efficiency by dynamically adjusting speed and acceleration based on real-time environmental and road conditions.

[0164] The inputs to Model 3 (1003) may include route information (1051), maximum speed and acceleration (1045), environmental data (1035) (such as temperature, wind, precipitation, and dust storms), truck parameters (e.g., tyre performance and strut data (1092)), road conditions data (including road curvature, rolling resistance, and grade), and telematics (1119) (like gradeability, gradient per segment, and current speed).

[0165] The outputs from Model 3 (1003) may include the maximum speed and acceleration (1045) per road segment, the optimal speed limit and acceleration limit (1048), and recommendations for optimal braking strategies.

[0166] Model 3 (1003) may employ a combination of machine learning algorithms for pattern recognition and predictive analytics, coupled with optimisation algorithms to calculate the safest and most efficient speed and acceleration settings. Furthermore,it may receive data from loT sensors and mine software through real-time data streams. Model 3 (1003) may employ edge computing (1022) for immediate processing and decision-making on speed adjustments, and cloud computing (1020) for aggregating and analysing historical data patterns.

[0167] FIG. 1G shows Model 4 (1004), which may utilize vision-based data along with truck telemetry to assess and predict the conditions of haul road segments. This model may leverage machine learning techniques, particularly image processing algorithms, to analyse road surface images and telematics (1119) to estimate road conditions effectively. This analysis may aid in proactive maintenance decisions and ensures optimal safety and efficiency for haul operations.

[0168] The inputs to Model 4 (1004) may include truck strut sensor data, coordinates (1052), road surface vision images, and tire-road interaction data - collectively referred to as truck sensors road images (1053).

[0169] The outputs from Model 4 (1004) may include road surface conditions (1062), rugosity (1060) (estimated from road surface images), rolling resistance (estimated from road surface images), and request for a grader (1061) (triggered based on assessed road conditions).

[0170] Model 4 (1004) may use convolutional neural networks (CNNs) for image processing to analyse road surface conditions. Model 4 (1004) may also integrate data fusion techniques to combine sensor readings and visual data for more accurate condition assessments.

[0171] Model 4 (1004) may receive real-time data from loT sensors (1043) placed on trucks and along the road and may use edge computing (1022) for immediate image processing and decision-making at the site, with cloud computing (1020) for data storage and deeper analysis.

[0172] FIG. 1H shows Model 5 (1005), which may integrate advanced control strategies to optimize the electric drive system (e-drive) of hybrid trucks. Model 5 (1005) may use intelligent PLC (Programmable Logic Controller) techniques (1068), thus enhancing operational efficiency across varying torque demands and maximizing the benefits of regenerative braking. The model may dynamically adjust power outputto meet the real-time driving conditions and load demands, thereby optimizing energy use and improving vehicle performance.

[0173] The inputs to Model 5 (1005) may include battery status (1063), electric motor performance (1064), motor controller (inverter) settings, gearbox conditions (1065), BMS, regenerative breaking, and outputs from Model 1 (1001).

[0174] The outputs from Model 5 (1005) may include power optimization (1070) (for heavy, medium, and low torque scenarios), power optimisation in regenerative braking mode, command execution (1071) for the electric drive controller, and performance reporting (1072) for the haul truck's e-drive system.

[0175] Model 5 (1005) may employ real-time control algorithms that can process input data from the PLC to dynamically optimize the power settings for different conditions. Techniques like PID control, fuzzy logic, or machine learning-based predictive control may be utilized to fine-tune the e-drive performance.

[0176] Model 5 (1005) may be directly connected to the PLCs associated with the truck's e-drive system and may use specific communication protocols for real-time data exchange and command transmission between the PLC and the Al model via the Control and Communication Interface Layer (184). Furthermore, Model 5 (1005) may leverage edge computing (1022) for real-time processing and response, situated within the truck's own computing system.

[0177] FIG. II shows Model 6 (1006), “Generative Al,” which may act as an intelligent advisor within the Mobile Al Layer (186), utilising advanced generative algorithms to synthesise and extrapolate from extensive datasets. This model may enhance decision-making by producing novel patterns, predictions, and operational strategies, integrating insights from both the fusion mechanism (1079) and the Master Al, z.e., Model 7 (1007). Its primary role may be to refine and generate actionable information, guiding mine planning, production reporting, geology, and reconciliation through smarter, data-driven recommendations.

[0178] Inputs to Model 6 (1006) may include combined and refined data (1074) from the fusion mechanism within the Mobile Al Layer (186) as well as analytical outputs from Model 7 (1007).

[0179] The outputs from Model 6 (1006) may include route recommendations (1076) (automatically generating and alerting to more efficient routing options) and an enhanced overall decision support (1077) for haul truck operations. Outputs may also include providing a customized generative Al interface for analysing and presenting information (information analysis (1078)).

[0180] Generative Al in Model 6 (1006) may involve using deep learning techniques like Generative Adversarial Networks (GANs) or Variational Autoencoders (VAEs) that are well-suited for generating new data instances by learning the distribution of input data.

[0181] Model 6 (1006) may allow for direct integration with the Mobile Al Layer's data processing streams, and it may utilize APIs (1021) for communicating with other models within the Mobile Al Layer and accessing external data from mine planning software. Model 6 (1006) may further leverage cloud computing (1020) and edge computing (1022) for processing large datasets and storing model outputs.

[0182] FIG. 1J shows Model 7 (1007), the “Master Al,” which may act as an overarching decision agent, capable of resolving discrepancies or conflicts among outputs from various models within the Mobile Al Layer. Model 7 (1007) may leverage advanced analytics accessed via cloud APIs (1021), possibly through third- party database managers. Master Al's intelligent feedback mechanism, the “Fusion Mechanism (FM),” (1079) may allow for continuous refinement of operational algorithms, dynamically adjusting decision-making thresholds and enhancing data collection protocols based on real-time performance feedback. This may ensure that operational decisions are informed by the most up-to-date and relevant data, maintaining the system’s cutting-edge performance.

[0183] Inputs to Model 7 (1007) may include a fleet management system (1087), mine planning software (1081), customer relationship management (CRM) (1082), a data warehouse (1083), production software (1086), an energy management system (1085), and fusion mechanism (FM) outputs (1084) (consensus data and model outputs).

[0184] Outputs from Model 7 (1007) may include ultimate decision-making (i.e., high- order decisions that guide operational parameters and strategic planning), input to generative Al (and FM to refine data), and input to user dashboards.

[0185] The Master Al Model (1007) may employ sophisticated data processing architectures including deep learning and decision trees to analyse and synthesize data from various sources. It may be designed to handle large-scale data integration and complex decision-making processes. The Master Al Model (1007) may directly integrate with mine software systems (1033) through APIs (1021). It may also be hosted on an edge device for real-time edge computing (1022) but also connected to cloud computing (1020) for additional computational support and data storage. Furthermore, the Master Al Model (1007) may utilize APIs (1021) for seamless integration with various fleet software applications, enabling comprehensive data analysis (1094) and decision-making.

[0186] The Master Al Model (1007) may handle all communication with mine software applications. Hence, if this model needs input from those applications, a Master Al Model (1007) feedback loop may be necessary. The Master Al Model (1007) may provide feedback to all the models that incorporate mine software data. The outputs of the Master Al Model (1007) may include ultimate decision regarding haul truck operation, input for generative Al and the FM, and information for user dashboards.

[0187] FIG. IK shows Model 8 (1008), Predictive Maintenance, which may employ data-driven strategies to anticipate maintenance needs before failures occur, improves vehicle uptime, and operational efficiency. This model analyses a comprehensive range of inputs concerning vehicle wear and tear, driving patterns, and road conditions to schedule timely maintenance activities. This proactive approach reduces unexpected breakdowns and extends the lifespan of critical vehicle components.

[0188] Inputs to Model 8 (1008) may include vehicle components' wear and tear data (1091) (e.g., engine parts, brakes, tires, suspension system), performance data (e.g., speed, acceleration, braking intensity, load weight), environmental and road conditions (e.g., weather, road usage), and fleet utilization data (from Model 9 (1009) and historical records).

[0189] The outputs from Model 8 (1008) may include generating automated alerts for vehicle maintenance (z.e., maintenance management (1096) based on predictive wear and tear analyses. This includes scheduling interventions before failures (z.e., predictive failures (1095)) occur, enhancing fleet reliability and uptime. The outputs further may include scheduling predictive maintenance for critical truck systems like the fuel system, based on data-driven predictions of wear and potential failures. They further may include enabling fleet managers to quickly approve maintenance tasks and route adjustments with a single click, streamlining the decision-making process. They further may include monitoring wear and tear indicators to identify early signs of component fatigue or failure, allowing for timely interventions. This proactive approach prevents larger issues and reduces the risk of unexpected breakdowns. They further may include utilizing data from wear and tear indicators to optimally schedule repairs and replacements (z.e., data analysis (1094)). This strategic planning may help avoid both premature maintenance and overdue service, balancing operational needs with maintenance costs. The outputs further may include more efficient allocation of resources, focusing efforts and expenditures on actual needs rather than routine schedules, thereby reducing unnecessary maintenance expenses. Continuing, the outputs may additionally include ensuring that the suspension system and other critical components are maintained in good condition, the model enhances the safety and reliability of haul truck operations, contributing to overall operational safety. The outputs further may include better forecasting of part replacement needs, managing inventory more efficiently and ensuring that critical components are available when needed. Finally, the outputs may include maintaining the suspension system and other key truck components in optimal condition. This ensures that haul trucks (105) operate at peak performance, improving productivity and reducing operational costs.

[0190] Model 8 (1008) may use statistical modelling and machine learning algorithms, such as regression analysis or neural networks, to predict maintenance needs based on patterns detected in the input data. The architecture may also incorporate decisionmaking algorithms to prioritize and schedule maintenance tasks. The data sources for Model 8 (1008) may include on-board loT sensors (1043) devices that access fleet software systems through APIs (1021) for real-time and historical data (1030). Model 8 (1008) may also utilize cloud computing (1020) for aggregating large datasets andrunning complex analytics. Furthermore, Model 8 (1008) may employ edge devices for immediate data processing and issuing alerts.

[0191] FIG. IL shows Model 9 (1009) implements a “Vote to Reduce or Increase Truck Numbers.” Model 9 (1009) may be designed to democratically manage the fleet of trucks by allowing trucks to vote on the fleet utilization. This model may process operational data, mine plans (1102), and payload types (1101) to make informed decisions about whether trucks should be temporarily stood down or reactivated (e.g., update the number of haul trucks (105)). The model may ensure that at least two- thirds of the trucks agree before actioning any changes, optimizing fleet efficiency and responsiveness to real-time site needs.

[0192] The inputs to Model 9 (1009) may include site speed and acceleration limits (from Model 2 (1002)), mine plan and mine plan targets (from mine software (1044)), payload types (including waste (115), ore (116), tailings information from loT sensors (1043) and mine software (1044)), and truck modelling data (from Model 1 (1001)).

[0193] The outputs from Model 9 (1009) may include truck fleet utilization change (1104) (the voting results determining if trucks stand down or up), updated number of haul trucks (105) in operation (1105), and speed alert changes communicated to other trucks (1106).

[0194] The Al Model Architecture of Model 9 (1009) may include employ a distributed decision-making architecture, using consensus algorithms or similar methods to aggregate and process votes from individual trucks. This approach may enhance the adaptability and scalability of fleet management.

[0195] Model 9 (1009) offers a potential system integration benefit: It may receive input from other Al models, mine software systems (1044), and loT sensors (1043) and use advanced communication protocols to enable secure and reliable data exchange among trucks and between trucks and the central system. Furthermore, it may utilize edge computing (1022) on each truck to process votes locally and communicate with the central system for final decision-making.

[0196] FIG. IM shows Model 10 (1010), which is a critical analytics tool that may compare predicted data from predictive models with actual operational data. It may focus on evaluating predictions related to productivity, emissions, volume moved, andcarry back against real-world outcomes. This comparison may help in assessing the accuracy and effectiveness of predictive algorithms, facilitating continuous improvements in predictive modelling and operational adjustments.

[0197] The inputs to Model 10 (1010) may include optimal productivity, emissions, volume, and carry back (1107) from Model 1 (1001); predicted productivity targets (1108) (from mine engineering / mine software); payload calculation, cycle time, loading overrides, material overrides, and truck parameters (1109) (from the Shovel); as well as truck modelling data from Model 1 (1001).

[0198] The outputs from Model 10 (1010) may include the amount of material moved over energy (1112) and comparison results, such as, e.g., predicted productivity vs. actual (1110), predicted emissions vs. actual (1113), predicted volume moved vs. actual (1111), and calculated carry back vs. actual (1114).

[0199] The Al Model Architecture of Model 10 (1010) may use statistical analysis tools and discrepancy metrics (like mean squared error, R-squared) to measure the differences between predicted and actual figures. It may also utilize visualization tools to present these discrepancies clearly for analysis by decision-makers.

[0200] Potential system integration benefits from Model 10 (1010) may include receiving predictive output directly from Model 1 (1001) and operational data from mine software (1044), utilizing APIs (1021) to fetch data efficiently and securely from various sources within the mining operation's digital infrastructure, and employing cloud-based analytics platforms (z.e., cloud computing (1020)) for heavy computational tasks and data comparison.

[0201] FIG. IN shows Model 11 (1011), which may focus on optimizing fuel consumption and operational efficiency for hybrid haul trucks by analysing various data points related to energy usage, route scenarios, and truck telematics (1119). Model 11 (1011) may integrate this data to propose optimal routes and fleet management decisions that align with energy consumption targets and carbon emission goals.

[0202] The inputs to Model 11 (1011) may include production metrics from Model 10 (1010); carbon emissions (actual and planned); energy consumption rates and fuel efficiency of current fleet operations; data on route gradients, gradeability, andoperational scenarios affecting energy usage; market energy costs and specific truck operational metrics like speed profiles and traction forces, truck modelling data from Model 1 (1001); engine load and efficiency from maintenance Model 8 (1008); and road conditions from Model 4 (1004).

[0203] The outputs from Model 11 (1011) may include optimised routes (1123) based on energy efficiency and carbon emission targets; fuel consumption rates and energy usage (1120) for hybrid mode; regenerative energy capture metrics; suggested number of trucks per route to optimize energy usage; and production metrics and carbon emission calculations for hybrid operations.

[0204] The Al Model Architecture may allow for the use of complex algorithms that incorporate predictive analytics, optimization, and possibly machine learning to analyse input data and generate efficient operational strategies. The architecture may also involve multi-objective optimization techniques to balance between fuel efficiency, production output, and environmental impact.

[0205] Potential system integration benefits may include integrating with mine software (1044) for operational data, and loT sensors (1043) for real-time truck telemetry. They may also include cloud computing (1020) for large-scale data analysis (1094) and storage, with edge computing (1022) on trucks for real-time data processing and immediate operational adjustments. Furthermore, APIs (1021) may be employed for seamless data fetching and integration with external systems like energy markets or third-party environmental databases.

[0206] FIG. IP shows Model 12 (1012), “Autonomous Operation,” which may leverage Al to enhance the autonomy of mining vehicles, thus improving energy management and operational efficiency significantly. By integrating sensors and loT sensors (1043), Model 12 (1012) may enable trucks to autonomously navigate and operate within the mining environment, making real-time decisions about speed, routing, and other critical operational parameters. The autonomous operation system may be designed to enhance safety, compliance, and efficiency, adapting dynamically to the mining environment (z.e., prediction and response).

[0207] The inputs to Model 12 (1012) may include those related to vehicle performance and navigation data, such as the output from Model 1 (1001), GPS data (1139), andvehicle systems (1130). These data may be in the format of truck modelling (from Model 1 (1001)) for operational baselines, sensor data (speed, acceleration, braking intensity) for real-time vehicle dynamics, geolocation information from GPS (1145) for precise navigation, vehicle dynamics data for understanding vehicle behaviour and constraints.

[0208] The inputs may also include those related to environmental and operational context data, such as LiDAR (1138), cameras, aerial robots (1144), mine maps, geo data (1148), and environmental data (1035). These data may be in the format of LiDAR (1138) and camera data for obstacle detection and navigation, aerial imagery for topographical mapping and situational awareness, digital map data and geological data from mine maps and geo-data for terrain navigation, and weather data to adjust operations based on environmental conditions like rain, wind, and temperature.

[0209] The inputs to Model 12 (1012) may include those related to communication and management data (1036), such as data integration (1037) from loT devices, communication networks (1146), traffic systems (1149), and operation systems. These data may be in the format of data from loT sensors (1043) for device and equipment status within the mining site, communication system data to ensure robust data transmission and fleet coordination, and traffic management data (1137) for managing vehicle flow and avoiding bottlenecks.

[0210] The outputs from Model 12 (1012) may include operational outputs, such as autonomous vehicle routing (optimizing routes based on real-time data to enhance operational efficiency), autonomous vehicle operation (managing vehicle operations including driving, loading, and dumping autonomously), and traffic management (coordinating vehicle traffic to optimise flow and reduce congestion).

[0211] The outputs from Model 12 (1012) may also include safety and compliance outputs, such as automatic alerts (issuing warnings and notifications to operators and fleet managers regarding potential issues or adjustments needed for safety and efficiency), and speed control (dynamically adjusting vehicle speed based on operational conditions, traffic data, and environmental factors).

[0212] Finally, the outputs from Model 12 (1012) may also include predictive and responsive outputs such as unpredictable physical adjustments (reacting to immediateunforeseen environmental changes to maintain operational safety and efficiency), and predictable physical adjustments (planning and adjusting operations based on known variables and predicted conditions).

[0213] The Al Model Architecture for Model 12 (1012) may use a combination of deep learning for perception tasks (image, LIDAR data processing), reinforcement learning for decision-making under uncertainty, and classical path-planning algorithms like A* or Dijkstra's for route optimisation.

[0214] Model 12 (1012) may offer the possibility for system integration of data sources (integrating with on-board sensors, loT frameworks, and mine software systems (1044) through APIs (1021)), edge computing (1022) (utilizing on-board computing power for immediate data processing and decision-making), and cloud computing (1020) (employing cloud services for data aggregation, long-term planning, and backup decision support.

[0215] FIGs. 2 and 2A-2H shows an embodiment of the inputs to various models in the Al and Autonomy System. A single spreadsheet has been divided into several figures for these figures. Each line on each of the figures lists an input. The columns of the table correspond to the system integration (z.e., in what part of the system the input exists), the data source, the data type, which model the input enters into, the input name, a brief description of the input, the model that uses the input, a possible implementation of the model that uses the input, the model number of the model that uses the input, an output that results from using the input, and the output description. The upper lefthand portion of the spreadsheet is shown in FIG. 2. The next portion to the right of the spreadsheet is shown in FIG. 2A (z.e., for the same rows as shown in FIG. 2). The top right-most portion of the spreadsheet is shown in FIG. 2B. FIG. 2C shows the next collection of rows of the spreadsheet on the left-hand side beneath FIG. 2. FIGs. 2D and 2E, respectively, are the next collections of columns for the same rows shown in FIG. 2C. FIG. 2F is the bottom left portion of the spreadsheet. FIGs. 2G and 2H are the next portions of columns, from left to right, for the same rows as shown in FIG. 2F. This list may be different for a different embodiment of the Al and Autonomy System. This list is only for illustrative purposes.

[0216] FIG. 3 A shows a network diagram of inputs and outputs. This diagram lists the 12 models, their inputs, and their outputs. Inputs that are common to many modelsare marked differently than inputs that only enter into a single model. Each model is numbered in the diagram. The FM and the Master Al Model (1007) are indicated along with model number 10 and 12. FIG. 3B presents the same information as FIG. 3 A, but in list format. Each row presents a particular model, and the columns shows which other models provide input to that particular model.

[0217] FIG. 3C presents a list that aids in understanding the various attributes detailed in this document. In particular, attributes are put in four categories pertaining to trucks, shovels, routes, and block models.

[0218] Since Al methods and, in particular, machine learning methods, are used in many of the models of the Al and Autonomy System, a brief review will be included here to give examples of the some of the possible forms they may take.

[0219] Artificial intelligence, broadly defined, is the extraction of patterns and insights from data. The phrases “artificial intelligence,” “machine learning,” “deep learning,” and “pattern recognition” are often convoluted, interchanged, and used synonymously throughout the literature. This ambiguity arises because the field of “extracting patterns and insights from data” was developed simultaneously and disjointedly among a number of classical arts like mathematics, statistics, and computer science. For consistency, the term artificial intelligence (Al), will be adopted herein. However, one skilled in the art will recognize that the concepts and methods detailed hereafter are not limited by this choice of nomenclature.

[0220] Al model types may include, but are not limited to, neural networks, random forests, generalized linear models, and Bayesian regression. A general feature of Al models is the type of “learning” that the Al model attempts. Learning, in this context, simply refers to a method for extracting patterns from data. Artificial intelligence models may undergo supervised learning, which requires significant assistance on the part of humans in providing labels for data, or description, for which the Al model may make reference to. In contrast, unsupervised learning does not require human assistance but may require greater data volume in order for a particular pattern to be recognized or extracted. A third type of learning is sometimes referred to as semisupervised learning which combines elements of both supervised and unsupervised learning. An example of a semi-supervised learning model is reinforcement learning. Al model types are usually associated with additional “hyperparameters” whichfurther describe the model. For example, hyperparameters providing further detail about a neural network may include, but are not limited to, the number of layers in the neural network, choice of activation functions, inclusion of batch normalization layers, and regularization strength. The selection of hyperparameters surrounding a model is referred to as selecting the model “architecture.” Generally, multiple model types and associated hyperparameters are tested and the model type and hyperparameters that yield the greatest predictive performance on a hold-out set of data is selected.

[0221] In accordance with one or more embodiments, the Al and Autonomy System may include a neural network. A diagram of a neural network is shown in FIG. 4. From a visual perspective, a neural network (400) may be graphically depicted as being composed of nodes (402), where here a circle represents a node, and edges (404), represented as directed lines or arrows. The nodes (402) may be grouped to form “layers” (405) which are nodes (402) operating a specific depth or specific location within the neural network (400). A neural network (400) will have at least two layers (405), where the first layer (408) is considered the “input layer” and the last layer (414) is the “output layer.” Generally, there is one input node (402) per input variable and one output node (402) per output variable. FIG. 4 displays four layers (408, 400, 402, 404) of nodes (402) where the nodes (402) are grouped into columns, however, the grouping need not be as shown in FIG. 4. The edges (404) connect the nodes (402). Edges (404) may connect, or not connect, to any node(s) (402) regardless of which layer (405) the node(s) (402) is in. That is, the nodes (402) may be “fully,” “sparsely,” and “residually” connected. Layers between the input layer (408) and output layer (414), such as the intermediate layers (410) and (412) shown in FIG. 4, are usually described as a “hidden layer.” A neural network (400) may have zero or more hidden layers (410, 412) and a neural network (400) with at least one hidden layer (410, 412) may be described as a “deep” neural network or as a “deep learning method.”

[0222] Nodes (402) and edges (404) each have additional characteristics. First, each edge is characterized by a numerical value. The edge numerical values, or even the edges (404) themselves, may often be referred to as “weights” or “parameters.” While training a neural network (400), numerical values are assigned to each edge (404). Inaddition, every node (402) is characterized by a numerical variable and an activation function. Activation functions traditionally follow the form:(4), where i is an index that spans the set of “incoming” nodes (402) and edges (404) and f is a pre-selected function. Activation functions are not limited to any particular functional class. Some functions for f may include the linear function (x) = x,sigmoid function / (x) =e-x, and rectified linear unit function / (x) = max(0, x),however, many additional functions are commonly employed. Every node (402) in a neural network (400) may be characterized by a different activation function.

[0223] When the neural network (400) receives an input, the input is propagated through the network, starting at the input layer (408), from one node (402) to another, according to the activation functions and incoming node (402) values and edge (404) values to compute a value for each subsequent node (402). That is, the numerical value for each node (402) may change for each input that it receives. Nodes (402) may also be assigned fixed numerical values, such as the value of 1, that are not affected by the input or altered according to edge (404) values and activation functions. Fixed nodes (402) may be referred to as “biases” or “bias nodes” (406), represented in FIG. 4 with a dashed circle.

[0224] In some implementations, the neural network (400) may contain specialized layers (405), such as a normalization layer, or layers that perform additional connection procedures, like concatenation. One skilled in the art will appreciate that these alterations do not exceed the scope of this disclosure.

[0225] As noted, the training procedure for the neural network (400) includes assigning values to the edges (404) and updating them. To begin training, the edges (404) are assigned initial values. These values may be assigned randomly, assigned according to a prescribed distribution, assigned manually, or by some other assignment mechanism. Once edge (404) values have been initialized, the neural network (400) may act as a function, such that it may receive inputs and produce an output. As such, at least one input is propagated through the neural network (400) to produce an output.Recall, that a given data set will be composed of inputs and associated target(s), where the target(s) represent the “ground truth,” or the otherwise desired output.

[0226] In accordance with one or more embodiments, the input of the neural network (400) is the real-time haul truck data, environment data, haul load efficiency data, and the mine plan, each of which may be pre-processed. For simplicity, consider again the scenario where the mine plan specifies only a mining production target. The objective may then be to determine the optimal haul truck velocity to achieve the mining production target in the shortest amount of time, or while consuming the least amount of fuel, or according to other criteria. The neural network (400) would then act as a function of the input data to determine the predicted mining production given the values of the input data. In the context of training, a set of training data may be used where the training data includes various mining productions achieved for various values of the input data. In one or more embodiments, a set of training data may be obtained by recording the mining production achieved (as well as other variables, such as fuel used or carbon dioxide emitted, elapsed time, etc.) while operating the haul trucks (105) at the mine at various velocities and accelerations and across various routes.

[0227] Training the neural network (400) involves iteratively comparing the output predicted by the neural network (400) with the associated ground truth. Continuing from the example above, the inputs from set of training data may be processed by the neural network (400) to obtain the predicted mining production which is then compared with the ground truth mining production also included in the training data. The comparison of the neural network (400) output to the ground truth is typically characterized by a so-called “loss function,” although other names may be used such as “error function,” “misfit function,” and “cost function.” Many different loss functions are known in the art, and a popular choice is the mean-squared-error function. The primary purpose of the loss function is to provide a numerical evaluation of the similarity between the neural network (400) output and the associated ground truth. The loss function can also be constructed to impose additional constraints on the values assumed by the edges (404), for example, by adding a penalty term, which may be physics-based, or a regularization term. Generally, the goal of a training procedure is to modify the edge (404) values suchthat the neural network (400) output and is similar to the ground truth over the set training data inputs. The loss function is used to guide changes made to the edge (404) values, typically through a process called “backpropagation.”

[0228] While a full review of the backpropagation process exceeds the scope of this disclosure, the following provides a brief summary. Backpropagation includes computing the gradient (or multivariable derivative) of the loss function across the edge (404) values. The gradient indicates the direction of change in the edge (404) values that results in the greatest change to the loss function. Because the gradient is computed from one edge value to another, the gradient is considered local to edge (404) values. Accordingly, edge (404) values are typically updated by a “step” in the direction indicated by the gradient following their locality. The step size, or magnitude of change of the edge value, is often referred to as the “learning rate” and need not remain fixed during the training process. As the neural network (400) is trained, the step size may change depending on the training data. Additionally, the step size and direction may be informed by previously seen edge (404) values or previously computed gradients that have been recorded. Methods relying on previously seen edge values or computed gradients for determining the step direction may be referred to as “momentum” based methods.

[0229] Once the edge (404) values have been updated, or modified from their initial values (for example, through a backpropagation step), the neural network (400) will likely produce different outputs. The new outputs are again compared with the ground truth, and the process repeats. The procedure of propagating at least one input through the neural network (400), comparing the neural network (400) output with the associated ground truth via computation of a loss function, computing the gradient of the loss function with respect to the edge (404) values, and updating the edge (404) values with a step guided by the gradient, is repeated until a termination criterion is reached. Common termination criteria include reaching a fixed number of edge (404) updates or iterations (otherwise known as an iteration counter), employing a diminishing learning rate, noting no appreciable change in the loss function between iterations, and reaching a specified performance metric as evaluated on the training data or on a separate hold-out data set. Once the termination criterion is satisfied, and the edge (404) values are no longer intended to be altered, the neural network (400) issaid to be “trained.” Continuing from the previous example with a mine plan specifying a mining production target, the neural network (400) is considered train when it can accurately (within some predetermined limit) determine hauling performances according to the training data.

[0230] To summarize, in one or more embodiments, the Al and Autonomy System includes a neural network (400). However, it is emphasized that the Al and Autonomy System may employ other or additional Al model types. For example, the Al and Autonomy System may be of any ML model type known in the art, such as a decision tree (or ensemble of decision trees such as a random forest or gradient boosted trees), a support vector machine, a regression model, an algorithm using k-means clustering or mean-shift, an algorithm using Gaussian processing techniques, or a reinforcement learning model. In one or more embodiments, the Al and Autonomy System includes an optimizer that is used alongside the neural network (400) or other Al model architecture.

[0231] In accordance with one or more embodiments, an optimization algorithm or method (or simply optimizer) is used to invert, or intelligently probe the parameter space mapped by the Al and Autonomy System to determine parameters of interest. The neural network (400) of the Al and Autonomy System, or other employed Al model type, may then be used to determine parameters. The optimizer is used to efficiently explore the parameter space.

[0232] A commonly used non-linear optimizer is the genetic algorithm (GA). An overview of the typical steps used in the genetic algorithm (GA) is provided in FIG. 5. The genetic algorithm (500) begins by generating an initial population or multiple populations (502). A population consists of one or more “individuals.” In the context of the genetic algorithm (500), an individual is a single representation, or encoding, of the function parameters over which optimization is to occur. The number of individuals in a population, and the method of initially generating individuals, are hyperparameters chosen by the user. When multiple populations are used, commonly referred to as an “island” scheme, the populations need not be initialized using the same set of hyperparameters.

[0233] Once a population(s) has been generated (502), the “fitness” of every individual in the population(s) is evaluated (504). After the fitness of each individual isdetermined, a stopping criterion is checked (506). Many stopping criteria exist, including, but not limited to, the number of iterations the genetic algorithm has run, the maximum or minimum fitness score achieved by an individual, the relative change in fitness scores between iterations, and the similarity of individuals in a population, or combinations of these criteria. If a stopping criterion is met, typically the most fit individual(s) seen during the genetic algorithm process is selected (512) as representing the optimal parameter values and the algorithm terminates. However, even if the genetic algorithm continues, one or more individuals from the population(s) may be selected (508) for creating the next “generation” of individuals. This selection may be done by simply selecting the portion of the population with the highest fitness scores, or through a tournament process, or other selection mechanism.

[0234] Once individuals have been selected (508), the individuals may be propagated through without alteration, removed, or modified through so-called crossover, mutation, and differential evolution methods to create “offspring” (510). The offspring are themselves individuals; that is, new representations, or encodings, of the function parameters. It is noted that many evolutionary methods exist to create offspring and the preceding list is not all-inclusive and should be considered nonlimiting. The offspring are then evaluated for fitness (504) and the process is repeated until the genetic algorithm stopping criterion is met.

[0235] Again, the description of the genetic algorithm (GA) provided in FIG. 5 is generalized and one skilled in the art will appreciate that many modifications can be made, and are regularly made, to the genetic algorithm (GA) without departing from its intended scope. For example, an enhanced genetic algorithm (eGA) is developed by including additional features such as: enforcing deliberately diverse initial populations; using heterogenous hyperparameters for each population; dynamically updating or scheduling changes to the selection process and offspring creation process; using a unidirectional migration policy between populations; adding additional checks, such as a premature convergence check; using a self-adaptive differential evolution method; and performing a localized exhaustive search in regions of stagnation or saturation. Other linear and non-linear optimizers may also be employed to determine the optimal parameter values using the neural network (400) (or other Al model architecture) of the Al and Autonomy System. The optimizercould be a Bayesian-based optimizer, such as a Markov Chain Monte Carlo (MCMC) method, which elects new parameters based on an analysis of the updated posterior distribution.

[0236] The Al and Autonomy System may be implemented on a computer that is similar to the computer depicted in FIG. 10 and described below. The computer for implementing the Al and Autonomy System may either be centrally located or distributed. For example, briefly returning to FIG. 1, haul trucks (105) may be connected to the wireless communication system (113) and transmit their real-time haul truck data to a base station (125). The Al and Autonomy System may be located on a computer at the base station (125), along with the mine plan, and haul load efficiency data. The computer may further be configured to receive environment data obtained by weather equipment (129). In this case, the computer may be said to be centrally located. Alternatively, the computer implementing the Al and Autonomy System may be distributed among one or more haul trucks (105), the base station (125), and any combination thereof. That is, in some embodiments, each haul truck (105) may be equipped with its own computer that is similar to the computer depicted in FIG. 10 and described below, and each haul truck (105) may communicate with every other haul truck (105), in addition to the base station (125), using the wireless communication system (113).

[0237] Once the optimal parameter values of the Al and Autonomy System are determined, the operation of at least one haul truck (105) in the fleet of haul trucks may be adjusted. In embodiments where the Al and Autonomy System determines an optimal haul truck velocity for every haul truck (105) in the fleet, then the velocity of each haul truck (105) in the fleet may be adjusted to its respective optimal haul truck velocity. In embodiments where the Al and Autonomy System determines the optimal haul truck velocity for one or more particular haul routes (101), or for one or more particular haul route segments, then the velocity of one or more haul trucks (105) may be adjusted when the one or more haul trucks (105) drives along the respective one or more routes or one or more route segments. The above description applies similarly to implementing the optimal haul truck acceleration. That is, the acceleration of at least one haul truck (105), up to every haul truck (105) in the fleet, may be adjusted according to the optimal haul truck acceleration, and the one morehaul trucks (105) may drive according to the optimal haul truck acceleration along particular routes or route segments.

[0238] In one or more embodiments, haul truck velocities and accelerations are adjusted automatically. As described above, each haul truck (105) may be equipped with a computer. Whether the computer that implements the Al and Autonomy System is distributed among the haul trucks (105) or is centrally located, the haul truck computer may determine or receive the optimal haul truck velocity or the optimal haul truck acceleration. The haul truck computer may be interfaced with a powertrain control module (PCM) combining an engine control unit (ECU) and transmission control unit (TCU). When the haul truck computer receives or determines the optimal haul truck velocity, the haul truck computer may transmit commands to the PCM to adjust the operation of the haul truck (105) accordingly. In one or more embodiments, haul truck velocities and accelerations are adjusted through interaction between the Al and Autonomy System and a haul truck operator that drives the haul truck or by a remote operator. Rather than adjusting the haul truck velocity automatically, the haul truck operator driving the haul truck (105) may manually adjust the haul truck velocity. Alternatively, a remote operator not driving the haul truck (105) may adjust the haul truck velocity through remote or wireless communication with the haul truck (105). In embodiments where haul truck velocities are adjusted manually, then the real-time haul truck data recorded by the haul truck (105) may also include a measure of how closely the haul truck (105) adhered to the optimal haul truck velocity determined by the composite Al model. If a haul truck operator or remote operator fails to drive a given haul truck (105) according to the optimal haul truck velocity, the Al and Autonomy System may need to compensate for the bias or error introduced by the human operator.

[0239] As mentioned above, the Al models for Models 1 through 12 may be of any type known in the art. In one or more embodiments, an Al model may be a convolutional neural network. Although it is beyond the scope of the present disclosure, a brief description of convolution neural networks is provided below. A convolutional neural network (CNN) is a type of neural network similar to the neural network shown in FIG. 4 and its accompanying description. Like a neural network (400), a CNN may be graphically represented by a series of edges (404) and nodes(402) grouped to form layers. However, the configuration of a CNN may be better understood as structural groupings of weights such that weights within a group have a spatial relationship. CNNs are useful at recognizing and learning patterns when the input data also have a structural and spatial relationship, for example, a spatial relationship where one input is always considered “to the left” of another input. Route image data, including images or video of haul routes (101) for example, has such a structural relationship because each data element, or pixel, in an image or video of a haul route (101) has a spatial location. Consequently, a CNN may be trained to identify and learn the structural groupings of pixels collectively representing haul routes (101).

[0240] A structural grouping, or group, of weights may be referred to as a “filter” in the context of CNNs, or as the name suggests, as a “convolution kernel.” The number of weights in a filter (or kernel) is typically much less than the number of inputs, where here the number of inputs refers to the number of pixels in an image or video. A single group of weights is often used to discern features of a specific structural scale in the image or video. In a CNN, the filters can be thought as “sliding” over, or convolving with, the inputs to form an intermediate output or intermediate representation of the inputs which still possesses a structural relationship. Similar to ordinary neural networks (400), the intermediate outputs within layers or groupings of weights in the CNN are often further processed with an activation function. Several filters may be applied to the inputs to form an arbitrary number of intermediate representations depending on the available computing resources. Additional filters may be formed to operate on the intermediate representations creating yet further intermediate representations. Each filter, or group of weights, may be conceptually understood as being trained to learn features of the data on different scales, although the exact meaning of what a group “learns” may become more abstract with increasing layer depth. This process may be repeated as necessary. At some stage in the CNN, there is a “final” group of intermediate outputs at which no additional filters act the intermediate output. It is often common to restructure the final intermediate output through processes known as “flattening” or “ablation,” where the multidimensional output of the CNN (organized according to a structural or spatial grouping) is reduced to a single dimensional vector. The flattened representation may be passed to a neural network (400) including fully connected layers to produce a final output. A CNN istrained in a manner similar to an ordinary neural network (400): first filter weights and the edge (404) values of the internal neural network (400) are initialized, and then they are repeatedly updated with the backpropagation process in accordance with a loss function using a set of training data.

[0241] To process both the real-time haul truck data and the route image data to determine the conditions of a haul route (101), the Al model may use a convolutional neural network architecture with both convolutional layers that process the route image data as well as fully connected layers that process the final output of the convolutional layers as well as the real-time haul truck data.

[0242] FIG. 6A presents a schematic representation of a recurrent neural network (RNN). An RNN includes, at least, an RNN Block (610) and a recurrent connection (650). The RNN Block may be thought of as a function that accepts an Input (620) and a State (630) and produces an Output (640), similar to an ordinary neural network (400) that considers multiple inputs. As a general representation, the output of the RNN Block (610) may be written:Output = RNN Block( / nput, State).(5)The RNN Block (610) generally includes at least one matrix and at least one bias vector. The elements of the matrices and bias vectors are commonly referred to as “weights” or “parameters” in the literature such that the matrices may be referred to as weight matrices or parameter matrices without ambiguity. The weights of an RNN Block (610) may be contained in higher order tensors rather than in matrices or vectors to account for higher dimensional output, if necessary. For clarity, an illustrative example is given using Inputs (620) that are vectors or scalars such that the RNN Block (610) comprises at least one weight matrix and at least one bias vector. However, one with ordinary skill in the art will appreciate that this choice does not impose a limitation on the present disclosure. Consider an RNN Block (610) that has two weight matrices and a single bias vector and that are distinguished with an arbitrary naming nomenclature. A standard naming convention is to label one weight matrix W and the other U and to label the bias vector as b.

[0243] Using the terms defined above, the process of the RNN Block (610) defined by EQ. 5 may be generally written as:Output = RNN Block(input, state) = f(U ■ state + W ■ input + b),(6) where IV, U, and b are the weight matrices and bias vector of the RNN Block (610), respectively, and f is an activation function like the activation functions discussed in reference to FIG. 4. Example functions for f include the sigmoid function (%) = i+ e_. and the rectified linear unit (ReLU) function / (%) = max(0, x), however, many additional functions are commonly employed, and a person of ordinary skill in the art will appreciate these examples as non-limiting.

[0244] As noted, RNNs are well-suited for processing sequential or ordered data, for example, a time-series. This is because the ordering and sequential nature of the data plays a direct role in how the data are processed. In the RNN, the Input (620) may be a single part of a sequence or an entire sequence. As an illustration, consider a sequence including Y elements. Each element may be considered an input, and therefore the sequence may be expressed as sequence = [inputs input2, inputY-1, inputY], Each Input (620) (e.g., inputy of the sequence) may be a scalar, vector, matrix, or higher-order tensor, as previously stated. Recall that in one or more embodiments, an Al model may process time series data obtained by one or more haul trucks (105) in the form of haul truck data, environment data, and conditions of a haul route (101). In accordance with one or more embodiments, each input (or element of a sequence) is an array of measurements from the real-time haul truck data at a single time step, an array of measurements of environment data at a single time step, an array of measurements of the conditions of a haul route (101) or an array combining measurements from the real-time haul truck data, environment data, and conditions of a haul route (101).

[0245] To process the sequence, the RNN uses the RNN Block (610) to process the first ordered Input (620) of the sequence e.g., inputy) along with a State (630) according to EQ. 5 to produce an Output (640). The State (630) determines the type and size of the Output (640), and each may be of any type and size, such as a scalar, vector, matrix, or tensor of any rank. The State (630) is usually initialized with all ofits elements set to the value zero, but for the second element in the sequence (e.g., input2), the State (630) is set to the value of the Output (640) obtained after processing the first element. Put differently, like the first ordered Input (620), the second ordered Input (620) is processed according to EQ. 5, however, the State (630) received by the RNN Block (610) has the value of the Output (640) determined when processing the first ordered Input (620). The process of recurrently assigning the State (630) the value of the most recently (in terms of the ordered sequence) produced Output (640) is depicted with the recurrent connection (650) in FIG. 6A. Every Input (620) in the sequence is processed by the RNN Block (610) in the same manner. In some cases, every Input (620) and Output (640) pair within a sequence is stored for later processing and use. In other implementations, only the final Output (640) is stored or, following the example sequence above, only the Output (640) that is produced when the Input (620) is inputYis processed by the RNN Block (610), is stored.

[0246] FIG. 6B depicts an “unrolled” version of the RNN of FIG. 6A. Unrolling the RNN demonstrates how the sequential inputs, indexed by t in FIG. 6B, produce sequential outputs (also indexed by t) and how the state transforms and is passed through various inputs of the sequence. Although the “unrolled” depiction appears to show multiple RNN Blocks (610), each RNN Block (610) is in fact the same RNN Block (610) that includes the same weight matrices and bias vector. The RNN Block (610) is shown repeatedly to demonstrate recurrently processing the sequential inputs.

[0247] Training an RNN generally follows the training procedures outlined for other Al model types previously discussed. As previously stated, generally, training an Al model requires that inputs paired with known outputs (z.e., ground truth or “targets”) are passed to the Al model. Considering the RNN described above, the RNN receives a sequence, wherein the sequence can be partitioned into one or more sequential parts (Inputs (620) above), and maps the sequence to an overall output, which may also be a sequence. For example, the real-time haul truck data, conditions of a haul route (101), and environment data may be expressed as measurements at different spatial locations in a particular order (for example, measurements corresponding to each haul route segment in a haul route (101). Accordingly, the output may then be the maximum safe hauling at each segment in the route, which may also be representedas a sequence. To remove ambiguity and distinguish the final output of an RNN from any intermediate Outputs (640) produced by the RNN Block (610), the final output will be referred to herein as an RNN result. Thus, an RNN receives a sequence and returns an RNN result. Like the neural network (400) described in FIG. 4, the training procedure for the RNN includes assigning values to the weight matrices and bias vector of the RNN Block (610), z.e., the RNN weights. The RNN weights are initialized (or assigned values) randomly, according to a prescribed distribution, manually, or by some other mechanism. When processing sequences, the RNN result produced by the RNN is compared to the associated ground truth. Again, like the neural network (400) described in FIG. 4, the comparison of the RNN result to the target(s) is typically performed by a loss function. The goal of the training procedure is to modify the RNN weights to promote similarity between the RNN results and associated ground truth over the training dataset. The process of updating the values of the RNN weights according to the loss function may be achieved through a process referred to as “backpropagation through time” given that sequences often refer to elements corresponding to moments in time (although this need not be the case).

[0248] Long short-term memory networks (LSTMs) were designed to build on the strengths of RNNs while overcoming some of their limitations. For example, although RNNs perform well when processing ordered data, RNNs do not easily retain information between sequence elements that are significantly distant from each other in the sequence. This is because when processing a given element in the sequence, the state is always defined as the output obtained when processing the previous element. Consequently, elements close to each other in the sequence influence each other more strongly than distant elements. To address this limitation, LSTMs also utilize another data structure commonly referred to as the “carry” that serves a similar function to the state. Like the state and input, the carry may be a scalar, vector, matrix, or tensor of any rank depending on the application. Unlike the state, the carry is meant to retain the history of all previous elements in the sequence. Accordingly, a long short-term memory (LSTM) network may be considered a more complex embodiment of an RNN.

[0249] FIG. 7 depicts an unrolled LSTM where the internal components of the LSTM are illustrated as labelled abstractions, following FIG. 6B. Like the RNN describedin FIGs. 6A and 6B, the LSTM has a recurrent connection with the same property whereby the output produced by a single input in a sequence is used as the state of the subsequent input. For the following example and description of the LSTM, the input and carry will be considered vectors. The LSTM receives an input, state, and carry and produces an output and a new carry. According to the recurrent structure, the output and the new carry are passed to the LSTM as the state and carry for the subsequent input. This sequential and recurrent process, indexed by t, may be described functionally as: outputt, carry t) = LSTM Block(inputt, carryt-1(statet) = LSTM Block(inputt, carryt-1, outputt-x).(7)The LSTM block, like the RNN Block, includes at least one weight matrices and at least one bias vector.

[0250] As will be appreciated by a person of ordinary skill in the art, LSTMs may be configured in many ways. The processes depicted in FIG. 7 represent a common configuration. As shown in FIG. 7, an LSTM Block receives an input (input t), a state (state t), and a carry (carryt-i). Again, assuming that the inputs, carry, and outputs are all vectors, the weights of the LSTM Block may be sufficiently represented by eight matrices and four bias vectors. These matrices and vectors are conventionally referred to as W , Ui, W y, Uf, Wc, Uc, Wo, Uoand bt, bf, bc, b0, respectively. The processes of the LSTM Block are as follows. The first through third operations (760, 765, 770) follows the same structure as EQ. 6 for the RNN, but use different weights and bias vectors, respectively.

[0251] Block 760 represents the following first operation (760): f = a uf■ statet+ Wf ■ inputt+ by),(8) where a is an activation function applied elementwise and resulting in the vector f. Block 765 implements the following second operation with the same structure as the first operation (760) but with different weights and bias:where a2is an activation function (that may be the same or different to a- . The resulting vector is i. Block 770 implements the following third operation with the same structure as the second and third operations but again, with different weight and bias: c = a3( Uc■ statet+ Wc■ inputf+ bc),where a3is an activation function (that may be the same or different to either a or a2). The resulting vector is c. Each intermediate output obtained thus far represents a different encoding of the information obtained from the current input and current state. In block 775, vectors i and c are multiplied according to a fourth operation: z3= i O c,(11) where O indicates the Hadamard product (i.e., elementwise multiplication). Note that none of the first four operations (760, 765, 770, 775) have included the carry. However, in block 785 the carry vector from the previous sequential input (carryt-i) vector and the vector f are multiplied according to a fifth operation: z4= carryt-1O f.(12) The results of the operations of blocks 775 and 785 (z3and z4, respectively) are added together in block 780, a sixth operation (780), to form the new carry (carryt): carryf= z3+ z4.(13)

[0252] Thus, the first operation (760) through the sixth operation (780) may be understood as structure that enables the LSTM to determine how relevant all the accumulated previous information (represented by the carry) and how the most recent information (represented by the state) is to the current input.

[0253] In block 790, the current input and state vectors are processed according to a seventh operation:o = a4( Uo■ statet+ Wo■ inputt+ b0),(14) where a4is an activation function (that, again, may be unique or identical to any other used activation function). The result is the vector 6. In block 795, an eighth operation, the new carry (carryt) is processed by yet another activation function a5.The activation a5is commonly the hyperbolic tangent function but may be any known activation function. The eighth operation of block 795 may be represented as: z5= a5(carryt).(IV Finally, the output of the LSTM Block (output t) is determined in a ninth operation of block 798 by calculating the Hadamard product of z5and o, expressed mathematically: outputt= z5O o.(16)The output of the LSTM Block is used as the state vector for the subsequent input. Expressed in words, the new carry (carry ) was obtained by considering the current input and state alongside the previous carry and represents the information obtained during previous iterations that was found to be relevant to the current input and state. Subsequently, the new carry is processed alongside the current input and state to produce an output.

[0254] Similar to the RNN, the outputs of the LSTM Block obtained for each element of a sequence may be stored and further processed while in some implementations only the final output is retained. In addition, although the processes of the LSTM Block described above used vector inputs and outputs, an LSTM network may be applied to sequences of any dimensionality. A person of ordinary skill in the art will recognize that there are many alterations and variations that can be made to the general LSTM structure described herein, such that the description provided does not impose a limitation on the present disclosure.

[0255] FIG. 8A represents an illustration of a reinforcement learning scenario (800). Reinforcement learning is inspired by biological organisms that learn to achieve various tasks according to repeated action and observation of various outcomes,eventually learning a mapping that indicates which action to take in a given state in order to receive a particular award. In a reinforcement learning scenario (800), an agent (803) navigates an environment (812) using a policy “TT” (806) to determine an action “a,” (809) based on the observed state (808) of the environment, in order to eventually receive a reward “r” (818). After the agent (803) executes the action (809), the state (808) changes, and certain actions (809) made from certain states (808) will result in a reward (818). The reward (818) is constructed such that the agent (803) learns to achieve a predetermined objective over time by executing certain actions (809) that lead to the greatest cumulative reward (818). A computational challenge for both Al models and biological organisms is that not every action (809) necessarily results in a reward (818). That is, rewards (818) are often delayed. A common example for reinforcement learning is to consider a mouse (the agent (803)) that is trained to (learn a policy (806)) for navigating (the action (809)) a maze (the environment (812)) to find a block of cheese (the reward (818)). Because navigating the maze requires many steps, it may be difficult for the mouse to determine which initial actions led it to reach the block of cheese. Accordingly, it is often required to engineer intermediate rewards that map to the final reward (818). Reinforcement learning is sometimes referred to as “semi-supervised” learning because the agent may be left free to act (bounded in its actions only by its free parameters), although the final reward (818) and potentially intermediate rewards are predefined (or labeled and identified by engineers).

[0256] Policies (806) may be probabilistic in order to allow for variation between both the environment (812) and the agent (803). That is, given a state s (808), the policy (806) returns the probability of taking an action a (809). In this way, a policy (806) may be considered a type of Markov decision process. In some instances, the policy (806) is strictly deterministic, and every time the agent (803) observes a state s (808) the same action a (809) is always taken.

[0257] There are many approaches for determining which action (809) to take in a given state (808). First, it is observed that for many objectives, certain states (808) are more valuable than others because they more closely lead to rewards (818). Returning to the example of the mouse in a maze, being only a short distance or a small number of turns away from reaching the cheese may be considered a morevaluable state (808) than being several turns away and at a great distance. Accordingly, a mathematical function is typically used to define the value of a given state (808) as follows:EQ. 17 reads, the value V of the state s (808) given the policy n (806) is the expectation E of the sum of future rewards Rt(818) multiplied by a discount rate ytwhen starting at the state So= s and enacting the policy n (806). The discount rate is typically a number 0 < yt< 1 that allows for discounting future rewards (818) in favor of immediate rewards. The value function allows for different states (808) to be compared to each other for a given policy. Furthermore, by sampling different policies (806), an optimal policy may be found, whereby the optimal policy (806) obtains the maximum value each possible initial state s (808). Sampling different policies (806) may involve iteratively sampling initial states (808), and for each state (808), considering each possible action (809) and calculating the value function according to EQ. 17 for the given policy. Many algorithms are known in the art for utilizing the value function of EQ. 17 to find the optimal policy.

[0258] Determining the optimal policy (806) may also be guided through mathematically characterizing certain actions as more beneficial (useful, or important) than others. This is especially applicable when the probabilities of subsequent states (808) (or rewards (818)) arising are not initially known. That is, sampling policies (806) often requires a model of the environment (812), which may or may not be known. Accordingly, a mathematical function is typically used to define the quality (to distinguish from the “value” described above) of a given action (809) for a particular state as Q(st, at) where the subscript t denotes a particular state (808) and action (809). Using the quality function Q(s, a) to determine an optimal policy (806) is achieved as following (here subscripts have been dropped to indicate reference to all possible actions (809) and states (808)). An agent (803) begins at an initial state. Subsequently, at each time step t, the agent selects an action (809) and enters a new state. Taking a particular action a (809) updates the quality function as follows:y max Q(st+1, at)].(18)EQ. 18 recites that an action (809) atis taken resulting in the state (808) st+1and in a new estimate of the quality function Qnew(s, a). The quality function Qnew(s, a) is the sum of the current value of the quality function Qold(s, a) multiplied by 1 minus the learning rate a (where 0 < a < 1), the learning rate multiplied by the immediate reward (818) from taking the action Rt+1, and the learning rate multiplied by the discount rate y and the maximum Q value that can be obtained from the state st+1. The learning rate defines how important new information is compared to old information, while, again, the discount rate defines how important future rewards are relative to immediate rewards. Again, by sampling different states (808) and actions (809), the Q function may be used to determine the optimal policy.

[0259] In a reinforcement learning scenario (800), the environment (812) may have many states (808) and the agent (803) may have many possible actions (809). Accordingly, sampling different policies, actions, and states is computationally intractable for many realistic scenarios. What is needed is a method for mapping, for each state (808), the approximate reward (immediate and cumulative) obtained by taking a certain action (809). As previously noted, Al models are well-suited for approximating and learning complicated functions of many variables. Thus, the policy (806), which dictates which actions (809) an agent (803) should take to achieve a certain reward (818), may be represented by an Al model of any type known in the art, for example, a neural network (400).

[0260] FIG. 8B depicts a reinforcement learning scenario (850) where the policy (806) is represented by a deep neural network (DNN) (821) in accordance with one or more embodiments. Recall that, as discussed in reference to FIG. 4, a DNN is a neural network with one or more hidden layers. As neural networks (400) have already been discussed, both in their structure and implementation, a detailed description will not be provided herein. In brief, a DNN (821) may be trained to learn an optimal policy (806) through guidance of the Q function described above. A DNN (821) may be trained such that it considers a state (808) and determines every Q value associated with each possible action (809).

[0261] FIG. 9 presents the workflow of the method described above. In Step 901, operational data is obtained at a base station (125), wherein the operational datacomprises one or more selected from the following group: haul truck telemetry data from a fleet of haul trucks, route image data, environment data, haul load efficiency data, and payload data.

[0262] In Step 903, a mine plan of a mine is also obtained, the mine plan comprising mine plan targets and route constraints describing a set of haul routes (101) at the mine.

[0263] In Step 905, using an Al and Autonomy System comprised of Al models and connected to a base station (125), operational parameters are determined, selected from the following group: a CO2 emission, a productivity, an energy use, a haul truck velocity, and a haul truck acceleration, given the operational data.

[0264] In Step 907, the operational parameters of at least one haul truck (105) in the fleet of haul trucks are adjusted, automatically and according to the mine plan and the mine plan targets.

[0265] Embodiments may be implemented on a computer system. FIG. 10 is a block diagram of a computer system (1902) used to provide computational functionalities associated with described algorithms, methods, functions, processes, flows, and procedures as described in the instant disclosure, according to one or more embodiments. The illustrated computer (1902) is intended to encompass any computing device such as a server, desktop computer, laptop / notebook computer, wireless data port, smart phone, personal data assistant (PDA), tablet computing device, one or more processors within these devices, or any other suitable processing device such as an edge computing device, including both physical or virtual instances (or both) of the computing device. An edge computing device is a dedicated computing device that is, typically, physically adjacent to the process or control with which it interacts. For example, the Al model may be implemented on an edge computing device in order to quickly provide optimal sets of operation parameters and microbe parameters to associated devices or their controllers (e.g., control system).

[0266] Additionally, the computer (1902) may include a computer that includes an input device, such as a keypad, keyboard, touch screen, or other device that can accept user information, and an output device that conveys information associated with theoperation of the computer (1902), including digital data, visual, or audio information (or a combination of information), or a GUI.

[0267] The computer (1902) can serve in a role as a client, network component, a server, a database or other persistency, or any other component (or a combination of roles) of a computer system for performing the subject matter described in the instant disclosure. In some implementations, one or more components of the computer (1902) may be configured to operate within environments, including cloudcomputing-based, local, global, or other environment (or a combination of environments).

[0268] At a high level, the computer (1902) is an electronic computing device operable to receive, transmit, process, store, or manage data and information associated with the described subject matter. According to some implementations, the computer (1902) may also include or be communicably coupled with an application server, e- mail server, web server, caching server, streaming data server, business intelligence (BI) server, or other server (or a combination of servers).

[0269] The computer (1902) can receive requests over network (1930) from a client application (for example, executing on another computer (1902) and responding to the received requests by processing the said requests in an appropriate software application. In addition, requests may also be sent to the computer (1902) from internal users (for example, from a command console or by other appropriate access method), external or third-parties, other automated applications, as well as any other appropriate entities, individuals, systems, or computers.

[0270] Each of the components of the computer (1902) can communicate using a system bus (1903). In some implementations, any or all of the components of the computer (1902), both hardware or software (or a combination of hardware and software), may interface with each other or the interface (1904) (or a combination of both) over the system bus (1903) using an application programming interface (API) (1912) or a service layer (1913) (or a combination of the API (1912) and service layer (1913). The API (1912) may include specifications for routines, data structures, and object classes. The API (1912) may be either computer-language independent or dependent and refer to a complete interface, a single function, or even a set of APIs (1912). The service layer (1913) provides software services to the computer (1902)or other components (whether or not illustrated) that are communicably coupled to the computer (1902). The functionality of the computer (1902) may be accessible for all service consumers using this service layer. Software services, such as those provided by the service layer (1913), provide reusable, defined business functionalities through a defined interface. For example, the interface may be software written in JAVA, C++, or other suitable language providing data in extensible markup language (XML) format or another suitable format. While illustrated as an integrated component of the computer (1902), alternative implementations may illustrate the API (1912) or the service layer (1913) as stand-alone components in relation to other components of the computer (1902) or other components (whether or not illustrated) that are communicably coupled to the computer (1902). Moreover, any or all parts of the API (1912) or the service layer (1913) may be implemented as child or submodules of another software module, enterprise application, or hardware module without departing from the scope of this disclosure.

[0271] The computer (1902) includes an interface (1904). Although illustrated as a single interface (1904) in FIG. 10, two or more interfaces (1904) may be used according to particular needs, desires, or particular implementations of the computer (1902). The interface (1904) is used by the computer (1902) for communicating with other systems in a distributed environment that are connected to the network (1930). Generally, the interface (1904) includes logic encoded in software or hardware (or a combination of software and hardware) and operable to communicate with the network (1930). More specifically, the interface (1904) may include software supporting one or more communication protocols associated with communications such that the network (1930) or interface's hardware is operable to communicate physical signals within and outside of the illustrated computer (1902).

[0272] The computer (1902) includes at least one computer processor (1905). Although illustrated as a single computer processor (1905) in FIG. 10, two or more processors may be used according to particular needs, desires, or particular implementations of the computer (1902). Generally, the computer processor (1905) executes instructions and manipulates data to perform the operations of the computer (1902) and any algorithms, methods, functions, processes, flows, and procedures as described in the instant disclosure.

[0273] The computer (1902) also includes a memory (1906) that holds data for the computer (1902) or other components (or a combination of both) that can be connected to the network (1930). The memory may be a non -transitory computer readable medium. For example, memory (1906) can be a database storing data consistent with this disclosure. Although illustrated as a single memory (1906) in FIG. 10, two or more memories may be used according to particular needs, desires, or particular implementations of the computer (1902) and the described functionality. While memory (1906) is illustrated as an integral component of the computer (1902), in alternative implementations, memory (1906) can be external to the computer (1902).

[0274] The application (1907) is an algorithmic software engine providing functionality according to particular needs, desires, or particular implementations of the computer (1902), particularly with respect to functionality described in this disclosure. For example, the application (1907) can serve as one or more components, modules, applications, etc. Further, although illustrated as a single application (1907), the application (1907) may be implemented as multiple applications (1907) on the computer (1902). In addition, although illustrated as integral to the computer (1902), in alternative implementations, the application (1907) can be external to the computer (1902).

[0275] There may be any number of computers (1902) associated with, or external to, a computer system containing computers (1902), wherein each computer (1902) communicates over network (1930). Further, the term “client,” “user,” and other appropriate terminology may be used interchangeably as appropriate without departing from the scope of this disclosure. Moreover, this disclosure contemplates that many users may use one computer (1902), or that one user may use multiple computers (1902).

[0276] Although only a few example embodiments have been described in detail above, those skilled in the art will readily appreciate that many modifications are possible in the example embodiments without materially departing from this invention. Accordingly, all such modifications are intended to be included within the scope of this disclosure as defined in the following claims. In the claims, means-plus-function clauses are intended to cover the structures described herein as performing the recitedfunction and not only structural equivalents, but also equivalent structures. Thus, although a nail and a screw may not be structural equivalents in that a nail employs a cylindrical surface to secure wooden parts together, whereas a screw employs a helical surface, in the environment of fastening wooden parts, a nail and a screw may be equivalent structures. It is the express intention of the applicant not to invoke 35 U.S.C. § 112(f) for any limitations of any of the claims herein, except for those in which the claim expressly uses the words ‘means for’ together with an associated function.

Claims

CLAIMSWhat is claimed:

1. A method, comprising: obtaining operational data at a base station, wherein the operational data comprises one or more selected from the following group: haul truck telemetry data from a fleet of haul trucks, route image data, environment data, haul load efficiency data, and payload data; obtaining a mine plan of a mine, the mine plan comprising mine plan targets and route constraints describing a set of haul routes at the mine; determining, using an Al and Autonomy System comprised of Al models and connected to a base station, operational parameters selected from the following group: a CO2 emission, a productivity, an energy use, a haul truck velocity, and a haul truck acceleration, given the operational data; and adjusting, automatically and according to the mine plan and the mine plan targets, the operational parameters of at least one haul truck in the fleet of haul trucks and one or more haul trucks in operation.

2. The method of claim 1, wherein the mine plan further comprises site speed and acceleration limits.

3. The method of claim 1, wherein the mine plan targets comprise one or more items selected from the following group: a maximum energy per tonne moved, a maximum CO2 emission, and a minimum production rate.

4. The method of any one of claims 1-3, wherein the Al and Autonomy System further comprises a Master Al that fuses at least one output from the Al models.

5. The method of any one of claims 1-3, wherein the Al models comprise a generative Al model that produces one or more items selected from the following group: novel patterns in operational data, predictions of future operational states, and operational strategies for optimizing fleet management.

6. The method of any one of claims 1-3, wherein the Al and Autonomy System adjusts the operational parameters with commands to a Programmable Logic Controller on at least one haul truck.

7. The method of any one of claims 1-3, wherein the operational data is one or more selected from the following group: loT sensors, aerial robots, GPS, LiDAR, road surface cameras, geo data, weather sensors, tire pressure sensors, truck strut sensors, aerial imagery, digital maps, and weather data.

8. The method of any one of claims 1-3, wherein the operational parameters further comprise one or more selected from the following group: a power usage across a varying torque demand and a regenerative braking capability.

9. The method of any one of claims 1-3, further comprising performing one or more actions selected from the following group: sending automatic alerts for route recommendations to manage overall site speed limits, requesting a grader for the at least one haul route based on determined conditions of the at least one haul route, and requesting a water truck for the at least one haul route based on the determined conditions of the at least one haul route.

10. A system, comprising: a base station, configured to be connected to an Al and Autonomy System and to receive operational data from sensors; the sensors, configured to record operational data comprising one or more selected from the following group: haul truck telemetry data from a fleet of haul trucks, route image data, environment data, haul load efficiency data, and payload data; a fleet of haul trucks, configured to record telemetry data; and the Al and Autonomy System, comprised of a plurality of Al models, configured to: determine operational parameters selected from the following group: a CO2 emission, a productivity, an energy use, a haul truck velocity, and a haul truck acceleration, given the operational data, and adjust, automatically and according to a mine plan and mine plan targets, the operational parameters of at least one haul truck in the fleet of haul trucks and a number of haul trucks in operation.

11. The system of claim 10, wherein the mine plan comprises site speed and acceleration limits.

12. The system of claim 10, wherein the mine plan targets comprise one or more items selected from the following group: a maximum energy per tonne moved, a maximum CO2 emission, and a minimum production rate.

13. The system of any one of claims 10-12, wherein the Al and Autonomy System further comprises a Master Al that fuses at least one output from the plurality of Al models.

14. The system of any one of claims 10-12, wherein the plurality of Al models comprises a generative Al model that produces one or more items selected from the following group: novel patterns in operational data, predictions of future operational states, and operational strategies for optimizing fleet management.

15. The system of any one of claims 10-12, wherein the Al and Autonomy System adjusts the operational parameters with commands to a Programmable Logic Controller on at least one haul truck.

16. The system of any one of claims 10-12, wherein the operational data is one or more selected from the following group: loT sensors, aerial robots, GPS, LiDAR, road surface cameras, geo data, weather sensors, tire pressure sensors, truck strut sensors, aerial imagery, digital maps, and weather data.

17. The system of any one of claims 10-12, wherein the operational parameters further comprise one or more selected from the following group: a power usage across a varying torque demand and a regenerative braking capability.

18. The system of any one of claims 10-12, further comprising performing one or more actions selected from the following group: sending automatic alerts for route recommendations to manage overall site speed limits, requesting a grader for the at least one haul route based on determined conditions of the at least one haul route, and requesting a water truck for the at least one haul route based on the determined conditions of the at least one haul route.

Images