System, vehicle controller, traffic control system, mining vehicle, kit, and methods for monitoring and controlling vehicles in a mining environment

Abstract

A system, vehicle controller, a traffic control system, a kit, and methods therein are provided for monitoring and controlling operation of a mining vehicle from a plurality of mining vehicles in a mining site. As the mining vehicle is operating in the mining site, one or more inertial sensor measurements may be acquired from at least one inertial sensor coupled to a movable implement of the vehicle. A trained machine-learning model may be applied to the inertial sensor measurements to predict a subsequent operating state of a set of operating states of the vehicle, wherein the subsequent operating state belongs to a cycle comprising a sequence of operating states from the set of operating states. At least one productivity indicator from the predicted subsequent operating state, and a result representation is generated comprising information on the predicted subsequent operating state and the cycle to which the predicted subsequent operating state belongs.

Description

Docket No.: PS56113PC00 TITLE SYSTEM, VEHICLE CONTROLLER, TRAFFIC CONTROL SYSTEM, MINING VEHICLE, KIT, AND METHODS FOR MONITORING AND CONTROLLING VEHICLES IN A MINING ENVIRONMENT TECHNICALFIELD

[0001] The disclosure relates to systems and methods for monitoring and controlling operation of a mining vehicle from a plurality of mining vehicles in a mining site. The disclosure further relates to a vehicle controller, a mining vehicle, a kit, a computer program product, and a computer-readable storage medium for monitoring and controlling operation of a mining vehicle. BACKGROUND

[0002] In mining and tunnelling, developments are constantly underway to improve efficiency, productivity, and safety. One of the leading areas in which changes / improvements are increasingly taking place is automation, full or partial, of various processes occurring in mining / tunneling.

[0003] Mining vehicles, e.g., trucks, for underground mining and tunneling can perform various tasks in environments that are dark and often inaccessible by foot and may generally be not comfortable for human drivers. Thus, it is often desirable that mining vehicles or machines that operate in an underground environment can be driven in a fully autonomous mode, i.e., without an onboard operator being required to control the machines during machine operation.

[0004] An example of mining vehicles where automated operation is typically considered to be beneficial are so-called load-haul-dump (LHD) machines. These mining vehicles represent vehicles that may be used to remove and transport broken rock / ore from a certain location, e.g., a position where blasting has been performed, to a particular place where the broken rock is dumped. After dumping their load, at the place that may be referred to as a dump point or location, the LHD machines typically return to an initial (start) location to pick up a new load. Thus, these vehicles often travel the same route over and over again, which makes the travel between load and dump locations well suited for automation. There are also various other situations where automation may prove beneficial.

[0005] The mining vehicle may be operating in one out of different possible operating states, and a current state is typically recorded by a vehicle operator. This may however be a tiring and cumbersome process for a human. At the same time, accurate and timely prediction of operating states of the mining vehicle may affect the productivity of the entire mining operation.Docket No.: PS56113PC00 SUMMARY

[0006] In an aspect, a method for monitoring and controlling operation of a mining vehicle from a plurality of mining vehicles in a mining site is provided. The method comprises, as the vehicle is operating in the mining site, acquiring one or more inertial sensor measurements from at least one inertial sensor coupled to a movable implement of the vehicle; applying a trained machine-learning model to the inertial sensor measurements to predict a subsequent operating state of a set of operating states of the vehicle, wherein the subsequent operating state belongs to a cycle comprising a sequence of operating states from the set of operating states; determining at least one productivity indicator from the predicted subsequent operating state; and generating a result representation comprising information on the predicted subsequent operating state and the cycle to which the predicted subsequent operating state belongs.

[0007] The technical benefits and advantages automatic prediction or detection of operating cycles, which allows improved control over productivity. In this way, overall efficiency and productivity of mining operations may be increased. Furthermore, because of the determining of operating states and productivity indicators being performed in real time, timely control and intervention may be implemented in fleet management systems. Valuable information on fleet operations is generated, enabling decisions to be made on dynamic operational assignments. As a further advantage, the provided approach is equipment agnostic, such that it may be integrated with various mining machinery and equipment without requiring extensive modifications or specialized hardware. This flexibility not only simplifies implementation, but also allows for wider adoption in different mining configurations. Security of fleet operation may be increased because operating states are identified for mining vehicles automatically i.e. without involvement of a human operator, and

[0008] Real-time detection capability is another advantage of the present disclosure, which allows instant identification of operating states, enabling timely actions and interventions as needed. This timely response contributes to the overall efficiency and safety of mining operations.

[0009] Additionally, the provided system, vehicle, kit, and methods therein are developed and configured to operate with low computational resource requirements. This feature allows for the provided technique to be implemented in onboard vehicle controllers which typically have limited processing capabilities. By using resources efficiently, the provided technique may ensure optimal performance even in challenging computing environments. Furthermore, various events may be triggered in response to predicting the operating states and determining productivity indicators such as e.g., predictive maintenance of vehicles, optimizing routes traveled by the vehicles, defining and adjusting assignments for the vehicles, and others.Docket No.: PS56113PC00

[0010] In some examples, the method may comprise evaluating a veracity of a state transition from a preceding operating state to the predicted subsequent operating state. In some examples, the evaluating the veracity of the state transition from the preceding operating state to the predicted subsequent operating state comprises: responsive to determining that the predicted subsequent operating state is not an initial operating state, determining whether the predicted subsequent operating state is different from the preceding operating state, wherein a current operating state is set to the preceding operating state; responsive to determining that the predicted subsequent operating state is different from the preceding operating state, determining whether the predicted subsequent operating state is predicted to occur within a margin for a threshold transition time indicating an expected time for transition from the preceding operating state to the predicted subsequent operating state; and, responsive to determining that the predicted subsequent operating state is predicted to occur within the margin for the threshold transition time, confirming the veracity of the state transition from the preceding operating state to the predicted subsequent operating state and updating the current operating state to be set to the predicted subsequent operating state.

[0011] In some examples, the method may comprise generating the result representation responsive to updating the current operating state to be set to the predicted subsequent operating state and / or responsive to determining that detection of the cycle to which the predicted subsequent operating state belongs is complete.

[0012] In some examples, the trained machine-learning model may be trained in dependency on at least one property of the mining site.

[0013] In some examples, the method may comprise controlling the vehicle in dependence on the predicted subsequent operating state after the veracity of the state transition from the preceding operating state to the predicted subsequent operating state is confirmed. The vehicle may be controlled to transition to a next operating state. In some examples, the method may comprise assisting in automating the vehicle when the vehicle is transitioned from a non-autonomous control mode to an autonomous control mode.

[0014] In some examples, the at least one inertial sensor comprises one inertial measurements unit, IMU, comprising an accelerometer, a gyroscope, and a magnetometer. The

[0015] In some examples, the trained machine-learning model is agnostic to a model of the vehicle, a brand of the vehicle, and a manufacturer of the vehicle.

[0016] In some examples, the method may comprise sending the result representation to an external device or system. The external device or system may comprise a traffic control system such as e.g. a fleet controller configured to control mining vehicles in a mining environment or site.Docket No.: PS56113PC00

[0017] In an aspect, a mining vehicle comprising a movable implement is provided. The mining vehicle comprises a controller that comprises processing circuitry configured to, as the vehicle is operating in the mining site, acquire one or more inertial sensor measurements from an inertial sensor coupled to on the movable implement of the vehicle; apply a trained machine-learning model to the inertial sensor measurements to predict a subsequent operating state of a set of operating states of the vehicle, wherein the subsequent operating state belongs to a cycle comprising a sequence of operating states from the set of operating states; determine at least one productivity indicator from the predicted subsequent operating state; and generate a result representation comprising information on the predicted subsequent operating state and the cycle to which the predicted subsequent operating state belongs.

[0018] In some examples, the processing circuitry may be configured to evaluate a veracity of a state transition from a preceding operating state to the predicted subsequent operating state. In some examples, the evaluating the veracity of the state transition from the preceding operating state to the predicted subsequent operating state comprises: responsive to determining that the predicted subsequent operating state is not an initial operating state, determining whether the predicted subsequent operating state is different from the preceding operating state, wherein a current operating state is set to the preceding operating state; responsive to determining that the predicted subsequent operating state is different from the preceding operating state, determining whether the predicted subsequent operating state is predicted to occur within a margin for a threshold transition time indicating an expected time for transition from the preceding operating state to the predicted subsequent operating state; and, responsive to determining that the predicted subsequent operating state is predicted to occur within the margin for the threshold transition time, confirming the veracity of the state transition from the preceding operating state to the predicted subsequent operating state and updating the current operating state to be set to the predicted subsequent operating state and updating information on the cycle to which the predicted subsequent operating state belongs to indicate that the cycle includes the predicted subsequent operating state.

[0019] In some examples, the processing circuitry may be configured to generate the result representation responsive to updating the current operating state to be set to the predicted subsequent operating state and / or responsive to determining that detection of the cycle to which the predicted subsequent operating state belongs is complete.

[0020] In some examples, the processing circuitry may be configured to control the vehicle in dependence on the predicted subsequent operating state after the veracity of the state transition from the preceding operating state to the predicted subsequent operating state is confirmed.Docket No.: PS56113PC00

[0021] In some examples, in the mining vehicle, the at least one inertial sensor comprises one inertial measurements unit, IMU, comprising an accelerometer, a gyroscope, and a magnetometer. In some examples, the movable implement comprises a bucket. The inertial sensor may detect and record movements of the bucket or another tool of the movable implement.

[0022] In an aspect, a traffic control system comprising at least one processor for coordinating movements of one or more vehicles out of a plurality of vehicles in a mining site, wherein each vehicle of the one or more vehicles may comprise a controller comprising processing circuitry that is configured to perform a method for monitoring and controlling operation of a mining vehicle from a plurality of mining vehicles in a mining site, in accordance with examples of the present disclosure.

[0023] In some examples, the at least one processor of the traffic control system may be configured to receive, from the vehicle of the one or more vehicles, a representation comprising information on a predicted subsequent operating state of the vehicle and a cycle to which the predicted subsequent operating state belongs; generate at least one command to the vehicle in dependence on the representation; and send the at least one command to the vehicle.

[0024] In an aspect, a kit is provided which comprises at least one inertial sensor and computer program product comprising computer-executable which, when executed by processing circuitry on a mining vehicle from a plurality of mining vehicles in a mining site, cause the processing circuitry to perform the method in accordance with examples of the present disclosure. The method comprises, as the vehicle is operating in the mining site, acquiring one or more inertial sensor measurements from at least one inertial sensor coupled to a movable implement of the vehicle; applying a trained machine- learning model to the inertial sensor measurements to predict a subsequent operating state of a set of operating states of the vehicle, wherein the subsequent operating state belongs to a cycle comprising a sequence of operating states from the set of operating states; determining at least one productivity indicator from the predicted subsequent operating state; and generating a result representation comprising information on the predicted subsequent operating state and the cycle to which the predicted subsequent operating state belongs.

[0025] The kit may be configured to be deployed on the mining vehicle. In some examples, the computer program product of the kit may be installed on a vehicle controller of the mining vehicle. In some examples, the computer program product of the kit may be implemented as a controller device, such that the controller device may be installed and deployed on the vehicle in addition to existing one or more controllers of the vehicle. In some examples, in the mining vehicle, the at least one inertial sensor comprises one inertial sensor. The one inertial sensor may comprise one inertial measurementsDocket No.: PS56113PC00 unit, IMU, comprising an accelerometer, a gyroscope, and a magnetometer. In some examples, the movable implement of the mining vehicle comprises a bucket.

[0026] According to an aspect of the disclosure, a computer program product is provided that comprises computer-executable instructions, which, when executed by processing circuitry, cause the processing circuitry to perform the method and actions in accordance with embodiments of the present disclosure.

[0027] According to an aspect of the disclosure, a computer-readable storage medium is provided. The computer-readable storage medium has stored thereon a computer program product comprising computer-executable which, when executed by processing circuitry, cause the processing circuitry to perform the method and actions in accordance with embodiments of the present disclosure.

[0028] Additional features and advantages are disclosed in the following description, claims, and drawings. Furthermore, additional advantages will be readily apparent from the present disclosure to those skilled in the art or recognized by practicing the disclosure as described herein. There are also disclosed herein control units, computer program products, and computer-readable media associated with the above discussed technical effects and corresponding advantages. BRIEFDESCRIPTION OF THEDRAWINGS

[0029] With reference to the appended drawings, below follows a more detailed description of aspects of the disclosure cited as examples.

[0030] FIG.1A illustrates an example of a mining environment in which a method in accordance with examples of the present disclosure may be implemented.

[0031] FIG.1B illustrates an example of a mining vehicle.

[0032] FIG.2 is a block diagram illustrating a system in which a method in accordance with examples of the present disclosure may be implemented.

[0033] FIG.3 is a flowchart illustrating a method for monitoring and controlling operation of a mining vehicle from a plurality of mining vehicles in a mining site, in accordance with example embodiments of the present disclosure.

[0034] FIG.4 is a flowchart illustrating a method for generating and training a machine-learning model, in accordance with example embodiments of the present disclosure.

[0035] FIG.5 is another flowchart illustrating a method for monitoring and controlling operation of a mining vehicle from a plurality of mining vehicles in a mining site, in accordance with example embodiments of the present disclosure.Docket No.: PS56113PC00

[0036] FIG.6 is a flowchart illustrating a method for evaluating a veracity of a state transition from a preceding operating state to a predicted subsequent operating state, in accordance with example embodiments of the present disclosure.

[0037] FIG.7 is a schematic block diagram illustrating an example of a control system, in accordance with example embodiments of the present disclosure. DETAILED DESCRIPTION

[0038] In a mining environment, there may be multiple machines or vehicles, e.g., autonomous vehicles, in a confined area. The vehicles may be remotely controlled and / or operated autonomously. Integration of autonomous equipment and remote control, among other technological solutions, allows for stable and controlled processes while reducing risks to personnel and operations. At the same time, remote control of a vehicle has its challenges, for example, those related to a lack of operator that would record operational information related to the vehicle, e.g., operational states of the vehicle. This limits certain functionalities of control systems that depend on operator interaction to record relevant operational information. Furthermore, because autonomous equipment is typically controlled by specific controllers which differ among manufacturers, it may be challenging to implement a coordinated control of vehicles by multiple manufacturers. This may be a case even for a control of vehicles in the same fleet.

[0039] In the mining sector, knowledge of operating cycles of loading and haulage equipment, such as e.g. LHD vehicles, is of considerable interest. This is even more relevant today due to increasing integration of autonomous or remotely controlled vehicles.

[0040] The LHD vehicle operating cycle was formally introduced as a performance indicator in 2015, as mentioned in [1]. This cycle involves the LHD vehicle loading blasted ore from the mine face using a bucket, then hauling it to the dump point, unloading it, and finally returning to the mine face. By detecting and tracking this operating cycle, it is possible to calculate the continuous working time of the LHD vehicle.

[0041] As mentioned in [2], ore hauling is a common task in all mines, regardless of whether they are open pit or subway. Vehicles or machines such as e.g. haul trucks and loaders are used to perform this task. These machines repeat work cycles that may be divided into the four operational states such as loading, loaded haulage, unloading and unloaded haulage. Accurate tracking of the operational cycles generated by these states may allow creating analytical tools to assist mine control systems in achieving efficient production and planning. Currently, information collected on these cycles is often based on direct reporting from an operator of a vehicle, which may be unreliable and subject to self-bias and user error. Furthermore, this task may be repetitive and tedious for a human. For example, asDocket No.: PS56113PC00 mentioned in [2], in deposits with a high number of workers such as in medium and large mining, each shift manager must manually request work cycle information from up to 20 operators and then use it to complete the corresponding forms.

[0042] There are certain existing solutions for detecting operational states of LHD machines, which solutions rely on use of machine's own internal information obtained through a CAN BUS of the equipment, as described, e.g., in [3], [4], and [5]. This type of solutions are not agnostic to a brand and model of the equipment, which leads to a limitation that these solutions may not be applicable to other equipment. In some cases, as e.g. in [3], an Internet connection is required to process CAN BUS data in real time.

[0043] Additionally, in subway mining, a limitation exist that a GPS sensor may not be sufficiently accurate to geospatially track an LHD vehicle and detect its productive cycle. In other cases, detection of operating cycles have been performed using RFID local positioning system has been approached, but this approach requires more infrastructure within the subway mine and may not be feasible for all fleet operators.

[0044] Machine learning models have been used in [1] and [2], but the described approaches are very complex and have a high computational load, which limits their implementation in field computers of lower computational capacity. In subway mining, capacity of connection to a cloud to process data and execute models may be limited.

[0045] In [6], two IMU sensors and a pre-loaded digital map are used. However, this system is limited to the subway mine structure being of a so-called Manhattan type and to have a previously digitized interior map.

[0046] U.S. Patent Application Publication 2022 / 0099533 describes a monitoring system that is based on engine speed, GPS signals, and vibrations measured with an accelerometer located inside an operator cabin. The operation of the machine is based on a series of previously defined thresholds. Variables related to the engine are specific to the machine, and the proposed approach would therefore not be agnostic to equipment of a mining vehicle.

[0047] Accordingly, although solutions exist to tracking operating states of vehicles in a mine, this task remains to be a challenge. For example, according to [6], tracking a workflow of a mining fleet is a key challenge in a case of subway mines. At the same time, considering the number of operating contexts, difficulty in information transfer, and external factors influencing operating parameters, determining a precise location of the mining machines in subway excavations is required for analysis of data collected by monitoring systems.

[0048] A system and method are provided for controlling autonomous mining vehicles such as e.g. front-end loaders or Load-Haul-Dump (LHD) vehicles with a moving or movable implement such as e.g.Docket No.: PS56113PC00 a bucket. The approach in accordance with examples of the present disclosure is agnostic to the model and brand of a vehicle, consolidates operational information and integrates this information into control systems that do not need to rely on presence of an operator in the vehicle. The system and method employ a machine learning model for detection of operating states of one or more vehicles in a mining site, using data acquired from one inertial measurement unit (IMU) sensor. The machine-learning model enables real-time detection of operating states of the one or more vehicles, and allows determining productivity indicators for the vehicles. The operating states may comprise material loading and unloading, downtime, and moving or transfers. The productivity indicators comprise to a time associated with each of the determining operating states, a weight of a load, and other indicators.

[0049] The technique in accordance with examples of the present disclosure employs one IMU sensor external to the LHD equipment, offers the possibility of detecting operating states in a machine- agnostic way, and is generally a low-cost approach. The developed models are light enough to process data and detect LHD operating states in real time on a controller installed onboard of the vehicle, which may also be referred to as a field computer.

[0050] In some aspects, a method is provided that may be performed by a controller of a mining vehicle.

[0051] In some aspects, the controller of the mining vehicle is provided that is configured to implement a method in accordance with examples of the present disclosure.

[0052] In some aspects, a mining vehicle is provided in which a method in accordance with examples of the present disclosure may be implemented.

[0053] In some aspects, a traffic control system, which may be a fleet control system, is provided that comprises one or more mining vehicles out of a plurality of mining vehicles in which a method in accordance with examples of the present disclosure may be implemented. The traffic control system may be controlling operation and movements of the one or more mining vehicles, and operating states of the vehicles of the one or more mining vehicles may be determined in accordance with examples of the present disclosure.

[0054] In some aspects, a system is provided that comprises the traffic control system and one or more mining vehicles out of a plurality of mining vehicles in which a method in accordance with examples of the present disclosure may be implemented.

[0055] In some aspects, a kit may be provided that is implemented in accordance with examples of the present disclosure. The kit may be a standalone sensing unit or system, and it may be deployed on new or existing machines. The kit may include an inertial sensor and computer program product comprising computer-executable instructions, which, when executed by processing circuitry, cause the processing circuitry to perform a method in accordance with examples of the present disclosure. InDocket No.: PS56113PC00 some examples, the inertial sensor may be one IMU. The inertial sensor may be coupled to a mining vehicle and the computer program product may be installed on the vehicle, e.g., as a vehicle controller or as part of an existing, such as a third-party, vehicle controller. In this way, inertial sensor measurements acquired by the inertial sensor are received by the vehicle controller configured to execute the computer-executable instructions of the computer program product, and the vehicle controller is thus configured and adapted to perform a method in accordance with examples of the present disclosure. The computer program product may include a trained machine-learning model which may have been previously trained, and / or the computer program product may allow retraining or training the machine-learning model. In some example, the computer program product may also include computer-executable instructions that may be installed on a controller or control system that is external to the mining vehicle, e.g., a traffic control system such as e.g. a fleet control system. The fleet control system may thus be enabled to monitor and control operation of one or more mining vehicles each of which has the inertial sensor coupled thereto and also has the computer-executable instructions, installed on its control unit or controller, that are configured to be executable by processing circuitry of the controller to perform the method in accordance with examples of the present disclosure.

[0056] FIG.1A depicts an example of a mining environment or site 10 comprising multiple, four in this example, mining vehicles 12a, 12b, 12c, and 12d. The vehicles may be of the same or different types. The vehicles may be underground mining vehicles or surface mining vehicles. One or more of the mining vehicles 12a-12d may be load-haul-dump (LHD) machines. In some examples, the mining vehicles 12a-12d may be autonomous vehicles, such as e.g., fully or partially autonomous vehicles. In some examples, the mining vehicles 12a-12d may be part of one or more vehicles in the fleet. The vehicles in the fleet may be assigned work or tasks.

[0057] The mining site 10 may be a site of surface mining or a site of mining in an underground environment. The mining site 10 may be any type of a mine-like underground environment. In some examples, the mining site 10 may be a subway mine. Although not shown in FIG.1A, The mining site 10 may have a specific configuration. For example, if the mining site 10 is an underground mining site, it may include tunnels. A vehicle in the mining site 10 may have a moving or movable implement, such as, e.g., a bucket or another movable implement. The vehicle may be an excavator, a backhoe, or another type of a vehicle with a movable implement.

[0058] As shown FIG.1A, each of the vehicles 12a-12d may have a respective inertial sensor 14a, 14b, 14c, and 14d, coupled to a corresponding movable implement 15a, 15b, 15c, and 15d of the vehicle. In some example, each of the inertial sensor 14a, 14b, 14c, and 14d may be a single i.e. one IMU sensor or IMU that allows external to detecting operating states of the respective vehicle in a machine-agnostic way. Because only one sensor, such as e.g. an IMU, is used, the provided approachDocket No.: PS56113PC00 is low cost and low maintenance. Also, machine-learning models used to predict or identify or determine operating states are light enough, require relatively low computational power, to process data and detect vehicle operating states in real time on the vehicle controller also referred to a field computer.

[0059] The movable implement may comprise a load picking and carrying implement or tool, e.g., a bucket or another work tool for digging and / or load picking and carrying, and an arm such as e.g. a hydraulic arm which couples the load picking and carrying tool to the vehicle. In examples herein, the inertial sensor may be coupled to the arm, e.g, to a joint of the arm of the movable implement. In examples herein, the inertial sensor may be positioned in proximity to the joint of the arm and in proximity to a tool for digging and / or load picking and carrying, e.g., a bucket or another work tool of the movable implement.

[0060] The inertial sensor may be positioned at a location of the movable implement where sensor measurements are informative of an operating or operational state of the movable implement. Also, an ease of installation of the inertial sensor is taken not consideration when selecting a specific location at which to couple the inertial sensor to the vehicle. If the vehicle comprises an operator cabin, the inertial sensor is positioned outside of the operator cabin.

[0061] The specific position of the inertial sensor on the movable implement may depend on a configuration of the movable implement, type of work performed by the vehicle, work environment conditions to which movable implements are subjected, the feasibility of installation and / or on other factors.

[0062] As further shown in FIG.1A, each of the vehicles 12a-12d also has a respective vehicle controller 16a, 16b, 16c, and 16d that may be configured to perform the method in accordance with examples of the present disclosure. Each of the vehicle controller 16a, 16b, 16c, and 16d includes various components not shown in this example, such as memory configured to store computer- executable instructions and processing circuitry configured to execute the computer-executable instructions that cause the computer-executable instructions to perform a method in accordance with examples of the present disclosure, as discussed in more detail below. Each of the vehicle controllers 16a-16d also has an input and output interface that is configured to communicate with other components, e.g., with the respective inertial sensor of the inertial sensors 14a-14d, as well as with a fleet controller or control system 20 and with other external systems.

[0063] Multiple mining vehicles 12a-12d may be operated in the mining site 10 for moving mined material from draw points or locations to one or more dumping points where the material is deposited. Thus, a vehicle of the mining vehicles 12a-12d may be operated in an operating or operational cycle that may comprises loading the vehicle to pick a certain amount of mined material at a draw point,Docket No.: PS56113PC00 moving or hauling the material to the dumping point, uploading or dumping the material at the dumping point, and moving or translating back to a draw location, without a load. The vehicle may also be in a standby mode or standby, when it is idling such that it is operating but not currently moving or performing any work. The cycle may include another sequence of operating states. For example, in some cases, the cycle may include loading the vehicle to pick a certain amount of mined material at a draw point, moving or hauling the material to the dumping point, and uploading or dumping the material at the dumping point. The cycle may be repeated multiple times during, for example, a shift of the operation of the vehicle. The shift may have any suitable number of cycles. For example, in examples in which the vehicle is autonomous, the shift may be performed until, e.g., the vehicle runs out of fuel or power.

[0064] In some examples, a kit 18 may be installed and deployed on a mining vehicles of one or more vehicles out of the plurality of vehicles 12a-12d in the mining site 10. The kit may comprise at least one inertial sensors and computer program product comprising computer-executable which, when executed by processing circuitry on a mining vehicle from a plurality of mining vehicles in a mining site, cause the processing circuitry to perform a method in accordance with examples of the present disclosure. FIG.1A illustrates schematically, for vehicles 12a and 12b, that the vehicles may comprise respective kits 18a and 18b, shown by a dot-dashed line. The kit 18a may comprise the inertial sensor 14a and computer program product that may be installed on the vehicle controller 16a. Similarly, the kit 18b may comprise the inertial sensor 14b and computer program product that may be installed on the vehicle controller 16b. The computer program products may in some examples be provided on separate e.g. dedicated controllers provided as part of a kit.

[0065] As shown in FIG.1A, operation of vehicles in the mining site 10, shown by way of example as the mining vehicles 12a-12d, may be controlled by a control system 20 such as e.g. a fleet controller or fleet control system. The control system 20 may track locations or positions of each of the mining vehicles 12a-12d, may receive data from the vehicles, and may send commands or instructions to the vehicles. The fleet control system may also be referred to as a traffic control system, and it may be any type of a control system that monitors multiple vehicles e.g. receives information from the vehicles, and sends commands or instructions to the vehicles. The fleet control system 20 comprises processing circuitry 22 that may execute computer-executable instructions which thereby cause the processing circuitry 22 to perform a method for controlling operation of one or more vehicles, in accordance with examples of the present disclosure. The fleet control system 20 may have various other components not shown herein for simplicity, e.g., memory, an input and output interface, etc.

[0066] FIG.1B illustrates an example of a mining vehicle 22, such as e.g., any of the vehicles 12a-12d shown in FIG.1A, or another type of a vehicle. FIG.1B illustrates schematically exampleDocket No.: PS56113PC00 locations or positions 24p1 and 24p2 of where an inertial sensor may be coupled to a movable implement 25 of the vehicle 12. The inertial sensor may be configured to acquire accelerometer, magnetometer, and triaxial gyroscope measurements. The measurements may be acquired at a desired frequency and may be used to infer when the vehicle lifts the movable implement 25 such as a hydraulic arm with a bucket or another tool attached thereto. The inertial sensor may be installed on the hydraulic arm of the movable implement 25 in proximity to the bucket, as shown in FIG.1B. In the example illustrated in FIG.1B, the example positions 24p1 and 24p2 may be evaluated for ease of installation and data quality of obtained sensor measurements. Both of the sensor may provide desirable results. The position 24p1, in this example, allows identifying a clearer observable pattern. It should be appreciated that the mining vehicle 22 is shown in FIG.1B as a non-limiting example only, as the inertial sensor may be positioned to be associated with a movable implement of a vehicle in any suitable location. In some example, two or more positions along the movable implement may be evaluated for placement of an inertial sensor. In some example, the evaluation is not performed.

[0067] The approach described herein, in accordance with examples of the present disclosure, comprises two workflows also referred to as pipelines, which in turn comprise multiple stages. One of the pipelines is a training pipeline, which is developed using historical or batch data, and includes pre- analysis stages such as e.g. sensor description and position, exploratory data analysis (EDA), and modeling stages such as data capture and processing, feature extraction, modeling, and validation. Another pipeline is a production pipeline, which is developed with data generated in real time, as the vehicle is operating in a mining environment or site, and includes such stages as e.g, queuing data, preprocessing, feature extraction, generating predictions, performing validation logic which includes confirming veracity of a state transition, determining productivity indicators, and generating results representation.

[0068] FIG.2 illustrates an example of a system 100 in which example embodiments of the present disclosure may be implemented. The system 100 may be employed, at least in part, in the mining environment or site such as e.g. mining site 10 shown in FIG.1A. The system 100 may comprise one or more mining vehicles, of which vehicle 12 is shown as a representative vehicle. The system 100 may also comprise a control system 20 such as, e.g. a traffic control system which may be a fleet controller when the one or more mining vehicles are part of a fleet in which coordinated control of mining vehicles may be performed.

[0069] As shown in FIG.2, the mining vehicle 12 may comprise a control system or vehicle controller 16 which may be located onboard the vehicle 12. The vehicle controller 16 may be a computing system in which a method in accordance with example embodiments of the presentDocket No.: PS56113PC00 disclosure may be implemented. The vehicle controller 16 may comprise processing circuitry 210 such as one or more processors and memory 220.

[0070] The memory 220 may comprise one or more memory units. The memory 220 comprises computer-executable instructions executable by the processing circuitry 210 of the vehicle controller 16. The memory 220 may be configured to store, e.g., information, data, etc., and the computer-executable instructions to perform, when executed by the processing circuitry 210, the methods in accordance with examples of the present disclosure.

[0071] The memory 220 may store data and computer-executable instructions that, when executed by the processing circuitry 210, may cause the processing circuitry 210 to perform the method for monitoring and controlling operation of the mining vehicle. As shown in FIG.2, the memory 220 may store inertial sensor measurements data 221 which may be acquired by the vehicle controller 16 from an inertial sensor coupled to a movable implement of the vehicle 12 (not shown in FIG.2). In some examples, a required amount of the inertial sensor measurements data 221 may be stored in the memory 220, and the data may be discarded once it has been used for further processing, to predict one or more operating states for the vehicle and determine productivity indicators. Also, in some cases, the inertial sensor data may be sent to an external system, e.g., to control system 20 and / or to another system.

[0072] The memory 220 may comprise an operating states registry 222 storing one or more operating states predicted or identified for the vehicle 12; a current operating state registry 224 storing a variable for the current operating state which variable may be associated with one or more parameters 226 stored in association with the current operating state. The one or more parameters related to the current operating state may be, features used to predict the current operating state, a duration of time for the current operating state, a start time and an end time for the current operating state, and other parameters. In some implementations, the operating states registry 222 and current operating state registry 224 may be part of a same storage or registry in memory 220 of the vehicle 12.

[0073] The memory 220 may also store a trained machine-learning (ML) model 228, and in some example embodiments also a model performance monitoring module or unit 230 which may be executed to allow the model to be automatically trained or retrained. The model performance monitoring module or unit 230 may not be present in some implementations.

[0074] The memory 220 may also store one or more productivity indicators or performance indicators 232 and result representation 234. The one or more productivity indicators 232 comprise indicators determined from one or more operating states predicted or identified for the vehicle. Non- limiting examples of the productivity indicators 232 comprise a time that the vehicle is detected to have spent in a certain operating state, average transfer times, haulage cycle counts, load weight counts andDocket No.: PS56113PC00 other metrics related to material loading and unloading, average operating times, or various other productivity indicators. The result representation 234 may comprise information on a predicted operating state, e.g., on a predicted subsequent operating state and a cycle to which the predicted subsequent operating state belongs. The result representation 234 may also comprise information on more than one predicted operating states and a corresponding cycle to which each of these operating state belongs. The result representation 234 may also comprise information on one or more cycles completed by the mining vehicle, each cycle comprising a sequence of vehicle operating states, and the result representation 234 may comprise any other information that is obtained or generated by the vehicle controller as part of performing the method for monitoring and controlling operation of the mining vehicle.

[0075] The processing circuitry 210 is shown to include a ML model execution unit 240 that is configured to execute the trained ML model stored in the ML model unit 228 in the memory 220. The ML model execution unit 240 of the processing circuitry 210 may also be configured to execute the model performance monitoring module 230, which may allow automatically training the machine- learning model by the processing circuitry 210 such as e.g. by the ML model execution unit 240. The processing circuitry 210 may include various modules and / units configured to perform methods and actions in accordance with examples herein.

[0076] It should be appreciated that any of the data and instructions stored in the memory 220 may be loaded onto the processing circuity 210, or used by the processing circuitry 210, to perform the method in accordance with example embodiments of the present disclosure. Thus, the modules stored in memory 220 are shown as part of memory 220 by way of example. The processing circuitry 210 may include other modules or unit configured to perform the method in accordance with example embodiments of the present disclosure.

[0077] Those skilled in the art will appreciate that the units of the vehicle controller 16 described above may refer to a combination of analogue and digital circuits, and / or one or more processors configured with software and / or firmware, e.g., stored in the vehicle controller 16, that, when executed by the respective one or more processors, may perform the methods in accordance with embodiments of the present disclosure. One or more of these processors, as well as the other digital hardware, may be included in a single Application-Specific Integrated Circuitry (ASIC), or several processors and various digital hardware may be distributed among several separate components, whether individually packaged or assembled into a system-on-a-chip.

[0078] It should be appreciated that the controller 16 and vehicle 12 may comprise various other components not shown in FIG.2 for the sake of simplicity. For example, the vehicle 12 may comprise an onboard display on which in some examples the result representation may be displayed. TheDocket No.: PS56113PC00 controller 16 comprises an input and output interface for communication with the inertial sensor coupled to a movable implement of the vehicle and to receive one or more inertial sensor measurements, and for communication with remote systems such as e.g. the control system 20.

[0079] FIG.2 also illustrates that the control system 20, such as e.g. a traffic control system, comprises processing circuitry 22 and memory 260. The traffic control system 20 may be a central controller positioned separately from the vehicles that it is configured to control, e.g., a remote controller. In some examples, the traffic control system 20 may be positioned in the mining environment, e.g., it may be part of or associated with one of the vehicles in the mining environment. Regardless of its specific implementation, the traffic control system 20 is configured to communicate with, e.g., receive information from and send information and control commands to one or more vehicles out of a plurality of vehicles in the mining environment.

[0080] The control system 20 may be a central controller or control system that may be configured to communicate with one or more mining vehicles out of a plurality of mining vehicles in a mining site, and controls, in a coordinated manner, movements and / or other functions of the mining vehicles. The mining vehicles may be fully autonomous, semi-autonomous, or manually controlled vehicles, and various type of signals and instructions may be received by the vehicles from the control system 20.

[0081] The memory 260 may comprise one or more memory units. The memory 260 comprises computer-executable instructions executable by the processing circuitry 22 of the control system 20. The memory 260 is configured to store, e.g., information, data, etc., and the computer-executable instructions to perform, when executed by the processing circuitry 22, the methods in accordance with examples herein. The control system 20 may additionally obtain information from an external memory. In some examples, the control system 20 may obtain information from a cloud storage.

[0082] The processing circuitry 22 of the control system 20 is configured to execute computer- executable instructions which thereby cause the processing circuitry 22 to control operation of one or more vehicles, which may be performed using information generated using the method in accordance with examples of the present disclosure. The control system 20 may have various other modules / units and components not shown for simplicity, e.g., memory, an input and output interface, etc. The memory 260 of the control system 20 may receive various information from the vehicle 16, e.g., one or more operating states, one or more operating cycles, one or more productivity indicators, a result representation that may comprise the one or more operating states and / or one or more productivity indicators, and other information. The control system 20 may generate and send commands to the vehicle 12, as well as to one or more of other vehicles in the plurality of vehicles e.g. in a fleet of vehicles. In some example embodiments, the vehicle 12 may be controlled, at least in part, in dependence on one or more operating states, one or more operating cycles, and / or one or moreDocket No.: PS56113PC00 productivity indicators, which may be provided by the vehicle controller 16 to the control system 20. In some examples, the control of the operation of the vehicle 12, in dependence on one or more operating states, one or more operating cycles, and / or one or more productivity indicators, may be at least in part performed by the vehicle’s own controller 16.

[0083] FIG.3 illustrates a method or process 300 for monitoring and controlling operation of a mining vehicle from a plurality of mining vehicles in a mining environment or site. The mining vehicle may be, e.g., any of the vehicles 12a-12d shown in FIG.1A, or another type of a mining vehicle in a mining environment or site such as e.g. mining site 10 of FIG.1A. The mining site may comprise a plurality of mining vehicles, and one or more of the plurality of mining vehicles may comprise a controller comprising processing circuitry configured to perform the method or process 300.

[0084] The mining vehicle comprises a movable implement with at least one inertial sensor coupled thereto. The mining vehicle may be included in a fleet of vehicles comprising one or more vehicles. The method or process 300 may be performed by a controller that comprises processing circuitry configured to perform the method.

[0085] At block 302, the method 300 comprises, as the vehicle is operating in the mining environment, acquiring one or more inertial sensor measurements from at least one inertial sensor coupled to a movable implement of the vehicle. In example embodiments, the at least one inertial sensor comprises one inertial measurements unit (IMU) comprising an accelerometer, a gyroscope, and a magnetometer. In some examples, the inertial sensor, which may be one sensor, may be coupled to a joint of an arm of the movable implement.

[0086] The inertial sensor measurements may be acquired in real time, as the vehicle is operating in the mining environment. The vehicle may be performing work and it may be in one of operating states from a set of operating states of the vehicle. The set of operating states of the vehicle may comprise loading state or loading comprising picking a load e.g. at a draw location, a hauling state or hauling comprising moving with the load, a dumping state or dumping comprising unloading or dumping of the load e.g. at a dumping location, and a translating state or translating comprising moving without the load. The vehicle may also be in an idle state. In some examples, the hauling and translating states may be referred to as a moving state. In the set, the operating states may be in any order.

[0087] A sequence of the states, such as loading, hauling, dumping, and moving, may form a cycle which includes moving a load from one location to another and traveling back to the initial location.

[0088] At block 312, the method 300 comprises applying a trained machine-learning model to the inertial sensor measurements to predict a subsequent operating state of the set of operating states of the vehicle, wherein the subsequent operating state belongs to a cycle comprising a sequence ofDocket No.: PS56113PC00 operating states from the set of operating states. As used herein, a subsequent operating state describes an operating state in which the vehicle may be currently operating, and it is the operating state that is being predicted or identified. It may be referred to as subsequent operating state for comparison to a previously recorded operating state, which is referred to as a current operating state. For example, a previously recorded operating state may be loading, and the vehicle may be currently moving to transport a load to another location, which is being determined based on the inertial sensor measurements acquired as the vehicle is moving and using the trained machine-learning model.

[0089] Thus, the current operating state may be considered as a variable which keeps track of a current state of the vehicle, and the variable is updated once the newly detected, subsequent operating state, is identified and the transition from the previously recorded operating state to the subsequent operating state is confirmed. Once a sequence of the operating state forming a cycle is detected, the cycle may be considered to be completed by the vehicle. The cycle may be interchangeably referred to as an operating cycle, operational cycle, productivity cycle, or a production cycle. The cycle constitutes completion of a work task by the vehicle, such as e.g. transferring a load of material from one location to another, and returning back to the initial location where the vehicle may begin a next cycle.

[0090] In some examples, the trained machine-learning model is agnostic to a model of the vehicle, a brand of the vehicle, and a manufacturer of the vehicle. The machine-learning model may be trained using data that is obtained from an inertial sensor coupled externally on the mining vehicles and thus being configured to acquire externally sensed movements of the vehicle and / or the vehicle movable implement. Thus, the machine-learning model may be trained without use of data which may be obtained from measuring values of parameters related to operation of internal components of the vehicle. Furthermore, the machine-learning model may be trained without use of imaging data and without use of positioning data. Although a position of each vehicle in a mining site is tracked, this information, e.g., acquired by a position sensor such as a Global Positioning System (GPS) tracker and / or a radio frequency identification (RFID) device, may be not used for the purposes of the present disclosure. Thus, only an inertial sensor, coupled to a movable implement of a mining vehicle, may be sufficient to provide sensor data that is used to predict, identify, or classify, in real-time, an operating state of the vehicle based on detected movements of the movable implement.

[0091] In some examples, the trained machine-learning model may be trained in dependency on at least one property of the mining environment. For example, in some cases, the machine-learning model may be retrained and / or adjusted based on properties or characteristics of the mining environment or site which may include e.g. a layout of the mining environment, distances to be traveled by the vehicle e.g. to and from a dumping point, properties of ground, and other properties that may affect detection of operating states for vehicles in the mining environment.Docket No.: PS56113PC00

[0092] In some examples, the machine-learning model may be trained as described in connection with FIG.4. In some examples, a training pipeline, described in more detail in connection with FIG.4, may be executed automatically, thereby a trained machine-learning model may be generated that is suitable for properties of the mining environment and takes into consideration other factors, conditions, and properties. Thus, in some examples, the machine-learning model may be generated as part of the processing at block 312.

[0093] In some examples, as discussed in more detail below, the method 300 comprises evaluating a veracity of a state transition from a preceding operating state to the predicted subsequent operating state. In some examples, the evaluating the veracity of the state transition from the preceding operating state to the predicted subsequent operating state comprises, responsive to determining that the predicted subsequent operating state is not an initial operating state, determining whether the predicted subsequent operating state is different from the preceding operating state, wherein a current operating state is set to the preceding operating state; responsive to determining that the predicted subsequent operating state is different from the preceding operating state, determining whether the predicted subsequent operating state is predicted to occur within a margin for a threshold transition time indicating an expected time for transition from the preceding operating state to the predicted subsequent operating state; and responsive to determining that the predicted subsequent operating state is predicted to occur within the margin for the threshold transition time, confirming the veracity of the state transition from the preceding operating state to the predicted subsequent operating state and updating the current operating state to be set to the predicted subsequent operating state.

[0094] At block 318, the method 300 comprises determining at least one productivity indicator from the predicted subsequent operating state. The at least one productivity indicator may be e.g. a time or operating time that the vehicle has spent in the predicted subsequent operating state. Non- limiting examples of productivity indicators may comprise a time that the vehicle is detected to have spent in a certain operating state, average transfer times, haulage cycle counts, load weight counts and other metrics related to material loading and unloading, average operating times, or various other productivity indicators. Determining some of the productivity indicators may require data collected on more than one operating state. For example, for some of the productivity indicators, data on operating state belonging to at least one cycle may be required.

[0095] The productivity indicators may also be referred to as performance indicators. In some examples, the performance indicators may comprise key performance indicators (KPIs) that may be defined as a set of quantifiable measurements used to assess performance and efficiency of the mining operation at the mining site.Docket No.: PS56113PC00

[0096] At block 320, the method 300 comprises generating a result representation comprising information on the predicted subsequent operating state and the cycle to which the predicted subsequent operating state belongs. The result representation may be generated, at least in part, by the controller of the vehicle, and it may be in a form that allows using the information, on the predicted subsequent operating state and the cycle to which the predicted subsequent operating state belongs, to assess productivity of the mining vehicle, assess productivity of more than one vehicle in the mining site e.g. productivity of the entire mining operation, identify a need for interference e.g. human interference, identify potential issues with the mine that needs to be addressed, detect trends in mine operation, determine whether there are deviations from requirements of processes performed by the vehicle(s) in the mining site, control operation of the mining vehicle, control other vehicles in the mining environment, and make decisions and perform other actions regarding control of operation of vehicles in the mining site. For example, it may be determined, e.g. based on predicting or detecting or identifying one or more vehicle operating states e.g. a time that the vehicle took for hauling, whether the mining vehicle is dumping loads at an appropriate location rather than, e.g. closer to a draw point but not sufficiently close to a required dumping location. Thus, control of the vehicle operation may be performed, for compliance with regulations, including work performance regulations and safety regulations.

[0097] In some examples, the method comprises generating the result representation responsive to updating the current operating state to be set to the predicted subsequent operating state and / or responsive to determining that detection of the cycle to which the predicted subsequent operating state belongs is complete.

[0098] The result representation may be in a graphical, textual, audio, or in any other computer- implemented format, or a combination of formats, and the result representation may be transmitted to a suitable entity. In some examples, the result representation may be in the form of a computer- implemented report. In some example, the result representation may be communicated for display to an operator of the vehicle, e.g., on an onboard display or in an application on a mobile device. The result representation may additionally or alternatively be communicated to an external device, such as to the traffic control system.

[0099] Thus, at block 322, the method 300 may optionally comprise sending the result representation to an external device or system. The external device or system may be, for example, the traffic control system or another device or system external to the vehicle. In some examples, the predicted or identified operating states of the vehicle, and / or the productivity indicators, may be communicated to the traffic control system or another external device or system in real time, once they are determined. The traffic control system, e.g. a fleet controller, may use the received result representation for various actions such as for automating fleet planning, determining use of the fleet,Docket No.: PS56113PC00 implementing transformation of one or more mining vehicles from a non-autonomous mode to an autonomous mode, for proactive maintenance and / or predictive maintenance of vehicles.

[0100] At block 324, the method 300 may optionally comprise controlling the mining vehicle. The controlling of the vehicle may be performed by the vehicle controller. In some cases, additional data and / or instructions may be received from an external control system, e.g. the traffic control system, for controlling operation of the vehicle in dependence on the predicted subsequent operating state. The controlling of the vehicle may include controlling transitioning of the vehicle from the preceding operating state, which has been confirmed as the current operating state, to a next operating state. The mining vehicle may be controlled in various other ways in dependence on the predicted subsequent operating state after the veracity of the state transition from the preceding operating state to the predicted subsequent operating state is confirmed. The control may include using the determined one or more productivity indicators.

[0101] In some examples, the processing at block 324 may be performed prior to performing the processing at block 322, or at least in part simultaneously with the processing at block 322. As the mining vehicle is operating in the mining environment, the vehicle may be communicating with a central control system such as e.g. the fleet controller, and the vehicle may be receiving information and / or instructions from the fleet controller and sending data to the fleet controller. Thus, the vehicle may be sending, in real time, predicted and confirmed operating states and productivity indicators determined by the vehicle controller as the operating states and productivity indicators are determined.

[0102] In general, the specific order of the blocks is shown in FIG.3 by way of example only, and the processing at the blocks of FIG.3 may be performed in another suitable order.

[0103] In some examples e.g. when the vehicle is not an autonomous vehicle, the method, system, and kit in accordance with examples of the present disclosure may be used in implementing transformation of the mining vehicles from a non-autonomous mode to an autonomous mode. In the autonomous mode, a next operating state for the vehicle may be predicted based on a previously identified operating state. Other forms of control may be implemented to transition, reconfigure, and / or otherwise configure the mining vehicle to operate as an autonomous vehicle.

[0104] The approach in accordance with examples of the present disclosure provides various advantages compared to existing solutions. Thus, it allows automatic prediction or detection of operating cycles, which allows improved control over productivity. In this way, overall efficiency and productivity of mining operations may be increased. Furthermore, because of the determining of operating states and productivity indicators being performed in real time, timely control and intervention may be implemented in fleet management systems. Valuable information on fleet operations is generated, enabling decisions to be made on dynamic operational assignments. As a furtherDocket No.: PS56113PC00 advantage, the provided approach is equipment agnostic, such that it may be integrated with various mining machinery and equipment without requiring extensive modifications or specialized hardware. This flexibility not only simplifies implementation, but also allows for wider adoption in different mining configurations. Security of fleet operation may be increased because operating states are identified for mining vehicles automatically i.e. without involvement of a human operator, and

[0105] Real-time detection capability is another advantage of the present disclosure, which allows instant identification of operating states, enabling timely actions and interventions as needed. This timely response contributes to the overall efficiency and safety of mining operations.

[0106] Additionally, the provided system, vehicle, kit, and methods therein are developed and configured to operate with low computational resource requirements. This feature allows for the provided technique to be implemented in onboard vehicle controllers which typically have limited processing capabilities. By using resources efficiently, the provided technique may ensure optimal performance even in challenging computing environments.

[0107] FIG.4 illustrates an example of a training pipeline or process 400 for training a machine- learning model, in accordance with example embodiments of the present disclosure. The training process 400 is performed to obtain a machine-learning model that can be used to detect or predict operational or operating states of the mining vehicle. The machine-learning model may be selected from one or more candidate machine-learning models. The operating states may be identified and defined from patterns detected in signals acquired by an IMU sensor. The training process 400 may be performed in advance, and a resulting trained machine-learning model may be provided, for example, as part of a kit provided in accordance with examples of the present disclosure. In some examples, the training may be performed by a traffic control system or another external control system.

[0108] In some examples, the training pipeline may be part of the method that e.g. a vehicle controller configured in accordance with examples of the present disclosure may perform the model training. For example, a kit, comprising an inertial sensor and computer program product, when installed on a mining vehicle such as on a vehicle controller, may allow the vehicle controller to perform training of the machine-learning model. In some examples, a model performance monitoring unit or module may be implemented as part of the method herein, wherein the model performance monitoring module may measure precision metrics in predictions and variation of data quality that may affect the model's predictions. Based on the measurements performed by e.g. the model performance monitoring unit, the training pipeline may be executed automatically, thereby generating a new machine-learning model according to new conditions e.g. conditions and properties of the mining environment in which the vehicle operates and other factors, conditions, and properties.Docket No.: PS56113PC00

[0109] At block 402, IMU data is acquired that comprises one or more inertial sensor measurements acquired from one or more vehicles operating in a mining environment or mining site. The IMU data, which may be referred to as training data, is acquired as the one or more vehicles are operating in the mining environment. The one or more vehicles may be any suitable type of a vehicle with a movable implement to which one or more inertial sensors such as an IMU may be coupled. In some examples, each of the vehicles used to acquire training data comprises one IMU sensor coupled thereto, e.g., on the vehicle movable implement. For example, as shown schematically in FIGs.1B, the IMU sensor may be installed on a hydraulic arm of the vehicle that attaches the movable implement to the vehicle, in proximity to a work tool such as e.g. the bucket. In some examples, the inertial sensor, which may be one sensor, may be coupled to a joint of the arm of the movable implement.

[0110] In some examples, the training data may be updated as more data is acquired from vehicles that are configured to implement a method in accordance with examples of the present disclosure.

[0111] At block 405, exploratory data analysis (EDA) may be performed on the acquired one or more inertial sensor measurements. The EDA may be performed to ensure that the resulting model is agnostic to a machine and a mining site. The EDA stage may include pattern analysis, statistical analysis, frequency analysis, and other types of processing. The objective of the exploratory analysis of the one or more inertial sensor measurements, such as IMU sensor data, is to find patterns that allow identifying a time window in which the vehicle performs the different operating states. As discussed above, the operating states include loading, when the vehicle adjusts and fills with material a vehicle’s bucket; moving, while transporting the material from one point to another; and dumping, when the vehicle performs a maneuver in which the material is deposited at a dumping or emptying point such as e.g. a pit, ore pass, stockpile, etc. In addition, periods of inactivity when the vehicle is on but idle may be included in the set of operating states that may be predicted / detected using the technique in accordance with examples of the present disclosure. The moving may be distinguished as hauling such as moving with the load of the material or translating, such as moving without the material e.g. with an empty bucket.

[0112] In some examples, a time window may be configured to train the model.

[0113] In some examples, the one or more inertial sensor measurements used for the training stage may be simulated data, or a combination of simulated data and actual sensor data acquired from one or more vehicles as they are operating in a mining site.

[0114] At block 407, the process 400 may comprise data preprocessing, which may involve various techniques to prepare the data for training. During this phase or stage, different data cleaning methods may be applied, such as e.g. outlier filtering, elimination of null or corrupted values, and signalDocket No.: PS56113PC00 smoothing using different techniques such as rolling, low-pass filtering or Fourier transform. In some examples, calibration of IMU sensor axis position may be performed. Also, secondary signals and / or additional transforms may be obtained that provide relevant information for a feature extraction stage, such as, e.g., envelope calculation, Fourier transform, logarithmic transform, wavelets, empirical mode decomposition, among others. Preprocessed data may be generated as a result of preprocessing of the one or more inertial sensor measurements.

[0115] At block 409, the process 400 may comprise performing feature extraction to identity one or more features to be used in the machine-learning model. For example, the preprocessed data may be labeled according to observed and identified patterns, distinguishing between different operating states or events, such as hauling / moving, loading, unloading, etc. The most relevant patterned signals and axes may be selected chosen to form variables, and sliding time windows are designed or generated with overlapping, adjusted to a duration of the operating states. In some examples, for each selected axis, statistical indicators may be calculated using the time windows as a basis, and a class is assigned with a name of the operating state based on the previously labeled pattern. These calculated characteristics and the class constitute the variables to be used for modeling. Non-limiting examples of the statistical indicators may comprise standard deviation, mean, median, kurtosis, skewness, etc.

[0116] At block 411, the process 400 may comprise training one or more candidate machine- learning models and selecting a model from the candidate models, which may be performed using any one or more of various approaches. A model may be selected from the candidate machine-learning models. For example, training data may be fitted to different supervised classification models. In some examples, the models comprise decision tree models, which use a class variable as a target. One or more models may be trained and assessed, and a model that gives a most accurate performance, and is less computationally expensive than other models, may be selected. For the evaluation, evaluation metrics such as e.g., one or more out of f1-score, recall, precision, accuracy, area under the curve (AUC), and others, may be considered. Also, techniques such as e.g. confusion matrices, classification reports, and others related to an importance of the variables may be used. In some examples, a model may be selected which may deliver results above 90% for all evaluation metrics used, before moving onto a model validation stage. In some examples, a model may be selected which may deliver results above 95% for all evaluation metrics used, before moving onto a model validation stage.

[0117] At block 413, the process 400 may comprise validating the trained machine-learning model generated at block 411. The validation may be performed using one or more of various validation techniques. For example, to ensure appropriate performance of the selected model, predictions may be made with unlabeled data, and metrics may be generated to evaluate and validate the model. The validation state is used to access whether the model, selected at the training and modelDocket No.: PS56113PC00 selection stage at block 411, is able to perform, i.e. recognize operating states of a mining vehicle, with performance similar to that exhibited by that model at the training and model selection stage.

[0118] It should be noted that a choice of a final model is an iterative process, subject to changes in the training data, so that monitoring, readjustment, and / or retraining may be considered in order to ensure that the modeling responds to an updated, most recent, representation of reality. Accordingly, as shown schematically in FIG.4, by arrow a1, the process 400 may return, from block 413, to block 411, to continue training the model, which may be performed iteratively.

[0119] Further, as shown in FIG.4, at decision block 415, it may be determined whether the validation is complete. If this is the case, the trained model may be output at block 417. Otherwise, as shown in FIG.4, the process 400 may return to block 413 to continue the validation process of the model.

[0120] FIG.5 illustrates an example of a method or process 500 for monitoring and controlling operation of a mining vehicle from a plurality of mining vehicles in a mining site. The mining vehicle may be, e.g., any of the vehicles 12a-12d shown in FIG.1A, or another type of a mining vehicle in a mining environment or site such as e.g. mining site 10 of FIG.1A. The mining site may comprise a plurality of mining vehicles, and one or more of the plurality of mining vehicles may comprise a controller comprising processing circuitry configured to perform the method or process 500.

[0121] The mining vehicle comprises a movable implement with at least one inertial sensor coupled thereto. The mining vehicle may be included in a fleet of vehicles comprising one or more vehicles. The method or process 500 of FIG: 5 illustrates in more detail the method 300 for monitoring and controlling operation of a mining vehicle depicted in FIG.3, and some of the processing, already described in connection with FIG.3, is not described in detail in connection similar processing as part of the process 500. Also, it should be appreciated that the description of the process 500 of FIG.5 applies to the process shown in FIG.3, and vice versa.

[0122] The method 500 may begin at a suitable time, e.g., as the mining vehicle is started and begins performing its work function in the mining environment or site. The mining vehicle may be started, e.g., in a beginning of a shift or at another time. The controller of the vehicle may be activated to perform the method 500.

[0123] At block 502, the method 500 comprises, as the vehicle is operating in the mining environment, acquiring one or more inertial sensor measurements from at least one inertial sensor coupled to a movable implement of the vehicle. In example embodiments, the at least one inertial sensor comprises one inertial measurements unit (IMU) comprising an accelerometer, a gyroscope, and a magnetometer. The inertial sensor measurements may be acquired in real time, as the vehicle isDocket No.: PS56113PC00 operating in the mining environment. In some examples, the at least one inertial sensor comprises one, i.e. single. IMU coupled to a movable implement of the vehicle.

[0124] One or more inertial sensor measurements may be received, handled, stored. and distributed for processing in following stages. In some examples, communication may be handled using a communication protocol such as e.g., Message Queue Telemetry Transport (MQTT), REdis Serialization Protocol (Redis), and / or other communication protocols. Any suitable communication protocol, e.g., an IoT protocol, may be used. The sensor data may be stored in memory of the vehicle controller, until an amount which is required for a next stage is obtained.

[0125] At block 504, it may be determined whether required amount of sensor data, i.e. the one or more inertial sensor measurements, is acquired. Responsive to determining that the required amount of sensor data is acquired, the process 500 may follow to block 506. Alternatively, responsive to determining that the required amount of sensor data is not yet acquired, the process 500 may return to block 502 as shown in FIG.5.

[0126] At block 506, preprocessing of the acquired one or more inertial sensor measurements, also referred to as sensor data, is performed. The sensor data may be processed by applying e.g. smoothing and / or other techniques, such as e.g. rolling, low-pass filtering, and / or Fourier transform. Magnitude of the detected signals may be adjusted, which may involve scaling or normalization of data values of the sensor data. For example, a minimum and maximum range of the data values can be defined and the data is thereby scaled within that range.

[0127] It should be noted that the preprocessing, e.g., treatment and scaling, of the sensor data is performed such that it matches corresponding techniques used during model training, e.g., as described in connection with blocks 405 and 407 of FIG.4.

[0128] At the sensor data preprocessing stage, the preprocessed data may be stored, e.g., in the memory of the vehicle controller, until an amount of data required to proceed to the next stage is obtained.

[0129] Accordingly, at block 508, it may be determined whether required amount of sensor data, i.e. the one or more inertial sensor measurements, is acquired for the preprocessing performed at block 506. Responsive to determining that the required amount of sensor data is acquired, the process 500 may follow to block 510. Alternatively, responsive to determining that the required amount of sensor data is not acquired, the process 500 may return to block 502 as shown in FIG.5.

[0130] At block 510, the method 500 comprises performing feature extraction from the preprocessed sensor data, such as processed one or more inertial sensor measurements. One or more features may be extracted for inputting into a trained machine-learning model. Non-limiting examples of the one or more features comprise statistical indicators such as e.g. standard deviation, mean, median,Docket No.: PS56113PC00 kurtosis, skewness, etc.; and indicators generated as a result of sensor signal analysis such as e.g. Fourier transform, wavelet transform, signal energy, peaks, zero crossings, etc. In some examples, from the processed sensor data, magnitudes of accelerometers and gyroscopes present in the IMU may be considered, each one in its 3 axes: X, Y, Z. Thus, in some examples, at least nine sources of sensor data may be available for analysis and identification of vehicle operating states. Feature may be extracted from this data and those features may be used that are most representative of a current state of the vehicle, as predicted based on movements of a bucket or another tool of a movable implement. In some examples, a sliding window may be defined to calculate parameters that characterize one or more patterns that may be detected and / or classified in the preprocessed sensor data. As one example, one type of a sliding window used may have a sampling frequency of 4Hz, duration of 18 seconds, and a step of 12 seconds, although any other values may be used, as embodiments of the present disclosure are not limited to any specific sliding window.

[0131] The one or more features extracted at block 510 are similar or the same features as those used for training the machine-learning model, e.g., as described in connection with block 409 of FIG.4.

[0132] At block 512, the method 500 comprises applying a trained machine-learning model to the inertial sensor measurements to predict a subsequent operating state of the set of operating states of the vehicle, wherein the subsequent operating state belongs to a cycle comprising a sequence of operating states from the set of operating states. Using the features extracted as a result of the feature extraction at block 510, predictions are made with the previously trained machine-learning model, regarding a subsequent operating state of the vehicle. As mentioned above, the subsequent operating state is a state that is currently being predicted / detected / identified, in real time, as the vehicle is operating in the mining site.

[0133] At block 514, the method 500 optionally comprises evaluating a veracity of a state transition from a preceding operating state to the predicted subsequent operating state, which is performed e.g. as described in connection with FIG.6. The evaluating of the veracity of a state transition logic may involve verifying the transitions between the operating states. It should be noted that the evaluating of the veracity of the state transition may be performed for an operating state after an initial operating state. In other words, if there is a previously recorded, preceding operating state from which the vehicle transitions to the subsequent operating state, the veracity of the transition may be evaluated. If the subsequent operating state, currently being predicted based on real-time sensor data, is the initial state being predicted for the vehicle, there is no need to perform the evaluation of the veracity of the state transition, as shown in more detail in connection with FIG.6.

[0134] At block 516, the method 500 comprises determining whether the veracity of the state transition from the preceding operating state to the predicted subsequent operating state is confirmed.Docket No.: PS56113PC00 If this is the case, the method 500 proceeds to block 518. Alternatively, responsive to determining that the veracity of the state transition from the preceding operating state to the predicted subsequent operating state is not confirmed, the process 500 may return to block 502 as shown in FIG.5. When the veracity of the state transition is not confirmed, the predicted subsequent operating state may be considered not to be valid and it may be discarded. The predicted subsequent operating state may be discarded when it is predicted as a false positive or a false negative.

[0135] At block 518, the method 500 comprises determining at least one productivity indicator from the predicted subsequent operating state. This may be performed similarly to the processing at block 318 of FIG.3 and the description of the processing is therefore not repeated herein.

[0136] At block 520, the method 599 comprises generating a result representation comprising information on the predicted subsequent operating state and the cycle to which the predicted subsequent operating state belongs. The generating of the result representation may be performed similarly to the processing at block 518 of FIG.5 and the description of the processing is therefore not repeated herein.

[0137] At block 522, the method 500 may comprise sending the result representation to an external device or system. The external device or system may be, for example, the traffic control system or another device or system external to the vehicle. In some examples, the predicted or identified operating states of the vehicle, and / or the productivity indicators, may be communicated to the traffic control system or another external device or system in real time, once they are determined. The traffic control system, e.g. a fleet controller, may use the received result representation for various actions such as for automating fleet planning, determining use of the fleet, implementing transformation of one or more mining vehicles from a non-autonomous mode to an autonomous mode, for proactive maintenance and / or predictive maintenance of vehicles.

[0138] Similar to processing at block 324 of the process or method 300 of FIG.3, the method 500 described in connection with FIG.5 may comprise controlling the mining vehicle in dependence on the predicted subsequent operating state. The controlling of the vehicle may be performed by the vehicle controller. In some cases, additional data and / or instructions may be received from an external control system, e.g. the traffic control system, for controlling operation of the vehicle in dependence on the predicted subsequent operating state. The controlling of the vehicle may include controlling transitioning of the vehicle from the preceding operating state, which has been confirmed as the current operating state, to a next operating state. For example, if the predicted subsequent operating state is a dumping or uploading operating state and the veracity of the state transition to this state is confirmed, the vehicle may be controlled to move to e.g. a draw point where the mined material is located. In some examples, the controlling of the vehicle to perform the next operating state may be performed simultaneously withDocket No.: PS56113PC00 monitoring the operation of the vehicle, in accordance with examples of the present disclosure, to predict the next operating state, e.g. to verify that the control is performed accurately. In this way, the ability of the process in accordance with the present disclosure to predict vehicle operating states may be refined.

[0139] The mining vehicle may be controlled in various other ways in dependence on the predicted subsequent operating state after the veracity of the state transition from the preceding operating state to the predicted subsequent operating state is confirmed. The control may include using the determined one or more productivity indicators.

[0140] In some examples, for the predicted subsequent operating state of the vehicle, a veracity of a state transition from a preceding operating state to the predicted subsequent operating state may be evaluated. The veracity may indicate a correctness of prediction that a current operating state, such as a previously recoded operating state, is followed by the subsequent operating state. The subsequent operating state is a state that is currently being predicted based on sensor data acquired in real time from an inertial sensor e.g. one IMU sensor, but the subsequent operating state may be not taken as a current operating state which is used in further processing, until the veracity of transition to this state is verified or confirmed.

[0141] FIG.6 illustrates an example of a method or process 600 for evaluating the veracity of the state transition from the preceding operating state to the predicted subsequent operating state. The method 600 may be performed e.g. by the vehicle controller. The method or process 600 may be performed as part of processing at blocks 514 and 516 of FIG.5, and as part of processing at block 312 of FIG.3. As mentioned above, the methods in accordance with examples of the present disclosure may involve maintaining a variable for a current operating state, which variable is updated as a next operating state is predicted / detected / identified and a veracity of state transition to this state is confirmed. The variable for the current operating state may be stored, e.g., in a current operating state registry in memory of the controller of the vehicle, as shown in FIG.2. The current operating state registry 224 may store one or more parameters related to the current operating state, such as, e.g. features used to predict the current operating state, a duration of time for the current operating state, a start time and an end time for the current operating state, and other parameters.

[0142] At block 602, the predicted subsequent operating state may be obtained. The process 600 may be performed as the subsequent operating state is predicted, as described above.

[0143] At decision block 604, it may be determined whether the predicted subsequent operating state is an initial operating state. The initial state may be a previously recorded operating state before a system e.g. the vehicle controller is initialized. For example, the previously recorded operating state may be stored in operating states registry 222 and in current operating state registry 224 shown in FIG.Docket No.: PS56113PC00 2. In cases in which there is no previously recorded operating state, the initial operating state may be a first operating state predicted for the vehicle as the vehicle starts operating in the mining site.

[0144] Responsive to determining that the predicted subsequent operating state is the initial operating state, the process 600 may follow to block 606 where the variable for the current operating state is set to the predicted subsequent operating state, or it may be said that the current operating state is set to the predicted subsequent operating state.

[0145] Responsive to determining that the predicted subsequent operating state is not the initial state, the process 600 may follow to decision block 608 where the process 600 comprises determining whether the predicted subsequent operating state is different from the preceding operating state, wherein the current operating state is set to the preceding operating state.

[0146] Responsive to determining that the predicted subsequent operating state is not different from the preceding operating state, the process 600 may follow to block 610 where one or more parameters related to the current operating state may be updated. Thus, if it has been previously predicted that the vehicle was in a certain operating state, and it has been now predicted that the vehicle is, e.g. still or again, in the same operating state, this indicates that the same operating state is considered to be a current operating state. When it is determined that the current operating state is maintained, the one or more parameters for this state, e.g., its duration, end time, and others, may be updated as shown at block 610.

[0147] At block 612, responsive to determining that the predicted subsequent operating state is different from the preceding operating state, the vehicle controller may acquire a threshold transition time, e.g., from the memory of the vehicle controller. In some examples, the threshold transition time may be obtained from a transition matrix. The transition matrix, an example of which is shown in Table 1, may store threshold transition times for transitions from one operating state to another. For example, as shown in Table 1, the times ti, with i ranging from 1 to 12 in this example, are stored for transition between inactivity or idle state, moving state, loading state, and dumping state. In some examples, the transition matrix may also include times for transition to and from two types of moving states such as hauling (moving with a load) and translating (moving without a load). The transition matrix may include times for transition to and from any suitable types of operating states in which the vehicle may operate.

[0148] Table 1.Docket No.: PS56113PC00

[0149] At decision block 614, responsive to determining that the predicted subsequent operating state is different from the preceding operating state, the process 600 further comprises determining whether the predicted subsequent operating state is predicted to occur within a margin for the threshold transition time indicating an expected time for transition from the preceding operating state to the predicted subsequent operating state. The margin for the threshold transition time may indicate an allowed deviation from the threshold transition time for the transition from the preceding operating state to the preceding operating state. As one example, when the preceding operating state for the vehicle is moving and the predicted subsequent operating is loading, the transition matrix may specify that the threshold transition time is 5 seconds. The margin for the threshold transition time from the moving state to the loading state may be, e.g., 1 second, indicating that the valid state transition from moving to loading is expected to occur from 4 to 6 seconds, inclusively. If the state transition to the loading state is predicted earlier than 4 seconds after the moving state is detected, the veracity of this state transition may not be confirmed. Similarly, the veracity of the state transition from moving to loading may not be confirmed if the state transition to the loading state is predicted later than 6 seconds after the moving state is detected.

[0150] Accordingly, responsive to determining at block 614 that the predicted subsequent operating state is predicted to occur outside of the margin for the threshold transition time indicating the expected time for transition from the preceding operating state to the predicted subsequent operating state, the process 600 proceeds to block 616 where the predicted subsequent operating state may be discarded since it may not be predicted or detected correctly.

[0151] At block 618, responsive to determining at block 614 that the predicted subsequent operating state is predicted to occur within the margin for the threshold transition time indicating the expected time for transition from the preceding operating state to the predicted subsequent operating state, the process 600 includes confirming the veracity of the state transition from the preceding operating state to the predicted subsequent operating state and updating the current operating state to be set to the predicted subsequent operating state.Docket No.: PS56113PC00

[0152] At block 620, the predicted subsequent operating state may be registered, e.g., in the operating states registry as shown in FIG.2.

[0153] The process 600 for evaluating the veracity of the state transition from the preceding operating state to the predicted subsequent operating state may be performed continuously, as the vehicle is operating at the mining site, and performance of work or task(s) by the vehicle is monitored via at least one, e.g. one or single in some examples, inertial sensor.

[0154] From the processing at blocks 606, 610, and 616, the process 600 may return to acquiring one or more inertial sensor measurements, as shown e.g. at block 302 of FIG.3 and block 502 of FIG. 5.

[0155] To perform the method steps of the method for monitoring and controlling operation of a mining vehicle from a plurality of mining vehicles in a mining site, the vehicle controller may be configured to perform the processing described in connection with FIGs.3, 5, and 6. In some examples, the vehicle controller may be configured to additionally perform, at least in part, the processing described in connection with FIG.4 which involve training a machine-learning model that is configured to be applied to inertial sensor measurements data to predict or identify or classify vehicle operating states.

[0156] FIG.7 illustrates an example of a method that may be performed by a control system such as e.g. control system 20 shown in FIGs.1A and 2. The control system 20, which may be a traffic control system, comprises processing circuitry 22 e.g. at least one processor 22 that is configured to coordinate movements of one or more vehicles out of a plurality of vehicles in a mining site. Each vehicle of the one or more vehicles may comprise a vehicle controller such as, e.g. vehicle controller of FIGs.1A and 2, the vehicle controller being configured to communicate with the control system 20.

[0157] At block 702, the control system may receive, from the vehicle of the one or more vehicles, a representation comprising information on a predicted subsequent operating state of the vehicle and a cycle to which the predicted subsequent operating state belongs. The representation may be a result representation generated by the vehicle controller of the vehicle when the vehicle controller is configured to perform the method in accordance with examples of the present disclosure.

[0158] At block 704, the control system may generate at least one command to the vehicle in dependence on the received representation.

[0159] At block 706, the control system may send the at least one command to the vehicle. The generating and sending may be performed in the same processing step or block.

[0160] In an aspect, a kit is provided which comprises at least one inertial sensor and computer program product comprising computer-executable which, when executed by processing circuitry on a mining vehicle from a plurality of mining vehicles in a mining site, cause the processing circuitry toDocket No.: PS56113PC00 perform the method in accordance with examples of the present disclosure. The method comprises, as the vehicle is operating in the mining site, acquiring one or more inertial sensor measurements from at least one inertial sensor coupled to a movable implement of the vehicle; applying a trained machine- learning model to the inertial sensor measurements to predict a subsequent operating state of a set of operating states of the vehicle, wherein the subsequent operating state belongs to a cycle comprising a sequence of operating states from the set of operating states; determining at least one productivity indicator from the predicted subsequent operating state; and generating a result representation comprising information on the predicted subsequent operating state and the cycle to which the predicted subsequent operating state belongs.

[0161] The kit may be configured to be deployed on the mining vehicle. In some examples, the computer program product of the kit may be installed on a vehicle controller of the mining vehicle. In some examples, the computer program product of the kit may be implemented as a controller device, such that the controller device may be installed and deployed on the vehicle in addition to existing one or more controllers of the vehicle. In some examples, in the mining vehicle, the at least one inertial sensor comprises one inertial sensor. The one inertial sensor may comprise one inertial measurements unit, IMU, comprising an accelerometer, a gyroscope, and a magnetometer. In some examples, the movable implement of the mining vehicle comprises a bucket.

[0162] In some examples, the method may comprise evaluating a veracity of a state transition from a preceding operating state to the predicted subsequent operating state. In some examples, the evaluating the veracity of the state transition from the preceding operating state to the predicted subsequent operating state comprises: responsive to determining that the predicted subsequent operating state is not an initial operating state, determining whether the predicted subsequent operating state is different from the preceding operating state, wherein a current operating state is set to the preceding operating state; responsive to determining that the predicted subsequent operating state is different from the preceding operating state, determining whether the predicted subsequent operating state is predicted to occur within a margin for a threshold transition time indicating an expected time for transition from the preceding operating state to the predicted subsequent operating state; and, responsive to determining that the predicted subsequent operating state is predicted to occur within the margin for the threshold transition time, confirming the veracity of the state transition from the preceding operating state to the predicted subsequent operating state and updating the current operating state to be set to the predicted subsequent operating state.

[0163] In some examples, the method may comprise generating the result representation responsive to updating the current operating state to be set to the predicted subsequent operating stateDocket No.: PS56113PC00 and / or responsive to determining that detection of the cycle to which the predicted subsequent operating state belongs is complete.

[0164] In some examples, the trained machine-learning model may be trained in dependency on at least one property of the mining site.

[0165] In some examples, the method may comprise controlling the vehicle in dependence on the predicted subsequent operating state after the veracity of the state transition from the preceding operating state to the predicted subsequent operating state is confirmed. The vehicle may be controlled to transition to a next operating state. In some examples, the method may comprise assisting in automating the vehicle when the vehicle is transitioned from a non-autonomous control mode to an autonomous control mode.

[0166] In some examples, the at least one inertial sensor comprises one inertial measurements unit, IMU, comprising an accelerometer, a gyroscope, and a magnetometer. The

[0167] In some examples, the trained machine-learning model is agnostic to a model of the vehicle, a brand of the vehicle, and a manufacturer of the vehicle.

[0168] In some examples, the method may comprise sending the result representation to an external device or system. The external device or system may comprise a traffic control system such as e.g. a fleet controller configured to control mining vehicles in a mining environment or site.

[0169] In an aspect, a mining vehicle comprising the kit is provided.

[0170] Operational steps described in any of the exemplary aspects herein are described to provide examples and discussion. The steps may be performed by hardware components, may be embodied in machine-executable instructions to cause a processor to perform the steps, or may be performed by a combination of hardware and software. Although a specific order of method steps may be shown or described, the order of the steps may differ. In addition, two or more steps may be performed concurrently or with partial concurrence.

[0171] The terminology used herein is for the purpose of describing particular aspects only and is not intended to be limiting of the disclosure. As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items. It will be further understood that the terms “comprises,” “comprising,” “includes,” and / or “including” when used herein specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0172] It will be understood that, although the terms first, second, etc., may be used herein to describe various elements, these elements should not be limited by these terms. These terms are onlyDocket No.: PS56113PC00 used to distinguish one element from another. For example, a first element could be termed a second element, and, similarly, a second element could be termed a first element without departing from the scope of the present disclosure.

[0173] Relative terms such as “below” or “above” or “upper” or “lower” may be used herein to describe a relationship of one element to another element as illustrated in the Figures. It will be understood that these terms and those discussed above are intended to encompass different orientations of the device in addition to the orientation depicted in the Figures. It will be understood that when an element is referred to as being “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or intervening elements may be present. In contrast, when an element is referred to as being “directly connected” or “directly coupled” to another element, there are no intervening elements present.

[0174] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. It will be further understood that terms used herein should be interpreted as having a meaning consistent with their meaning in the context of this specification and the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0175] It is to be understood that the present disclosure is not limited to the aspects described above and illustrated in the drawings; rather, the skilled person will recognize that many changes and modifications may be made within the scope of the present disclosure and appended claims. In the drawings and specification, there have been disclosed aspects for purposes of illustration only and not for purposes of limitation, the scope of the inventive concepts being set forth in the following claims. REFERENCES

[0176] [1] Qi, Z., Gu, Q., Meng, Y., Bai, G., & Ding, D. (2023). The load-haul-dump operation cycle recognition based on multi-sensor feature selection and bidirectional long short-term memory network. Measurement and Control, 00202940231161569.

[0177] [2] Skoczylas, A., Rot, A., Stefaniak, P., & Śliwiński, P. (2023). Haulage Cycles Identification for Wheeled Transport in Underground Mine Using Neural Networks. Sensors, 23(3), 1331.

[0178] [3] Krot, P., Sliwinski, P., Zimroz, R., & Gomolla, N. (2020). The identification of operational cycles in the monitoring systems of underground vehicles. Measurement, 151, 107111.

[0179] [4] Stefaniak, P., Zimroz, R., Obuchowski, J., Sliwinski, P., & Andrzejewski, M. (2015). An effectiveness indicator for a mining loader based on the pressure signal measured at a bucket's hydraulic cylinder. Procedia Earth and Planetary Science, 15, 797-805.Docket No.: PS56113PC00

[0180] [5] Polak, M., Stefaniak, P., Zimroz, R., Wyłomańska, A., Śliwiński, P., & Andrzejewski, M. (2016). Identification of loading process based on hydraulic pressure signal. International Multidisciplinary Scientific GeoConference: SGEM: Surveying Geology & mining Ecology Management, 2, 459-466.

[0181] [6] Witulska, J., Stefaniak, P., Jachnik, B., Skoczylas, A., Śliwiński, P., & Dudzik, M. (2021). Recognition of LHD Position and Maneuvers in Underground Mining Excavations—Identification and Parametrization of Turns. Applied Sciences, 11(13), 6075.

Claims

Docket No.: PS56113PC00 CLAIMS What is claimed is:

1. A method for monitoring and controlling operation of a mining vehicle from a plurality of mining vehicles in a mining site, the method comprising: as the vehicle is operating in the mining site, acquiring (302, 502) one or more inertial sensor measurements from at least one inertial sensor coupled to a movable implement of the vehicle; applying (312, 512) a trained machine-learning model to the inertial sensor measurements to predict a subsequent operating state of a set of operating states of the vehicle, wherein the subsequent operating state belongs to a cycle comprising a sequence of operating states from the set of operating states; determining (318, 518) at least one productivity indicator from the predicted subsequent operating state; and generating (320, 520) a result representation comprising information on the predicted subsequent operating state and the cycle to which the predicted subsequent operating state belongs.

2. The method of claim 1, comprising (514) evaluating a veracity of a state transition from a preceding operating state to the predicted subsequent operating state.

3. The method of claim 2, wherein the evaluating the veracity of the state transition from the preceding operating state to the predicted subsequent operating state comprises: responsive to determining that the predicted subsequent operating state is not an initial operating state, determining (608) whether the predicted subsequent operating state is different from the preceding operating state, wherein a current operating state is set to the preceding operating state; responsive to determining that the predicted subsequent operating state is different from the preceding operating state, determining (614) whether the predicted subsequent operating state is predicted to occur within a margin for a threshold transition time indicating an expected time for transition from the preceding operating state to the predicted subsequent operating state; and responsive to determining that the predicted subsequent operating state is predicted to occur within the margin for the threshold transition time, confirming (618) the veracity of the state transition from the preceding operating state to the predicted subsequent operating state and updating the current operating state to be set to the predicted subsequent operating state.

4. The method of claim 2 or 3, comprising generating the result representation responsive to updating the current operating state to be set to the predicted subsequent operating state and / or responsive toDocket No.: PS56113PC00 determining that detection of the cycle to which the predicted subsequent operating state belongs is complete.

5. The method of any one of claims 1 to 4, wherein the trained machine-learning model is trained in dependency on at least one property of the mining site.

6. The method of any one of claims 2 to 5, comprising controlling (324) the vehicle in dependence on the predicted subsequent operating state after the veracity of the state transition from the preceding operating state to the predicted subsequent operating state is confirmed.

7. The method of any one of claims 1 to 6, wherein the at least one inertial sensor comprises one inertial measurements unit, IMU, comprising an accelerometer, a gyroscope, and a magnetometer.

8. The method of any one of claims 1 to 7, wherein the trained machine-learning model is agnostic to a model of the vehicle, a brand of the vehicle, and a manufacturer of the vehicle.

9. The method of any one of claims 1 to 8, comprising sending (322) the result representation to an external device or system.

10. A mining vehicle (12) comprising a movable implement, the mining vehicle comprising a vehicle controller (16) that comprises processing circuitry (220) configured to: as the vehicle is operating in the mining site, acquire one or more inertial sensor measurements from an inertial sensor coupled to on the movable implement of the vehicle; apply a trained machine-learning model to the inertial sensor measurements to predict a subsequent operating state of a set of operating states of the vehicle, wherein the subsequent operating state belongs to a cycle comprising a sequence of operating states from the set of operating states; determine at least one productivity indicator from the predicted subsequent operating state; and generate a result representation comprising information on the predicted subsequent operating state and the cycle to which the predicted subsequent operating state belongs.

11. The vehicle (12) of claim 10, wherein the processing circuitry is configured to evaluate a veracity of a state transition from a preceding operating state to the predicted subsequent operating state.

12. The vehicle (12) of claim 11, wherein the evaluating the veracity of the state transition from the preceding operating state to the predicted subsequent operating state comprises:Docket No.: PS56113PC00 responsive to determining that the predicted subsequent operating state is not an initial operating state, determining whether the predicted subsequent operating state is different from the preceding operating state, wherein a current operating state is set to the preceding operating state; responsive to determining that the predicted subsequent operating state is different from the preceding operating state, determining whether the predicted subsequent operating state is predicted to occur within a margin for a threshold transition time indicating an expected time for transition from the preceding operating state to the predicted subsequent operating state; and responsive to determining that the predicted subsequent operating state is predicted to occur within the margin for the threshold transition time, confirming the veracity of the state transition from the preceding operating state to the predicted subsequent operating state and updating the current operating state to be set to the predicted subsequent operating state and updating information on the cycle to which the predicted subsequent operating state belongs to indicate that the cycle includes the predicted subsequent operating state.

13. The vehicle (12) of claim 11 or 12, wherein the processing circuitry is configured to generate the result representation responsive to updating the current operating state to be set to the predicted subsequent operating state and / or responsive to determining that detection of the cycle to which the predicted subsequent operating state belongs is complete.

14. The vehicle(12) of any one of claims 11 to 13, wherein the processing circuitry is configured to control the vehicle in dependence on the predicted subsequent operating state after the veracity of the state transition from the preceding operating state to the predicted subsequent operating state is confirmed.

15. The vehicle (12) of any one of claims 10 to 14, wherein the at least one inertial sensor comprises one inertial measurements unit, IMU, comprising an accelerometer, a gyroscope, and a magnetometer.

16. The vehicle (12) of any one of claims 10 to 15, wherein the movable implement comprises a bucket.

17. A traffic control system (20) comprising at least one processor (22) for coordinating movements of one or more vehicles out of a plurality of vehicles in a mining site, wherein each vehicle of the one or more vehicles is a vehicle of any one of claims 10 to 16.

18. The traffic control system (20) of claim 17, wherein the at least one processor is configured to: receive, from the vehicle of the one or more vehicles, a representation comprising information on a predicted subsequent operating state of the vehicle and a cycle to which the predicted subsequent operating state belongs; generate at least one command to the vehicle in dependence on the representation; andDocket No.: PS56113PC00 send the at least one command to the vehicle.

19. A computer program product comprising computer-executable which, when executed by processing circuitry, cause the processing circuitry to perform the method of any one of claims 1 to 9.

20. A computer-readable storage medium, having stored thereon a computer program product comprising computer-executable instructions which, when executed by processing circuitry, cause the processing circuitry to perform the method of any one of claims 1 to 9.

21. A kit (18) comprising at least one inertial sensor and computer program product comprising computer-executable which, when executed by processing circuitry on a mining vehicle from a plurality of mining vehicles in a mining site, cause the processing circuitry to: as the vehicle is operating in the mining site, acquire one or more inertial sensor measurements from the inertial sensor coupled to a movable implement of the mining vehicle; apply a trained machine-learning model to the inertial sensor measurements to predict a subsequent operating state of a set of operating states of the vehicle, wherein the subsequent operating state belongs to a cycle comprising a sequence of operating states from the set of operating states; determine at least one productivity indicator from the predicted subsequent operating state; and generate a result representation comprising information on the predicted subsequent operating state and the cycle to which the predicted subsequent operating state belongs.

22. The kit (18) of claim 21, wherein the at least one inertial sensor comprises one inertial sensor.

23. The kit (18) of claim 22, wherein one inertial sensor comprises one inertial measurements unit, IMU, comprising an accelerometer, a gyroscope, and a magnetometer.

Images