Material imaging system and method

Abstract

A spectral light detection and ranging (Lidar) monitoring system to monitor an earthen working equipment including a load container having a plurality of ground engaging tools (GETs) is described along with a monitoring method. The spectral lidar system emits at least one light pulse into a field of view to generate spatial and spectral data. The spatial data includes a plurality of spatial datapoints and the spectral data includes a plurality of detection wavelengths, and are used to generate one or more spectral point clouds. Each spatial datapoint in the point cloud is classified to determine if spectral data which is the measured reflectance values at the detection wavelengths match those expected for materials of interest such as GET material and earthen material. The spatial datapoints are then filtered to generate material specific point cloud datasets and shape analysis is performed on the material point cloud datasets to identify missing GETs, worn or broken GETs, as well as carryback volume, and load volumes.

Description

MATERIAL IMAGING SYSTEM AND METHODBACKGROUND

[0001] In mining operations heavy equipment such as mining shovels are used to excavate and transfer earthen material. The mining shovel includes an operating implement such as a load container with multiple Ground Engaging Tools (GETs) located on the tip. During operations these can become damaged or worn which can create a hazard or reduce efficiency of mining operations. Monitoring methods and systems may be used to try and detect lost, broken, or worn GETs. However current methods and systems often perform poorly in cases where earthen material is located in close proximity to the GETs or when such material gets stuck on or between GETs, making detection and identification difficult. There is thus a need for improved monitoring methods and systems, or at least a useful alternative to existing methods and systems.SUMMARY

[0002] According to a first aspect, there is provided a spectral light detection and ranging (Lidar) monitoring system to monitor an earthen working equipment including a load container having a plurality of ground engaging tools (GETs). The spectral lidar system includes a spectral lidar apparatus and a computing apparatus. The spectral lidar apparatus is configured to emit a plurality of pulses of light into a field of view (FOV) at a plurality of emitted wavelengths, the plurality of emitted wavelengths including a plurality of detection wavelengths. The plurality of pulses of light is either a plurality of single pulses each including the plurality of emitter wavelengths or are a plurality of sets of single wavelength pulses where each pulse of light in a set is emitted at one of the plurality of emitter wavelengths. The spectral lidar apparatus is also configured to capture spatial data and spectral data at the plurality of detection wavelengths, the spectral data including an intensity of a return signal at each of the plurality of detection wavelengths. The computing apparatus includes at least one processor and at least one memory and a communications interface configured to operatively connect to the spectral Lidar apparatus. The at least one memory includes computer-executable instructions for configuring the at least one processor to trigger the spectral Lidar apparatus to emit the plurality of pulses of light into a FOV at a plurality of emitted wavelengths, the plurality of emitted wavelengths including a plurality of detection wavelengths, and to capture spatial data and spectral data at the plurality of detection wavelengths, the spectral data including an intensity of a return signal at each of the plurality of detection wavelengths collected at a plurality of spatial datapoints in the field of view. The at least one memory also includes computer-executable instructions for configuring the at least one processor to receive and process the spatial data and spectral data from the spectral lidar apparatus, to generate one or more spectral point cloud datasets for the FOV. Each of the one or more spectral point cloud datasets includes the plurality of spatial datapoints within the FOV and each spectral point cloud dataset includes an estimate of a reflectance at at least one detection wavelength for each spatial datapoint within the FOV. The at least one memory alsoincludes computer-executable instructions for configuring the at least one processor to classify each spatial datapoint in the one or more spectral point cloud datasets for the FOV to determine if one or more materials of interest are located at the respective spatial datapoint using the estimate of the reflectance at each of the detection wavelengths for the respective spatial datapoint, wherein the one or more materials of interest include at least one load container material. The at least one memory also includes computer-executable instructions for configuring the at least one processor to filter the spatial datapoints to generate a material point cloud dataset for each of the one or more materials of interest, wherein each material point cloud dataset includes the set of spatial datapoints in a field of view classified as the respective material of interest. The at least one memory also includes computer-executable instructions for configuring the at least one processor to perform a shape analysis on the one or more material point cloud datasets using one or more reference shapes for at least a portion of the load container to determine one or more of a missing GET, a worn or broken GET, a carryback volume, a load container volume, or a payload.

[0003] According to a second aspect, there is provided a computer implemented method to monitor an earthen material and earth moving equipment including a load container having a plurality of ground engaging tools (GETs) using a spectral light detection and ranging (Lidar) apparatus. The method includes generating, one or more spectral point cloud datasets for a Field of view (FOV) from spatial data and spectral data generated by a spectral lidar apparatus, wherein the one or more spectral point cloud datasets include a plurality of spatial datapoints within the FOV and each spectral point cloud dataset includes an estimate of a reflectance at at least one detection wavelength for each spatial datapoint within the FOV such that the one or more spectral point cloud datasets include an estimate of reflectance for each of the plurality of detection wavelengths. The method also includes classifying each spatial datapoint in the one or more spectral point cloud datasets to determine if one or more materials of interest are located at the respective spatial datapoint using the estimate of the reflectance at each of the detection wavelengths for the respective spatial datapoint, wherein the one or more materials of interest include at least one load container material. The method also includes filtering the spatial datapoints to generate a material point cloud dataset for each of the one or more materials of interest, wherein each material point cloud dataset includes the set of spatial datapoints in a field of view classified as the respective material of interest. The method also includes performing a shape analysis on the one or more material point cloud datasets using one or more reference shapes for at least a portion of the load container to determine one or more of a missing GET, a worn or broken GET, a carryback volume, a load container volume, or a payload.

[0004] Other aspects and features will become apparent to those ordinarily skilled in the art upon review of the following description of specific disclosed examples in conjunction with the accompanying figures.BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Figure 1A is a schematic view of a mining shovel having a spectral lidar monitoring system, with the mining shovel in an engage substage of an excavate stage of the operational cycle when the load container is empty in accordance with one example;

[0006] Figure 1 B is a schematic view of the system of Figure 1A, with the mining shovel later in the excavate stage when the load container is full in accordance with one example;

[0007] Figure 2 is a flow diagram for a representative method for monitoring an earthen working equipment using a spectral lidar monitoring system;

[0008] Figure 3A is a schematic diagram of a multi-spectral lidar apparatus that can be used in the spectral lidar monitoring system of Figure 1A in accordance with one example;

[0009] Figure 3B is a graph showing reference reflectance data as a function of wavelength for various metals, steel, and earthen material;

[0010] Figure 3C is a schematic diagram of a computer apparatus of the spectral lidar monitoring system of Figure 1A in accordance with one example;

[0011] Figure 4A is a perspective view of the operating implement of the mining shovel of Figure 1A in accordance with on example;

[0012] Figure 4B is a plan view of an operating implement of the mining shovel of Figure 1A with portions exploded in accordance with one example;

[0013] Figure 5A is a top down view of a first field of view of the spectral lidar apparatus in Figure 1A showing the load container and background earthen material according to an example;

[0014] Figure 5B is an enlarged view of a portion of Figure 5A showing two adjacent GETs which are partially obscured by earthen material;

[0015] Figure 5C is a view of a first spectral point cloud at a first detection wavelength for the field of view shown in Figure 5A according to an example;

[0016] Figure 5D shows a set of four spectral point clouds at each of four different detection wavelengths for the field of view shown in Figure 5A according to an example;

[0017] Figure 5E shows a plot of the measured reflectance for each detection wavelength for a spatial datapoint (xy, y1tz-) on a GET in Figure 5B according to an example;

[0018] Figure 5F shows a plot of the measured reflectance for each detection wavelength fora spatial datapoint (x2, y2, z2) on a piece of earthen material wedged between two GETs as shown in Figure 5B according to an example;

[0019] Figure 5G shows a representation of the material point cloud dataset for GET material according to an example;

[0020] Figure 5H shows a representation of the material point cloud dataset for earthen material according to an example;

[0021] Figure 6A is a top down view of a first field of view of the spectral lidar apparatus in Figure 1 B showing the load container, earthen material in the load container and background earthen material according to another example;

[0022] Figure 6B is a top view of a first spectral point cloud at a first detection wavelength for the field of view shown in Figure 6A according to an example;

[0023] Figure 6C shows a representation of the material point cloud dataset for GET material according to an example;

[0024] Figure 6D shows a representation of the material point cloud dataset for earthen material in or on the load container with the background material filtered according to an example;

[0025] Figure 7A is a schematic diagram of a machine learning (ML) training method according to an example;

[0026] Figure 7B is a schematic diagram of the architecture of a neural network model which may be trained using the machine learning training method shown in Figure 7A.

[0027] Figure 8A shows a side view of the end portion of the mining shovel showing the location of the spectral lidar apparatus, and the geometrical orientation of the handle and load container in the positions corresponding to Figures 1A and 1 B according to an example;

[0028] Figure 8B is an orthogonal view of figure 8A to show the view along the tip of the load container according to an example; and

[0029] Figure 8C which shows a scale transformation of a point cloud dataset according to an example.DETAILED DESCRIPTION

[0030] Referring to Figures 1A and 1 B, there is shown a spectral lidar monitoring system 200 used to monitor heavy equipment operations 100, such as an input and output of earthen material from earthen working equipment (also known as earth moving equipment), or other, during an operating cycle. In the example shown, the earthen working equipment is a mining shovel 102 generally configured for excavating earthen material from a mine face or work bench 114 or an earthen material pile in an open pit mine shown generally at 110. The operational cycle may include excavate, swing-full, dump, and swing-empty stages, or some combination of these stages, and may also include idle stages.

[0031] In the illustrated example of Figures 1A-1 B, the mining shovel 102 includes a frame 120 pivotably mounted on a track 122, a boom 124 mounted to the frame 120, a handle 125 pivotably mounted to the boom 124, an operating implement 126 mounted to the handle 125, and a control mechanism shown generally at 128 for controlling the mining shovel 102 to perform an excavate operation of the mine face 114 or a dump operation of a load 156 excavated from the mine face 114 or the earthen material pile. The track 122 enables the mining shovel 102 to traverse across the open pit mine 110. The operator may be autonomous, semi-autonomous, ornon-autonomous. The pivotal mounting 131 between the frame 120 and the track 122 allows the frame 120 to rotate about a z-axis of a coordinate axes 130 relative to the track 122.

[0032] In this example the coordinate axes 130 is orientated that the x and y axes form a ground plane, and the z axis defines a height (or depth) coordinate with respect to the ground plane. The operating implement 126, which includes a load container commonly referred to as a bucket, may be pivotally mounted to the distal end 136 of the handle 125. Rotation and movement of the handle 125 generally translates into corresponding rotation and movement of the operating implement 126; however, the pivotal mounting 135 between the handle 125 and the operating implement 126 may also allow the operating implement 126 to rotate about the y-axis of the coordinate axes 130 relative to the handle 125. In other examples, the mining shovel 102 may include additional, fewer, or alternative components which may be coupled to each other via other mountings.

[0033] In the example shown in Figure 1A and 1 B, the mining shovel 102 is a cable-based shovel, but other configurations are possible, e.g., a backhoe, a hydraulic front loader, a dragline bucket, a dredge cutterhead, a roll crusher, a cone crusher, and the like.

[0034] As shown in Figures 1A and 1 B, a spectral lidar monitoring system 200 is configured to implement a monitoring method 201 as shown in Figure 2 to detect lost, broken, or worn GETs or to assess carryback or load container volumes during the operational cycle. The spectral lidar monitoring system includes a spectral lidar apparatus 210, which is configured to capture both spatial data and spectral data from light signals reflected from objects in a field of view 236. The spectral lidar apparatus 210 is mounted to the mining shovel 102 via a mounting bracket 182. In another example, the spectral lidar apparatus may be a distributed apparatus in which the scene generator 230 is mounted to the boom 124, whilst the spectral source 220 and / or spectral receiver 240 are located in other locations such as the frame 120. In other examples the spectral lidar apparatus 210, or at least the scene generator 230, may be mounted in another location on the mining shovel 102 where the field of view 236 captures the load container during at least part of the operational cycle. The field of view 236 may also include a portion of the mine face 114 lying behind or beneath the operating implement 126 during the various stages of the operating cycle. The computing apparatus 250 may be mounted with the spectral lidar apparatus 210 or located in another location such as on the frame 120 and operatively connected to the spectral lidar apparatus 210, for example over a wired or wireless communication link.

[0035] A computing apparatus 250 is configured to control the spectral lidar apparatus 210 and analyze the spatial and spectral data to implement the monitoring method 201 illustrated in Figure 2. A computing apparatus 250 shown attached to the earth working equipment is used to control the spectral lidar apparatus to generate the spatial and spectral data, and to analyze the spectral scene to classify (identify) materials of interest and perform shape analysis to identify missing GETs, worn or broken GETs, load container volume, payload, or carry back volume, but other configurations are possible, e.g. the computer apparatus may be located locally on thespectral lidar apparatus 210 or remote. The computer apparatus 250 may also generate any alerts or electronic reports based on the analysis. An example of the computing apparatus 250 is illustrated in Figure 3C and is further discussed below.

[0036] In other examples, the earthen working equipment may be any other type of equipment which engages the mine face 114 or any other type of equipment designed to transfer loads from one location to another, such as, for example, a hydraulic face shovel, a dragline shovel, a backhoe excavator, haul truck, or wheel loader.

[0037] As illustrated in Figures 1A-1 B, the earthen working equipment 102 includes an operating implement 126 such as a load container 154 with multiple Ground Engaging Tools (GETs) 150 located on the tip. During mining and transfer operations the GETs may become worn or lost (e.g., break off) which reduces mining efficiency and can create hazards. There are many different examples of different GETs that may be attached to the operating implement 126, such as adapters, points, wing shrouds, center shrouds, and wear caps. One such example can be seen in Figure 4A. The spectral lidar apparatus 210 is illustrated in Figure 1A to be mounted so that the load container 154 and the GETS are located within the field of view 236 during at least part of the operational cycle (e.g., during the excavate stage).

[0038] Referring to Figure 2, the monitoring method 201 begins with step 202 of generating one or more spectral point cloud datasets for the FOV from the spatial data and spectral data generated by a spectral lidar apparatus 210. The one or more spectral point cloud datasets include a plurality of spatial datapoints within the FOV and each spectral point cloud includes an estimate of a reflectance at at least one detection wavelength for each spatial datapoint within the FOV. Various data structures could be used be used to the spectral point cloud data for the FOV. In one example the one or more spectral point cloud datasets for the FOV are a single spectral point cloud for the FOV that includes a plurality of estimates of reflectance for each of the plurality of detection wavelengths at each of the plurality of spatial datapoints within the FOV. That is an array of reflectance values for the detection wavelengths are stored for each spatial datapoint (i.e., multiple spectral measurements are associated with each spatial datapoint). In another example, the one or more spectral point cloud datasets for the FOV are a plurality of spectral point clouds for each of the detection wavelengths for the FOV wherein each spectral point cloud includes the plurality of spatial datapoints for the FOV and an estimate of a reflectance at the respective detection wavelength for each spatial datapoint within the FOV. Thus when there are multiple spectral point clouds for a FOV, the multiple spectral point clouds will collectively include an estimate of reflectance for each of the plurality of detection wavelengths for each spatial datapoint. The plurality of detection wavelengths is a set of detection wavelengths selected based on reflectance curves of one or more materials of interest (See Fig 3B). The spectral lidar apparatus 210 is configured to emit a plurality of pulses of light into a field of view (FOV) 236 at a plurality of emitted wavelengths. The plurality of pulses of light (also referred to as light pulses) is either a plurality of single pulses each including the plurality ofemitter wavelengths or are a plurality of sets of single wavelength pulses where each light pulse in a set is emitted at one of the plurality of emitter wavelengths. The emitted light pulses are reflected off objects (e.g., earthen material and wear members) within the field of view and the spectral lidar apparatus captures spatial data and spectral data at the plurality of detection wavelengths from the reflected light signals. The emitted light pulse may be a band of wavelengths that include each of the detection wavelengths of interest that aid in determining a given material, or the emitted light pulse may be composed of multiple independent wavelength bands, such that the emitted pulse includes all of the detection wavelengths. Any additional wavelengths included in the bands are either not measured or not used in subsequent analysis. A band pass filter may be used in the laser source to limit the emitted wavelengths to a specific band (or bands), or the laser source be configured to only emit over an emission band (or bands), or only at specific wavelengths. In some examples the detection wavelengths may be same as the emission wavelengths. In another example, the emitted light pulse could be filtered prior to emission, or configured to only emit the detection wavelengths. The detection wavelengths could either be included in a single emitted pulse, or in a set or sequence of emitted pulses for example where each emitted pulse is emitted at one of the detection wavelengths until all of the detection wavelengths have been emitted. The spectral data includes an intensity of a return signal at each of the plurality of detection wavelengths collected at a plurality of spatial datapoints in the field of view and can be used to estimate the reflectance at the spatial datapoint for each of the detection wavelengths, e.g., a spectral point cloud is created for each of the set of detection wavelengths. The spatial datapoint is the point on the object in the field of view that reflected the return signal and may be determined using a pointing direction of the emitted light pulse, and a time of flight of the return signal.

[0039] The one or more spectral point cloud datasets form a spectral scene (or a spectral scene dataset) for the field of view. The spectral scene may either be a single spectral point cloud (e.g., a single set of spatial datapoints) along with spectral data for each spatial datapoint that includes the set of reflectance values at each of the detection wavelengths (i.e. an array of spectral data for each spatial datapoint). Alternatively, the spectral scene may be a set of (monochrome) spectral point clouds where each spectral point cloud represents a given detection wavelength and includes an estimate of a reflectance at the respective detection wavelength for each spatial datapoint within the spectral point cloud of the field of view 236. That is, each spectral point cloud is a three dimensional (3D) image of the field of view 236 created from spatial datapoints which includes reflectance values (spectral data) at one or more of the detection wavelengths and thus we have one or more spectral point clouds for a FOV. The spectral scene data may be stored for each FOV and other equivalent data structures could also be used to store the spectral scene data. Reflectance at a spatial datapoint may be estimated as the ratio of the intensity of the reflected light pulse to the intensity of the emitted light pulse. As the intensity of emitted light pulse may be the same, or approximately the same, for each of the detectionwavelengths, the intensity measurement may be used as the estimate of the reflectance, or the intensity measurement may be processed with additional data, such as a measurement or estimate of the intensity of the emitted light pulse. This may be at the same detection wavelength or at another reference wavelength.

[0040] In step 203, material classification is performed to classify each spatial datapoint in the one or more spectral point clouds for the FOV to determine if one or more materials of interest are located at the respective spatial datapoint. The classification uses the estimate of the reflectance at each of the detection wavelengths to generate a spectral signature, such as plot of a curve of reflectance over the set of detection wavelengths, to determine if one or more materials of interest are associated with (e.g., located at) the spatial datapoint. Each material of interest will have a different spectral signature or curve plot at the detection wavelengths (See Figure 3B). The one or more materials of interest includes at least one load container material and may include one or more materials used to form the GETs, the earthen material being mined, or other materials which may appear in the field of view. The load container and the GETs may be made of the same material, or the GETs may be formed of different material to that used for other parts of the load container such as the walls or bottom surface of the load container. The materials of interest thus form a set of material classes. A machine learning system can be trained to classify each material of interest, for example using a training dataset including multiple estimates of a reflectance for each detection wavelength, or spectral signatures, for each of the one or more materials of interest. The training data may be spectral point clouds collected from a field of view including the one or more materials of interest. For example, the spectral lidar system could be used to first generate training data from capturing scenes containing GETs, load containers and earthen material of interest. Once the system is operational, operational data could also be used to retrain and improve the classifier. The training data may also be generated from or include synthetic or augmented training generated using reference spectral signatures. For example, a scene could be simulated based on known spectral signatures for the materials of interest and CAD models of GETs and load containers. In some examples the training dataset includes labeled spectral scenes (spectral point cloud datasets) in which each spatial datapoint is labelled with a material class. The machine learning classifier is trained to identify differences between the spectral signatures for each material (for example between steel and gravel in Fig 3B). As shown in Figure 3B, the detection wavelengths could be individual wavelengths 312, 314, 316 and 318 (and in other examples more than four could be used), or the detection wavelengths could be the entire emission band 310. In this later case the machine learning classifier will effectively learn which of the wavelengths in the band are useful for distinguishing materials. In other examples the material classification may be performed using an analytical comparison in which the measured reflectance at each detection wavelength is compared with a reference spectral signature for each material of interest. A combination of machine learning and analytical comparisons could also be performed. If the measured reflectance values do not match any ofthe materials of interest, the spatial datapoint may be classified as a generic material type such as “other material” or a residual material.

[0041] In step 204, material filtering is performed by filtering the spatial datapoints using the material classifications to generate material point cloud datasets for the one or more materials of interest. Each material point cloud dataset is the set of spatial datapoints in a field of view classified as the respective material. In some examples, the material point cloud dataset will only include materials of interest with any other material removed through filtering. A residual material point cloud dataset could also be generated for any datapoints for which the material was not a material of interest or for which the material could not be determined. Thus, for each material determined, the plurality of spectral point cloud datasets for a field of view (e.g., a spectral scene dataset) are filtered to a single material point cloud dataset that displays one or more materials or only materials of interest. Material point cloud datasets collected at different points in time may be combined to generate a more comprehensive point cloud representation of the material.

[0042] In step 205, a shape analysis is performed on the one or more material point cloud datasets using one or more reference shapes for at least a portion of the load container to determine one or more of a missing GET, a worn or broken GET, a carryback volume, a load container volume, or a payload. Determining the amount of carryback (ore left in the load container after dumping) or the volume of ore in the load container or payload may include first identifying the GETs and the load container and then identifying ore material with locations consistent with being on or in the load container. The material point cloud datasets may be scaled or normalized to a reference scale or size as will be further discussed below.

[0043] The shape analysis may compare at least the material point cloud dataset for the GET material(s) with a reference GET shape. In the case of a determining whether a GET is missing, the reference GET shape may be used to define a region or volume where the GET is expected and if the material shows up for a predetermined threshold number (e.g., two datapoints) within the known GET location, then the GET is determined to be present. This method has the advantage of being faster and more efficient by reducing processing spend to a method that needs to compare the entire shape of the GET to perform a determination. If the GET material(s) is different to the load container material(s), the shape analysis may also compare the material point cloud dataset for the load container material(s) with a reference load container shape obtained from an earlier measurement or from a known 3D model or spatial dataset of the bucket and wear parts (e.g., a drawing). The comparison enables identification of a missing GET and / or worn GET, so an alert (step 206 below) can be issued to an operator of heavy equipment or an operations center. The shape analysis may be performed using a machine learning method, or through a comparison between the observed shape and a reference shape. The reference shape may be obtained from a CAD model such as illustrated in Figure 4A below, calibration data, or previously collected GET material point cloud datasets. The location of each GET in the reference shape could be identified to generate a set of reference spatial datapoints,surfaces, or volumes. The GET material point cloud dataset could then be analyzed to extract the portion corresponding to each GET, and then comparing the locations of the observed spatial datapoints to the expected locations. For example, a reference volume could be determined for each GET or shroud and the presence of a threshold number of spatial datapoints corresponding to the GET material within the reference volume used to determine if a GET was present or absent. This enables detection in cases where the GET is mostly obscured by earthen material, but at the threshold number (e.g., at least one spatial datapoint) corresponding to GET material is visible allowing positive identification of the GET. The threshold number could be a single spatial datapoint or a larger number of points could be used. Alternatively surfaces or volumes could be fitted to the spatial points in the GET material point cloud dataset and compared to equivalent surfaces or volumes for the GET from the reference shape. Scaling or normalization may be performed to ensure the comparison is made at the same physical scale (see discussion below in relation to Figures 8A and 8B). These approaches only require the positive identification of a few points on the GET to allow fitting of a curve, e.g., representing an edge, or a surface, and thus can robustly and reliably interpolate over points on the GET obscured by earthen material. This enables more robust edge detection for wear detection and / or presence detection than Lidar and camera systems which can only identify locations of the points preventing identification of a partially or fully obscured GET. Similarly, camera systems struggle in such cases when the GET or edge is partially covered or fully covered by material as they are unable to reliably identify the material at each pixel of an image. In some examples, the earthen material point cloud dataset may be used to support a decision that the GET is present. For example, a GET may support earthen material, and thus the presence of earthen material at a height within a threshold range above an expected height of the GET surface based on the reference model may indicate the GET is present because if the GET was missing the earthen material should be observed at the background depth.

[0044] In addition to detecting missing GETs, the shape analysis may be used to identify worn or broken GETs, for example where the tip is worn down and has a shorter length than expected. The shape analysis may be performed to identify or quantify the amount of wear. For example, the measured location of an exterior edge or surface of a GET could be compared with the expected location obtained from a CAD model or from a previous measurement of the same GET (e.g., previously generated material point cloud dataset). The distance between the two could be assessed to determine how much a GET has worn or the distance could be compared with a threshold amount to enable a binary classification of a worn or not worn status. The wear may be determined based on the area of the GET being smaller than expected, or an aspect ratio being different to an expected ratio. In this case, the x dimension is the size expected but the y dimension is smaller than expected generating a different aspect ratio than expected ratio. In other examples a shape detector or edge detector could be used to estimate a perimeter shape of the GET and compared with the expected perimeter shape. Trained machine learning modelsmay also be used to assess missing, broken, or worn GETs based on the use of training data including examples of normal GETs and missing, broken, and worn GETs.

[0045] In some examples, depth information could be used to assist in determining if a GET was missing. For example, if the background earthen material is present at a spatial datapoint where a GET is expected and the earthen material is a depth lower than the expected depth of the GET, then the GET may be classified as missing. Alternatively, if earthen material is sitting on a GET and obscuring most of the GET, the determination of the earthen material may be used to identify that the GET is still present as the earthen material is being supported at a different height to background earthen material. This provides advantages over a LIDAR or vision only system, which due to the partial obstruction would struggle to determine the status and may require extra images (e.g., digging cycles) and take longer to determine presence due to the obscurity, or they may falsely assess the GET as missing and generate a false positive alert. In some examples, background and foreground earthen material point cloud datasets may be used to assist in assessing missing, broken, or worn GETs.

[0046] In some examples, multiple material point cloud datasets collected at different time points, and then could be combined to generate a combined (temporally summed) material point cloud dataset to account for transiently obscured datapoints. For example, in one operational cycle earthen material may completely obscure a first GET. However, in the next operational cycle the earthen material may be dislodged revealing the first GET. The combined material point cloud would then include the first GET. Any transient earthen material on a GET is effectively filtered out since the respective spatial datapoint is positively identified as GET material in at least one point cloud dataset.

[0047] Shape analysis may also be performed to estimate carryback volume or the volume of a load in a load container. The volume estimation may be performed using analytical techniques and / or trained machine learning models. The shape analysis may identify the carryback material and then estimate the volume of carryback material. The load container material point cloud dataset may be used to identify or anchor the location of the reference spatial model of the load container 154, and then the depths or surfaces of the reference spatial model can then be used as depth thresholds applied to the earthen material dataset. This may be performed in several ways, for example by locating the edges of the GETs or load container, or other spatial features of the GETs and / or the load container such as the edge of the lip 160 and rear wall 140. If the GET material is different to the load container, then the GET material point cloud dataset may also (or additionally) be used for this assessment. The depth (location) of the upper surface of the GETs and load container in the reference spatial model can then be used as depth thresholds applied to the earthen material dataset in order to identify earthen material that is lying in the load container or on the load container, as well as to determine a reference surface for volume estimation.

[0048] Each pointing direction of the spectral lidar apparatus represents a solid angle in the field of view and thus each spatial datapoint has an associated solid angle. Each spatial datapoint is the measurement of depth along the respective pointing direction, and thus has an associated area based on the depth and solid angle. The depth coordinate of a spatial datapoint of earthen material corresponds to the top of the earthen material, and this can be used with the location of the surface of the load container estimated from the reference data (model or previous measurement) to determine a depth of material. The depth of material can be combined with the associated area based on the solid angle of the spatial datapoint. The total volume of earthen material in a load container can then be determined by performing this calculation for at each spatial datapoint in the material point cloud dataset (limited to the load container) to determine the volume of material. This is equivalent to integrating over the surface of the earthen material. A similar volume estimation can be performed for carryback material. In other examples, a top surface could be fitted to the spatial datapoints in the earthen material point cloud dataset. The volume could then be estimated using the height difference between the top surface and the reference surface of the load container.

[0049] A payload, which is the total mass and composition of materials in the load container, may also be estimated. The spectral information allows identification of the specific material at each point on the top surface of material in the load container which then enables the payload to be estimated using the estimated volume and reference densities for the materials in the load container. In one example, the payload is estimated on a point-by-point basis. For each spatial datapoint in the load container material point cloud dataset, a density for the identified material is obtained from a memory that stores reference densities for multiple of materials. This density can be multiplied with the volume estimate for the respective spatial datapoint to give a mass of material associated with that spatial datapoint. This can be repeated for each point in the load container material point cloud dataset to determine the total mass of material in the load container. A composition can also be determined. If the load container point cloud dataset is comprised of single material the composition is that material. If the load container point cloud dataset is comprised of multiple earthen materials, then a composition can be determined by determining the percentage of each individual material identified in the load container point cloud dataset. In one example the composition is determined based on component masses. A mass component can then be calculated for each component material in the load container material point cloud dataset. The percentage of the respective material would then be its mass component divided by the total mass. In another example the percentage of spatial datapoints in the load container material point cloud identified as the respective material could be used to estimate composition. For example, for each material, a count of the number of spatial datapoints classified as the respective material could be obtained. The estimates could each be converted to a percentage by dividing by the total number of spatial data points in the load container point cloud dataset. Similarly, the respective surface area of each material could be determined and used todetermine composition. This approach would also provide an alternative way to determine the mass components. The composition could first be determined, and for each material, the percentage could be multiplied by the total volume of the load container and the associated density of the respective material obtained from the memory. The total mass is then obtained by summing the mass components.

[0050] A trained machine learning shape analysis model may also be used to identify a shape such as the load container or a portion of the load container and may also be used to directly estimate a volume of carryback, volume of a load of earthen material, or a payload. The training dataset used to train the machine learning shape analysis model may include multiple estimates of the shapes of GETs and / or load containers generated from reference shape data. This may be CAD data or a point cloud dataset of the respective object, for example from scene datasets in which points that are not part of the target object are filtered out. In some cases, the machine learning shape analysis model could also be trained to perform a volume estimation. For example, spectral scenes containing a load container containing an earthen material on or in the load container could be labelled to indicate what object is at each spatial datapoint (e.g., GET, load container, earthen material), along with the volume of earthen material on or in the load container. The trained machine learning shape analysis model directly then estimates the volume earthen material on or in the load container, or the payload, from an input spectral scene dataset.

[0051] The use of a material filter and shape analysis enable accurate and robust determination of GET locations and wear which enhances the precision of GET wear detection. This has advantages over vision and other LIDAR only devices in that the spectral data enables accurate determination of edges of differing material and the identification of partially obscured GETs. For example, in the case of earthen material partially covering a GET a lidar system would only recognize a blob of material and would be unable to identify the edges of the GETs whereas the spectral information can be used identity individual spatial datapoints to which a surface can be fitted thus compensating for partially obscured GETs and providing more robust edge detection or wear detection than Lidar and camera systems. The shape analysis may perform individual recognition of GETs and then perform local shape analysis on each GET, or the shape analysis may be performed on all GETs at the same time. The shape analysis may include the use of multiple shape analysis methods to identify missing and worn or broken GETs, carryback, volume, and payload analysis. Machine learning methods may also be used. For example, individual machine learning models may be trained for each of these subtypes of shape analysis or to identify shapes in the material point cloud datasets such as GETs or the load container.

[0052] As noted above the material classification, filtering and shape analysis may be performed as combined operations or in a different order to that described above. For example, in a combined shape analysis and material classification step, a reference shape such as a GET or load container could be fitted to the spectral point cloud datasets to determine the reference location of the GETs and load container. Then material classification could be performed on eachspatial datapoint, and material point cloud datasets generated for comparison with the previously generated reference shape and locations to enable identification of missing, broken or worn GETs, or for carryback and load volume estimation.

[0053] In optional step 206 an alert or electronic report is generated based on the shape analysis. The alert may be an output signal such as a GET status signal. These may be sent to a computer 290 used by the operator of the earthen working equipment 102 and / or to an operations center 294 for the mining operation. Depending upon the system configuration, an alert or GET status signal may be used to indicate that at least one GET is missing, or the GET status indicator may be an “all clear” indicator that all the GETs are present (none detected missing). If all the GETs are present, no output signal may be generated, or an all-clear status indicator may be generated. Additional information on the analysis may be included in an electronic report, such as the results of the shape analysis, or material point cloud datasets. The output signal may be sent to the operator’s computer system 290 located within the operator station or an operations control center. A user interface may be configured to generate a visual, audible, tactile, and / or other type of alert in a case where it is determined that a GET is lost or missing.

[0054] The method 201 may be varied based on implementation considerations or the operational requirements of a specific application. For example, the material classification step 203 and material filtering step 204 could be combined in a single joint step. Similarly, the material filtering step 204 and shape analysis 205 could be combined in a single joint step. In other examples the shape analysis could be performed prior to material filtering such that the material filtering is applied to analyzed shapes to then assist with determine missing GETs, worn or broken GET, carryback volume, load container volume, or payload. In some examples the material classification, filtering and shape analysis could be combined in a single step. In some examples, material classification 203 and / or shape analysis 205 steps may be performed using one or more trained machine learning models. For example, a first machine learning classifier may be trained for performing material classification and a second machine learning model trained for performing shape analysis. Additionally multiple machine learning models could be trained for the shape analysis, each trained to identify a different condition (or subtype) such as a lost GET, a worn or broken GET, a carryback volume, a volume of a load container, or a payload. Multiple machine learning models could also be trained for each condition and a consensus result used to assess a worn or lost GET. In some examples, a combination of machine learning model and analytical techniques may be used to perform the material classification and shape analysis steps.Spectral lidar Apparatus 210

[0055] Figure 3A is a schematic diagram of an example of a multi-spectral Lidar apparatus 210 that can be used in the spectral lidar monitoring system 200 to implement the monitoring method 201 shown in Figure 2. The spectral lidar apparatus 210 includes a spectralsource 220, a scene scanner 230, and spectral receiver 240. The spectral lidar apparatus 210 is configured to generate spatial data and spectral data for a field of view 236 and receive the reflected signals to be processed to generate a set of spectral point clouds, also referred to as a spectral scenes, for the field of view 236. The spectral source 220 includes a seed laser 222 which generates pulses of light which subjected to amplification by a fiber amplifier 224 and broadening by a fiber-optic spectral broadening component 226, i.e. , the spectral source 220 may be a super continuous laser source, but other configurations for a spectral source are possible. The seed laser 222 generates pulses of light over a band of emission wavelengths. In other examples, a filter or other optical arrangement could be used to select specific emission wavelengths, or multiple emission bands. As one example, the seed laser 222 may be a Nd:YAG laser which after spectral broadening emits lights in the range from 1 .45 pm to 2.55 pm. A band pass filter may also be used to limit emissions to a specific emission band 310. Other laser sources such as high power diode laser (HPDL) and emission bands may be used. A transmission fiber 228 carries the emitted pulses to a scene scanner 230.

[0056] The scene scanner 230 includes an optical assembly 231 to emit and direct the light pulse in a specific pointing direction 233 within the field of view 236 and to receive a return light signal 234 reflected from an object in the line of sight. The return light signal is supplied to the spectral receiver 240 via an optical fiber 238. The optical assembly 231 may include an electronically steerable mirror 232 or similar component, to direct the light at a specific pointing direction 233 with the conical field of view 236. The scene scanner 230 is configured to follow a scan pattern to direct a sequence of light pulses (beams) over the field of view 236 and to receive the return light signals 234 to build up a set of spectral data clouds including the spatial and spectral data at each of the set of detection wavelengths. The set of spectral point clouds form a spectral scene dataset of the field of view (which can also be referred to as a spectral scene or a measurement scene). The scan pattern may be a set of progressive linear scans, a spiral like pattern, or a pseudo random sampling approach in which the scene scanner directs pulses to different directions until all desired points are captured (for example to avoid potential systematic biases). Other scan patterns may also be used.

[0057] The optical assembly 231 also includes an output window configured to capture an internal reflection from the output window of the emitted pulse. A further receiver mirror, or the electronically steerable mirror 232 is used to capture and direct the return light signal to spectral receiver 240 via optical fiber 238. The spectral receiver 240 includes a diffraction grating 242 (or similar component) that disperses the light signal into multiple distinct wavelengths, with each wavelength directed at a different angle with respect to the diffraction grating. A set of the wavelengths of interest are selected from the emission wavelengths, which are also referred to as detection wavelengths, based on the reflectance curves shown in Figure 3B for materials of interest and for each detection wavelength a light detector 244 is used to make a measurement of the return light signal from which a reflectance of the target object can be estimated for eachdetection wavelength. We will refer to each light detector 244 as a spectral channel and has an associated detection wavelength and is located at the corresponding angle that the wavelength emerges from the diffraction grating at. The light detector may be a device such as photodiode which generates an output signal proportional to the intensity of the received light, or power meter which the amplitude or power of the returned signal 234. The output of the light detector is converted to an n-bit digital measurement by one or more Analog-To-Digital (ADC) converters 264 in the computing apparatus 250. The ADC 265 may be a multi-channel to enable the detection wavelengths to be measured and processed in parallel. The set of detection wavelengths along with the reflected light may be used to form a spectral signature of the return light signal at a spatial datapoint. Selection of the detection wavelengths may be determined using a known spectral profile of a target material, such as material the GETs are constructed from allowing identification (classification) of the GETs in the field of view.

[0058] In this example the spectral receiver 240 is a multichannel receiver where a first spectral channel 244a corresponds to a wavelength of 1.5pm used as a dedicated distance channel, and four spectral channels 244b, 244c, 244d, 244e spanning wavelengths from 1.9um to 2.5 pm (e.g., 1 .9, 2.1 , 2.3, 2.5 pm) used as detection channels of the return light signal for the detection wavelengths. In this example an 8-bit 4 (or more) channel ADC is used to quantize each intensity measurement to a 0 to 255 range. The detectors 244b-244e may also be AC coupled (high pass filtered) so that continuous background signals (such as from solar radiation) can be filtered out.

[0059] The detection wavelengths are selected based on the known reflectance curves of materials of interest. These materials of interest include one or more materials used to form or manufacture the GETs, materials used to form the load container (if different to those for the GETs) and other materials of interest such as various ore material that is being mined or moved. Figure 3B shows reflectance as a function of wavelength in the range of 0.2 pm to 4 pm for a range of materials including metals and alloys such as Carbon steel, and Aluminum, and earthen materials such as gravel and a combination of sand and gravel. Reflectance curves may be obtained for any material of interest and for mixtures of materials of interest (for example different earthen materials). As shown in Figure 3B, the different materials each have different shaped curves or spectral signatures and thus the detection wavelengths may be selected based on wavelengths where there are significant differences between different materials. The selection of the detection wavelengths may also take into account the frequency range the spectral source is able to emit at, or conversely the selection of the spectral source may be based on a set of desired detection wavelengths.

[0060] In this example, the seed laser 222, such as a high power diode laser (HPDL) or Nd:YAG laser emits light pulses which are amplified 224 and spectrally broadened 226 to have an emission wavelength band 310 from 1.450-2.550 pm (microns). Four detection wavelengths 312, 314, 316, and 318 are selected at 1.9, 2.1 , 2.3, 2.5 pm, with the distance channel using awavelength of 1.5 m. The values of the reflectance curve at each of the detection wavelengths 312, 314, 316 and 318 form a spectral signature for the relevant material. From Figure 3B it can be seen that in the emission wavelength band 310, carbon steel which may be used as a material to form the GETs and load container, has a reflectivity that rises with a constant slope from around 0.6 to 0.75 whereas gravel (and earthen material) has a generally lower reflectance curve which is initially flat before steadily declining at a wavelength of around 1.8 microns from a reflectance of around 0.5 down to 0.4 around 2.5 pm. Spectral signatures may be obtained from these spectral curves (set of reflectance values at the detection wavelengths 312, 314, 316, and 318) for each material of interest (e.g., material forming one or more of the GETs, load container, and earthen material) can be compared with the observed spectral data to classify the material at a spatial datapoint.

[0061] Spectral reflectance at a wavelength is the ratio of the intensity of the reflected signal to the intensity of the emitted signal. It is also noted that the spectral signature captures relative changes over the detection wavelengths, and thus absolute measurements of reflectance are not required (although they could optionally be obtained). In some examples, the measurements of the intensity of the return light signal at each of the detection wavelengths may be used as an estimate (or proxy) of the reflectance at the respective detection wavelength. This is because the same emitted signal is used for each detection wavelength and so the intensity of the emitted signal is effectively a common scaling factor applied to all intensity measurements. In one example, a calibration process may be used to determine a set of adjustment factors to be applied to the measured reference intensity to obtain an estimate of the reference intensity for each of the detection wavelengths. In another example, each of the intensity measurements are divided by the intensity measurement at the first detection wavelength. The reflectance signature is then defined as the set of ratios for each of the other detection wavelengths representing how the reflectance changes with wavelength (e.g., intensity measurements are normalized by the intensity of first detection wavelength). More complex spectral signatures could be generated based on various combinations, functions or curve fitting / regression models applied to the measured intensities at each of the detection wavelengths. In some examples a measurement of the intensity of the emitted signal at an emitter reference wavelength may also be measured to obtain an estimate of the reflectance at each of the detection wavelengths.

[0062] To determine the distance to an object in the field of view, a further wavelength from the diffraction grating may be used as a distance (or timing) channel to measure the time of flight of the emitted and return light signal. In Figure 3A, a first channel 244a corresponding to a wavelength of 1.5um is used as a dedicated distance channel, or in other examples, one or more of the detection channels may be used a distance channel (e.g., is configured to measure both a time of flight and an intensity of the reflected signal at the detection wavelength). The distance channel 244a uses a threshold detector to start and stop a Time-To-Digital converter (TDC) 262 in the computer apparatus 250. The reception of the internally reflected emitted signal is used tostart a counter in the TDC and the subsequent reception of the return light signal is used to stop the counter in the TDC. The stop count is converted to a time by multiplying by the clock period of the counter (e.g., the duration of each count), and the distance to the reflecting object is then determined by multiplying half of the time of flight by the speed of light in air. The distance to the object, or a measurement from which the distance may be estimated such as the counter value or time of flight, is stored in a memory 254 along with the pointing direction to allow estimation of the location of the reflecting object in the field of view when creating the spectral point clouds.

[0063] The pointing direction 233 is stored or processed to enable estimation of the spatial location of spatial datapoint on an object in the field of view that reflected the emitted signal, when combined with the distance estimate. The pointing direction may be obtained using encoders on the electronically steerable mirror of the optical assembly. The pointing direction may be stored as Euler angles, or azimuthal and polar angles based on a spherical coordinate system defined with respect to the scene scanner based on a reference axis (or pointing direction) 233. The position of the object may be specified in this spherical coordinate system, where the distance to the object forms the radial dimension. In other examples, the spatial datapoint may be a location in a cartesian coordinate system (x,y,z) in which the location of the scene scanner 230 is known relative to the origin of the coordinate system origin and the pointing directions of the scene scanner, and radial distance to the objects are transformed into an (x,y,z) coordinate. In Figure 1A, the x and y axes of the reference cartesian coordinate system define a ground plane, and the z axis represents the height above the ground. Additional information, such as known size and geometry of the parts of the mining equipment 102 may be used to transform pointing directions and range measurements to (x,y,z) coordinates in cartesian coordinate system.

[0064] The spectral lidar apparatus 210 is used to generate spatial and spectral data by repeatedly firing the seed laser 222 into the field of view 236 according to the scan pattern in order to generate the one or more spectral point clouds for the field of view. The set of spectral point clouds may be stored in a spectral scene container or data structure, along with other related data and meta data such as time of capture, lidar configuration parameters, a location, an identifier of the heavy equipment, etc. The triggering and collection of spatial and spectral data on a field of view may be repeated to generate multiple sets of spectral point cloud datasets (spectral scenes) for example at different times or stages of the same operational cycle, as well as, over the course of multiple operational cycles of the heavy equipment 102. For example, a spectral scene dataset could be generated for the mining shovel 102 in the engage substage as shown in Figure 1A, and another spectral scene dataset generated for the mining shovel 102 in the excavate substage shown in Figure 1 B. Similarly spectral scene datasets could be obtained at the same stage (or substage) in different operational cycles. As will be discussed below, during analysis spectral scene datasets obtained at different time points may be combined, for example to build up a comprehensive or robust material point cloud dataset for the shape analysis.

[0065] In some examples the spectral lidar apparatus 210 is controlled by a dedicated lidar controller 260 which is used to capture the digital time and intensity measurements for the detection wavelengths and to provide the measurements to the computing apparatus 250 for storage in memory 254 for subsequent conversion and formatting into a scene dataset, and subsequent processing.

[0066] It will be understood that Figure 3A shows one example of a spectral lidar system and in other examples the structure could be varied. For example, in Figure 3A the spectral receiver 240 includes four channels for the four detection wavelengths, and a multichannel ADC with four (or more) channels or four separate ADCs may be used. However, in other examples a different number of detection wavelengths (spectral channels) may be used, for example 3, 5, 6, 7, 8, 9, 10, 12, 14, 16, 24 or more. Similarly, the ADC could be an ADC card with 8, 16, 32 or more channel ADC, or multiple 4 or 8 channels ADC cards could be used. The number of bits used to process the signal may also be varied, for example the number of bits may be a 4 (16 levels), 8 (256 levels), 12 (4096 levels) or more. The choice of bits may be based on the required resolution to distinguish between different materials (e.g., range of reflectance values). In some examples, the spectral receiver 240 may be a hyperspectral receiver with a large number of channels such as 50, 64, 100, 128, 256, or more, each of which are processed. In some examples, all hyperspectral channels (wavelengths) may be used as detection wavelengths to build a high-resolution spectral signature of the object. In other examples, only a subset of the total number of available channels (wavelengths) of the hyperspectral receiver (e.g., a multispectral receiver) may be selected as detection channels and saved, and the data from the remaining channels is discarded. This approach may provide flexibility in different working environments and materials of interest, as the set of detection wavelengths could be varied based on the specific materials of interest in a new location (e.g., a different ore / earthen material or GETs).

[0067] In the above example, the intensity of the emitted signal is not measured and the measurements of the intensity of the detection wavelengths are used as estimates of the reflectance of the object at the respective detection wavelength or are adjusted based on calibration factors. However, in other examples, an intensity measurement may be performed of the internally reflected emitted signal at a reference wavelength and used to estimate the reflectance at each of the detection wavelengths. For example, an additional ADC, or one of the ADC channels in a multichannel ADC card, may be used to measure an intensity of internally reflected emitted signal at a reference wavelength such as 1.7 pm (emitter channel). In another example, the spectral receiver could be configured to measure the intensity of both the internally reflected emitted signal and the return light signal at each of the detection wavelengths provided the detector or ADCs are fast enough to capture and quantize the intensity of the emitted signal before the return light signal is received.

[0068] In another example, one of the spectral channels may be used as both a spectral channel and as a distance channel. The spectral channels may be used as a detection channel by either generating a signal to trigger the TDC 262 when the intensity passes a threshold level, or by splitting the output signal from the detector to the TDC 262 and the ADC 264. In another example, the distance channel could be incorporated into the scene scanner. In this example, the return light signal could be split into a first signal sent to the spectral receiver 240 over optical fiber 228 and a second signal sent to a threshold detector for starting and stopping a TDC 262.

[0069] In another example, the spectral source 220 could further include a filter, or set of filters, after the spectral broadener, which are configured to only pass the detection wavelengths so that the emitted light pulse only contains the detection wavelengths and a wavelength for the distance channel.

[0070] In the above examples, the spectral source 220 may emit a single pulse and the spectral receiver captures the detection wavelengths in parallel by using a diffraction grating to split the return light signal into parallel channels (each of different wavelengths). However, in other examples, the spectral source may emit a sequential series of pulses, where each pulse in the sequence is emitted at a detection wavelength (e.g., monochrome) and the spectral receiver includes a single channel detector that measures and digitizes the return light signal. In these examples a separate laser source may be used for each detection wavelength, and each seed laser 222 is triggered in sequence and provided to the transmission fiber 228 (and the spectral broadener 226 is omitted). In another example, the seed laser 222 may be a tunable laser which can be tuned to each of the detection wavelengths. In another example, a set of interchangeable filters, each configured to pass one of the detection wavelengths may be used after the spectral broadener 226. A different filter is then inserted into the optical path for each pulse in the sequence to generate the sequence of pulses at the detection wavelengths.Computer Apparatus

[0071] Figure 3C is a schematic diagram of the computer apparatus 250 of the spectral lidar monitoring system 200 of Figure 1A. The computing apparatus 250 may be a single or unitary computing apparatus including one or more processors 252 (remote or local) and one or more memories, which may include one or more storage memories 254 and program memory 256 which stores instructions for controlling the spectral lidar, storing spectral data, and processing the data. The computing apparatus also includes an input / output (I / O) interface 258.

[0072] The computing apparatus may also be a distributed computing apparatus including a separate lidar controller 260 which is operatively connected to the computer apparatus 250 via a communications interface 258. The lidar controller 260 may contain the TDC 262 and ADCs 264, and may contain additional processors 252, memories 256 and an I / O interface 258. In some examples the lidar controller 260 is located with the spectral lidar apparatus 210 withina housing 180 mounted 182 to the boom, whilst the computer apparatus 250 is located on the frame 120. In some examples, the spectral lidar controller 260 performs data pre-processing of the measurements and provides a spectral scene dataset to the computer apparatus 250 for storage in memory 254. In some examples, a data processing computer 294, is used to process the spectral scene data and is located remote from the mining equipment, for example, a server in a central control facility, or a cloud server and is connected to the computing apparatus 250 over a wireless network 292. The computer apparatus 250 may also communicate with an operator computer apparatus 290 to provide alerts or results of analysis of the spectral scene data.

[0073] The I / O interface 258 includes an interface for communicating information with other components of the monitoring system 200, such as with the spectral lidar apparatus 210, the operator’s computer system 290 located within the operator station (Fig. 1A) and / or data processing computer 294 located remote from the mining shovel 102 (such as at the control center of the open pit mine 110 or in the cloud). Although only a single spectral lidar apparatus 210, operator computer apparatus 290, and data processing computer 294 are shown in Figure 3C, the I / O interface 258 may allow the one or more processors 252 to communicate with more than one spectral lidar apparatus 210, more than operator computer apparatus 290, or more than one data processing computer 294. In some examples, the one or more processors 252 may communicate with the data processing computer 294 via a wireless network 292 and communicate with the spectral lidar apparatus 210 and the operator computer system 290 over a wired network. In other examples, the one or more processors 252 may also communicate with one or more of the spectral lidar apparatus 210 and the operator computer apparatus 290 over the wireless network 258. The I / O interface 258 may include any communication interface which enables the processor 258 to communicate with the external components described above, including specialized or standard I / O interface technologies such as channel, port-mapped, asynchronous for example.

[0074] The storage memory 254 stores information received or generated by the one or more processors 252, such as the spectral point clouds (e.g., spatial data and spectral data) obtained or generated from the spectral lidar apparatus. Each spatial datapoint is a three dimensional location of an object, or a spatial datapoint on an object, within the field of view. This is measured in some reference coordinate system, and as outlined above is determined using the pointing direction of an emitted pulse of light and the time of flight of the emitted and reflected pulse. The plurality of spatial datapoints form a three dimensional point cloud representation of the field of view, capturing the three dimensional locations of objects, or points on objects, from which a return signal was received. Each spatial datapoint has associated spectral data including an estimate of reflectance of the object at the respective spatial datapoint (location) at each of the detection wavelengths. The spectral data may be stored as a reflectance estimate for a detection wavelength or a measurement which may be used to determine a reflectance estimate,such as an intensity measurement of the reflected light pulses 234 at the respective wavelength for each spatial datapoint. The material class (from material classification step 204 for each spatial datapoint may also be stored in the storage memory 254. This may be stored with each datapoint as an associated class, or for each material a data structure could store the set of spatial data points classified as the respective material. Reference and / or calibration data to enable an estimate of the reflectance to be calculated may also be stored, for example, the intensity of the emitted pulse at a reference wavelength, which may be a detection wavelength, or another wavelength. The set of spectral point clouds for a field of view, along with associated metadata, reference data and calibration data can be formatted and stored in a variety of formats. For example, the set of spectral point clouds can be grouped into a spectral scene dataset and stored in a spectral scene dataset container or other abstract data class in the storage memory 254. Alternatively, the set of spectral point clouds scene data can be stored as a single set of spatial datapoints (one for each spatial datapoint in each of the point clouds), and a set of reflectance values at each of the detection wavelengths for each spatial datapoint (e.g., a set of spectral / reflectance values for each of the detection wavelengths. The data may be stored in a database or other file structure.

[0075] The program memory 256 stores various blocks of code (processor instructions), including codes for directing the processor 252 to execute various functions, including performing various steps or sub-steps of monitoring method 201 . For example, the generating step 202 may be implemented using code blocks (sets of computer implemented programmable instructions) for controlling the spectral lidar 266 and for performing spectral point cloud scene data generation 270. Steps 203, 204, 205 and 206 may be performed by material classification code block 272, material filtering code block 274, shape analysis code block 276, and alert and reporting code block 286. The shape analysis code block 276 may contain code blocks for performing loss detection 278, wear detection 280, carryback estimation 282, or volume estimation 284. Volume estimation codeblock 284 may also be configured to perform a payload analysis using the estimated volume. A range of machine learning and data processing techniques may be used to classify or identify the type of material at each spatial datapoint in the field of view, for example using known spectral signatures of specific materials of interest. For example, a material point cloud dataset for the material used to form the GETs may be compared with reference spatial data or material point cloud data on the known shape of the GETs to identify worn or lost GETs and enable the issuing of alerts and / or follow-up action such as repair or replacement. The material point cloud datasets can also be used to determine the presence of ore in a load container after a dumping operation to enable an estimation of the carryback volume in the load container. Similarly, a range of machine learning and data processing techniques may be used to perform the shape analysis. For example, by comparing an observed shape of particular material with the known shape of the GET or load container.

[0076] The reference spectral signature of each material of interest may be stored in storage memory 254 of the computer apparatus 250, so that it can be compared with the measured or observed spectral scene dataset to determine the material located at each spatial datapoint. The reference spectral signature of each material of interest may be found, for example, from calibration measurements or reference library data (See Fig. 3B). Similarly, a machine learning (ML) model could be trained using a training dataset including multiple spectral point clouds collected from a field of view including the one or more materials of interest (e.g., experimentally obtained data from the spectral lidar system) which the ML models use to learn or encode spectral signatures for each of the materials of interest. In some examples the training dataset is labelled spectral scene datasets. The training data may also be generated from or include synthetic or augmented training generated using reference spectral signatures. The trained ML model is then used to estimate or classify the material associated with a spatial datapoint by providing or inputting the spectral scene dataset to the machine learning model or Artificial Intelligence neural network. The trained machine learning model may then be stored in storage memory 254 or program memory 256 (and the reference spectral signature then do not need to be stored by the computer), but other configurations are possible.

[0077] The program memory 256 may also store database management system codes for managing data stores in the storage memory 254. In other examples, the program memory 256 may store additional or alternative blocks of code for directing the one or more processors 252 to execute additional or alternative functions. The program and storage memories 256 and 254 may each be implemented as one or a combination of a random-access memory, a hard disk drive, flash memory, or phase-change memory, or any other computer-readable and / or -writable memory.

[0078] The one or more processors 252 are configured to execute the program codes stored in the program memory 256 to implement monitoring methods, and to receive and transmit information, including data and commands, with the spectral lidar apparatus 210, operator computer system 290, and the data processing computer 294 via the I / O interface 258. In the example shown, the one or more processors 252 may be optimized to perform image processing functions and may include one or more dedicated graphics processing unit (GPU) for accelerating image processing functions and machine learning models. In other examples, the computer apparatus 250 may be partly or fully implemented using different hardware logic, which may include discrete logic circuits and / or an application specific integrated circuit and may use a combination of hardware and software. In some examples, a distributed computing approach may be used in which the one or more processors 252 and program memory 256 may be located within the frame 120 of the mining shovel 102, whilst the storage memories 254 is located remote from the mining shovel 102. In such examples, the processor 252 may communicate with the program and storage memories 254 via the wireless network 292, forming a cloud-based processor circuit. Similarly, the program memory 256 may also be distributed between one ormore memories located with one or more processors 254 on the mining shovel, and one or more memories located in the data processing computer 294, and the monitoring may be performed in a distributed network or remote from local processing.

[0079] In operation the computer apparatus 250 is configured to implement the monitoring method as illustrated in Figure 2 for example to detect one or more of a missing GET, a worn or broken GET, a carryback volume, a load container volume, or a payload. These methods will now be explained in more detail.

[0080] Referring to Figures 4A-4B, the operating implement 126 is illustrated to include a plurality of ground engaging tools (GET), including teeth 150 and ground engaging lip shrouds 152. In other examples and depending on the function of the heavy equipment and the type of earthen material to be excavated from the mine face 114 or the earthen material pile, the operating implement 126 may include additional, fewer, or alternative GET components, and the load container may come in a range of shapes and geometries (e.g., buckets, shovels, cutterheads, rippers, crushers, etc.). In this example the load container 154 includes a lip 160, a rear wall 140, and an openable bottom wall 142 pivotably mounted to the rear wall 140 and operable to be opened and closed by the door control mechanism 144. The teeth 150 and the shrouds 152 are coupled to a leading edge 161 of the lip 160. In the example shown, each tooth of the teeth 150 includes an intermediate adapter 164 and a tip 165. The intermediate adapters 164, the tips 165 and the shrouds 152 are wearable parts of the operating implement 126 that may be replaced. During an excavate stage of the mining shovel 102, the teeth 150 and shrouds 152 engage the mine face 114 and capture earthen material into the load container 154 as a load 156. The teeth 150 and shrouds 152 protect the leading edge 161 of the lip 160 from wear due to contact with mine face 114. The load container 154 supports the load 156 until the openable bottom wall 142 pivotably opens relative to the rear wall of the load container 154 to dump the load 156 during a dumping stage.

[0081] In operation, the operator within the operator station 121 of the frame 120 will cause the mining shovel 102 to go through a series of activities which make up an operating cycle of the mining shovel 102. The operating cycle may include at least the following stages: (1) excavate stage, (2) swing full stage, (3) dump stage, (4) swing empty stage, and (5) idle stage. During the excavate stage, the operator controls the mining shovel 102 to extend the GET 150 of the operating implement 126 into the mine face 114 in order to excavate the earthen material from the mine face 114 and into the load container 154. During the swing full stage, the operator may control the mining shovel 102 to rotate the load container 154 laterally away from the mine face 114 and towards a desired dump location. During the dump stage the load container 154 is positioned above the desired dump location and the operator controls the mining shovel 102 to unload the load 156 from the load container 154 to the desired dump location. During the swing empty stage, the operator controls the mining shovel 102 to rotate the now empty load container 154 towards another position proximate the mine face 114 to start another operational cycleextract the earthen material therefrom in a subsequent excavate stage. The series of activities may be repeated for multiple operating cycles of the mining shovel 102. In other examples, the operating cycle of the mining shovel 102 may include additional, fewer or alternative stages. For example, an idle stage may be included between the stages of the operating cycle, or between a plurality of operating cycles, in which the operator causes the mining shovel 102 to idle (e.g., be still) to allow personnel or equipment to move into desired positions (e.g., a dump truck under the desired dump location) or for other reasons. For example, in certain situations, the operating cycle may include a bench cleaning sub-cycle which includes only (1) the excavate stage and (2) the dump stage.

[0082] Referring to Figure 1A, during the excavate stage, the operator will control the load container 154 to move along an arc to extend the GETs into the mine face to excavate earthen material from the mine face 114. The handle 125 transitions from the lowered configuration shown in Figure 1A to a substantially horizontal configuration shown in Figure 1 B (with a large angle relative to the z-axis and a small angle relative to the x-axis). The spectral lidar apparatus 210, is mounted on mining shovel 102 such that the field of view 236 captures the part of the load container (e.g., a bucket) including the GETs during the excavate stage.

[0083] Referring to Figure 5A, the GETs, including teeth 150 and shrouds 152 of the front includes a portion of the load container 154 are visible against a background of the mine floor or mine bench 114. Some earthen material 190 or a large boulder is located between the GETs, along with some carryback earthen material 191 within the load container after a previous dumping operation. Figure 5B is a close up of portion of Figure 5A showing two adjacent teeth 150 and the earthen material 190 that is wedged between the two teeth and partially obscures the edges of the teeth. As some of the earthen material is loose material, some datapoints along the edges of the teeth are still visible amongst the earthen material (e.g., are not obscured). Such datapoints are difficult to identify for the material wedged between the teeth in Lidar-only based systems, but as will be outlined below, the problematic situation can be detected using the inventive spectral lidar system discussed herein based on the observed spectral signature at the respective points. Additionally, Figure 5A shows a worn tooth 150’ with a slightly shorter tip. As will be described below such wear can be detected based on use of a spectral signature to identify the precise location of GET material to identify or determine the edge of the worn GET 150’. As the earthen material between teeth obscures the adjacent teeth, precise identification of teeth edges needed for wear detection is extremely difficulty in Lidar systems. In this example, the spectral lidar apparatus 210 is triggered to emit light pulses of light to scan the field of view 236 by lidar controller code block 266. The return signals are analyzed by the computing apparatus 250 as discussed above to generate spatial and spectral data by scene data generator code block 270 (step 202). Figure 5C shows a first spectral point cloud (3 dimensional image) obtained at a first detection wavelength 312. Each spatial datapoint in the spectral point cloud has an associated reflectance measurement at the first detection wavelength. A first spatial datapoint155 (x1 , y1 , z1) corresponding to a GET material and second spatial datapoint 195 (x2, y2, z2) corresponding to an earthen material are indicated in Figure 50. Figure 5D shows the combined spatial and spectral data as a set of four spectral point clouds for each of the four detection wavelengths 312, 314, 316, and 318 (see Figure 3B). The set of spectral point clouds form the spectral scene for the field of view. Each spatial datapoint includes the estimate of the reflectance at the respective detection wavelength of a spatial datapoint on an object that reflected the light pulse. As described above this spectral scene could equivalently be stored or represented as a single spatial point cloud where each spatial datapoint has an associated set of reflectance values for each of the detection wavelengths. Figures 5E and 5F are respective plots of the spectral signatures, which is reflectance as a function of wavelength, for the first spatial datapoint 155 (x1 , y1 , z1) and the second spatial datapoint 195 (x2, y2, z2). The observed spectral signatures have different curve shapes indicating different materials at the two spatial datapoints 155 and 195, namely steel used to form the GETs, and earthen material.

[0084] Material classification (step 203 discussed above) is performed by material classification code block 272 on the set of spectral point clouds (spectral scene) for the field of view to classify (or identify) whether at least one material of interest is located at the respective spatial datapoint. In one example, the materials of interest include at least load container material, and may be a GET material to enable identification of the GETs. GETs may include multiple materials, and thus the materials of interest may include each of these materials or at least one of these materials of interest. The material of interest may include at least one load container material to determine carryback, ore in the load container, or payload. In another example or as a part of this example, the at least one material of interest may include the ore being excavated. In another example, the materials of interest include earthen material worked at the mine site. In this example, this may be used to assist in identifying earthen material on the GETs which may obscure the underlying GET material making determination of missing, broken, or worn GETs more challenging or allow an estimate of the amount of carryback, ore in the load container, or payload. Material classification is based on using spectral scene to identify what material generated the reflected signals based on reference spectral signature or curve. The method 201 takes the measured reflectance for each detection wavelength at each spatial datapoint (e.g., 155 (x1 , y1 , z1)) on to plot a curve (see Figures 5E). In the illustrated example, at a spatial datapoint 155 (x1 , y1 , z1), the measured reflectance at each detection wavelength increases approximately linearly as a function of wavelength. This observed reflectance data can be compared with the spectral signatures of the reference reflectance curves shown in Figure 3B for each material of interest. In this example, the GET and load container is manufactured from Carbon steel, and the earthen material is gravel and the measured reflectance’s shown in Figure 5E are more similar to the curve for Carbon steel or Aluminum (AL) than gravel (earthen material) from wavelengths 314 to 316 (Figure 3B). The spatial datapoint would be classified as Carbon steel (GET material). Figure 5F shows the reflectance curve for spatial datapoint 195 (x2, y2, z2).In this example the observed reflectance at the first two detection wavelengths is flat and then begin declining which is more similar to the reflectance curve for gravel at the respective wavelengths 314-316 shown in Figure 3B than sand and gravel or Carbon steel and so this spatial datapoint would be classified as earthen material or more precisely as gravel. The classification step thus exploits the differences in reflectance of different materials at different wavelengths so that materials of interest can be identified in the spectral scene, enabling robust spatial datapoint by spatial datapoint material classification. Sensitivity can be increased by selecting detection wavelengths corresponding to significant differences between materials of interest, for example, to enable a binary choice between steel and earthen material.

[0085] Classification may be performed using a range of analytical classification techniques which compare the measured reflectance values with a set of reference values for a material of interest (e.g., from Fig 3B). For example, curve fitting, such as third or fifth order polynomial could be performed on the observed spectral data (e.g., to each plot shown in Figure 5E) for each spatial datapoint to obtain a parametric representation of the spectral data for the spatial datapoint. The curve could then be compared with a reference curve, or a parametric representation, obtained for a material of interest (e.g., Fig 3B). A material could then be classified based on a similarity measure, a statistical or a regression model, a correlation measure, or a counting based method. For example, the reference data could be used to generate a range of acceptable values for each detection wavelength (this could be specified as a range of values, or a mean and a variance). The observed reflectance values at each detection wavelength could then be compared with the range of acceptable values, and a count of how many detection wavelengths are within the range of acceptable values used. Classification may require a threshold number such as 70%, 90% or 100% of detection wavelengths to be within the range of acceptable values. Pre-processing of the data may also be performed, for example to normalize or scale the data to a common range (e.g., [0,1]).

[0086] In other examples, classification may be performed using a trained machine learning model that receives a set of spectral point clouds and classifies the material at each spatial datapoint. A range of machine learning classifiers may be trained as outlined in Figures 7A and 7B as discussed below. The training dataset includes multiple estimates of spectral signatures for each of the one or more materials of interest and / or multiple spectral point clouds collected from a field of view including the one or more materials of interest. For example, the spectral lidar system could be used to first generate training data from capturing scenes containing GETs, load containers and earthen material of interest. Once the system is operational, operational data could also be used to retrain and improve the classifier. In some examples the training data includes multiple sets of spectral point cloud datasets (spectral scenes) where each spatial datapoint is labelled with the type of material. The training data may also be generated from or include synthetic or augmented training generated using reference spectral signatures. For example, a scene could be simulated based on known spectralsignatures for the materials of interest and CAD models of GETs and load containers. The spectral signature is the intensity measurement or estimate of reflectance at each detection wavelength at a spatial datapoint. The labelled spectral scenes act as reference data to train the model. The training data may include multiple examples of materials of interest to train a multiclass classifier (for example GET material, earthen material, other material). Labelled data allows the classifier to learn the reference spectral signature directly from the data without the need for a specific reference curve or signature to be known or stored for the material. In some examples, the training data may be unlabeled, but include each of the materials of interest, and during training the machine learning algorithm learns the spectral signature (the class) of each of the materials of interest from the training data.

[0087] Material filtering (see step 204 above) is then performed by material filtering code block 274 to generate a material point cloud datasets for each of the one or more material datasets of interest. Material filtering allows identification of the three dimensional shape of the GETs, load container, and earthen material in or on the load container, and a shape analysis as described in step 205 above may be performed by shape analysis code block 276 to perform one or more of loss detection, wear detection, carryback estimation volume estimation, and payload analysis. Each spatial datapoint has an associate material class, and thus a material point cloud dataset can be generated for each material of interest by filtering the spatial datapoints to exclude any spatial datapoints that are not classified as the material of interest. This is illustrated in Figures 5G, which shows a representation of the material point cloud dataset for a steel material. In this example, the same steel material is used for both the GETs and the load container, and the surrounding earthen material 114 has been filtered out to only leave the material or materials of interest that makeup the GETs and the load container 154. Parts of the teeth 150 adjacent to the location of the earthen material 190 are missing as they are obscured by the presence of the earthen material. However, in this case and as shown in Fig. 5G and 6B, a few spatial datapoints 157 of the tooth 150 are still visible between gaps in the earthen material. As outlined above, a reference shape for the GET is obtained, for example from a CAD model (Figure 4B) or previously collected GET material point cloud datasets and loss detection code block 278 compares the locations of the observed spatial datapoints in the material point cloud dataset to the expected locations from the reference shape. Scaling and normalization may be applied to the reference shape, or to the material point cloud dataset. The detection of single spatial datapoints corresponding to GET material amongst earthen material can be used in the shape analysis to recognize a GET shape and thus determine whether a GET is missing or worn. In some examples, curve or surface fitting may be performed on the observed spatial datapoints for comparison with the reference surfaces of the GET.

[0088] Shape analysis may also be used to identify worn GETs, such as GET 150’ in Figure 5B which has a worn tip and a shorter length than expected. Material classification allows precise identification of the edge of a material and wear detection code block 280 can then identifyand / or quantify the amount of wear. For example, the measured location of the exterior edge or surface of a GET could be compared with the expected location obtained from a CAD model or from a previous measurement of the same GET (e.g., previously generated material point cloud dataset). The distance between the two could be assessed to determine how much a GET has worn or the distance could be compared with a threshold amount to enable a binary classification of a worn or not worn status.

[0089] These approaches only require the positive identification of a few points on the GET to allow fitting of a curve, e.g., representing an edge, or a surface, and thus can robustly and reliably interpolate over points on the GET obscured by earthen material. This enables more robust edge detection for wear detection and / or presence detection than Lidar and camera based systems. For example in a lidar-only based assessment the obscured teeth would appear as a large blob of material where the earthen material 190 was located making it difficult to identify presence of the teeth and / or measure wear accurately for the affected teeth amongst the earthen material as the blob may represent a larger boulder obscuring the presence of the teeth thus making a lengthwise measurement of wear difficult, especially if the earthen material is a thin layer with an uneven surface (e.g. height variation in the top surface).

[0090] In some examples the GETs 150 and load container 154 may be formed of different materials for different parts such as different steels, metals, and alloys for the bottom wall 142, lip 160, shrouds 163, intermediate adapter 164 and tip 165. In these examples material point cloud datasets could be formed for each component material and then combined to generate a combined material point cloud for a component or part of interest (e.g., GETs, or GETs and the load container). In some examples, material filtering may take into account spatial depth information (z coordinate) in addition to spectral data. The depth information may assist in recognizing the difference between earthen material in the background (e.g., on the ground) and earthen material on or in the load container or steel associated with the load container and steel associated with another part of the mining shovel or mining site. This is illustrated in Figure 5H which shows a representation of the earthen material point cloud dataset after application of a depth filter to identify the blob 190 partially obscuring some of the GETs and carryback material 191 within the load container (e.g., the background earthen material 114 has been filtered out). In some examples material filtering may take into account shape information, such as the perimeter of the load container 154 obtained from the material point cloud dataset for the material or materials used to construct the load container. In this case earthen material could be filtered to earthen material within the exterior perimeter of the load container for assessment of carryback , load volume, or payload. In some examples separate earthen material point cloud dataset could be produced for background earthen material and foreground earthen material (e.g., in or on the load container 154). These could be used to assist in determining a missing, broken, or worn GET compared to an obscured GET.

[0091] The material point cloud datasets may also be used to perform a carryback analysis. Carryback material 191 is shown in Figure 5C which is earthen material 192 left in the load container after a previous dump operation. Carryback material may also be located on the lip 160 and GETs (e.g., blob 190). Carryback code block 282 identifies the carryback material and then estimates the volume of carryback material. The GET material point cloud dataset is used to identify or anchor the location of the reference spatial model of the load container 154, and then the depths or surfaces of the reference spatial model can then be used as depth thresholds applied to the earthen material dataset. The depth (location) of the upper surface of the GETs and load container in the reference spatial model can then be used as depth thresholds applied to the earthen material dataset in order to identify earthen material that is lying in the load container or on the load container, as well as to determine a reference surface for volume estimation as described above.

[0092] The material point cloud datasets may also be used to perform a volume estimation using volume estimation code block 284. Figure 6A shows a top down view of the load container in the excavate stage shown in Figure 1 B, in which the load container 154 contains earthen material 192. As previously described, the spatial and spectral data is obtained to generate the scene dataset. Figure 6B is an illustrative plot of the spatial datapoints at the first detection wavelength 312. Similar plots may be generated for the other detection wavelengths to generate a spectral scene similar to that shown in Figure 5C. Material classification is performed at each spatial datapoint using the reflectance values at each detection wavelength to classify the material. Filtering is then performed to obtain a GET material point cloud dataset shown in Figure 6C and an earthen material point cloud dataset show in Figure 6D. In this example, the load container material and the GET material are the same material. The earthen material point cloud dataset shown in Figure 6D may be obtained by first identifying earthen materials and then using a depth or spatial filter to restrict the spatial datapoints to those in or on the load container and filter out any background earthen material. This may be performed using the GET material point cloud dataset shown in Figure 6C, for example to identify the height of the load container or the edges of the load container 154. A volume can then be determined using the location and area of the lower surface and upper surface as described above. The payload can also be determined using the volume estimate. The spectral information is used to identify the specific material at each spatial datapoint point in the load container, and associated density is obtained, for example from a database of reference densities stored in a memory. The relative amount of each material can be estimated along with the mass component of each material using the associated density and the volume.

[0093] Additionally in this example, earthen material 193 almost completely obscures a GET, and the worn GET 150’ has now broken off during the excavation which may be detected by loss detection code block 278 as outlined above. The shape analysis code block 276 thus performs both loss detection and volume estimation on the same material point cloud datasets.In the case of the earthen material 193, the tooth is almost completely obscured. However, provided as little as a single point, or a few points 157 of GET material are visible where the GET is expected to be, then a positive determination that the GET is present (not missing) can be made. In contrast camera systems struggle in such cases when the GET covered by material as they are unable to reliably identify the material at each pixel of an image whereas the present method only needs to detect as little as a single spatial datapoint in an expected location. In some examples, the earthen material point cloud dataset may be used to support a decision that the GET is present. For example, a GET may support earthen material, and thus the presence of earthen material at a height within a threshold range above an expected height of the GET surface based on the reference model may indicate the GET is present because if the GET was missing the earthen material should be observed at the background depth.

[0094] In some examples, material classification 203 and / or shape analysis 205 steps may be performed using one or more trained machine learning models. For example, a first machine learning classifier may be trained for performing material classification and a second machine learning model trained for performing shape analysis. Additionally multiple machine learning models could be trained for the shape analysis, each trained to identify a different condition (or subtype) such as a lost GET, a worn or broken GET, a carryback volume, or a volume of a load container. Multiple machine learning models could also be trained for each condition and a consensus result used to assess a worn or lost GET. In some examples, a combination of machine learning model and analytical techniques may be used to perform the material classification and shape analysis steps.

[0095] Figure 7A is a schematic diagram of a machine learning (ML) training method 700 according to an example and Figure 7B is a schematic diagram of the architecture of a neural network model which may be trained using the machine learning training method shown in Figure 7A. There are five main processes involved in the ML training method 700. First input data 702 undergoes pre-processing 704 which may include numerical procedures such as data normalization, data mapping, data augmentation, data serialization, and / or feature detection to generate a pre-processed (or normalized) dataset 706. Normalization may include scaling the data to ensure that all data is over the same range (e.g., 0 to 1 , or 0 to 255). Mapping may include mapping a spatial dataset to a predefined standard size such as 512x512x512 voxels and may include addition of padding voxels to fill missing voxel values. Data augmentation may include generation of artificial training data based on the input data, for example by scaling, offsetting and / or modifying the spatial datapoints locations and reflectance values. Data serialization may convert an array or point cloud dataset into a linear vector. Feature detection may include determining one or more features of the data, including average values, variances, as well as computer vision based feature detection such as shapes. A training / testing split 708 in which the pre-processed dataset 706 is then split into a training dataset 710 and a test dataset 712. In this example, the training dataset includes 70% of the pre-processed dataset 706 and the test dataset224 includes the remaining 30%, however other training:test splits may be used, such as 80:20 split.

[0096] In the case of a material classifier, the training data may include multiple sets of spectral point cloud datasets (spectral scenes) where each spatial datapoint is labelled with the type of material present or spectral signatures obtained from such data. The labelled training data acts as reference data to train the model. Multiple examples are used which include multiple examples of materials of interest to train a multiclass classifier (for example GET material, earthen material, other material). Unlabeled training data could also be used and during training the machine learning algorithm learns the spectral signature (the class) of each of the materials of interest. In the case of a machine learning shape analysis model, the training dataset includes multiple estimates of the shapes such as GETs and load containers. These may be generated from CAD data or a point cloud datasets of the respective object obtained using lidar system, including the spectral lidar system described herein, or 3D camera or similar system. The point cloud datasets may be obtained by capturing scene data and filtering out points that are not part of the target object. In some cases, the machine learning shape analysis model could also be trained to recognize one or more shapes or objects and perform a volume estimation. For example, spectral scenes containing a load container containing an earthen material on or in the load container could be labelled to indicate what object each spatial datapoint corresponds to (e.g., GET, load container, earthen material), along with the volume of earthen material on or in the load container. The reference shape of the load container could also be provided. The machine learning shape analysis model then trained to directly estimate the volume earthen material on or in the load container.

[0097] One or more machine learning training epochs are then performed. In each epoch, model fitting 714 is performed of an initial (or input) machine learning model 716 using the training data 710 to generate a trained machine learning model 718. The test set is then provided to the trained machine learning model 718 to evaluate 720 the performance of the trained machine learning model 718. Depending upon the choice of the machine learning model (e.g., specific algorithm) model fitting may be performed using a single pass through the training data or in an iterative fashion with multiple passes through the training data (multiple training epochs). In such cases hyperparameter adjustment 722 is performed to adjust the ML model and generate an adjusted ML model 716 for the next training epoch. In the next epoch, the model fitting 714 processes trains the adjusted ML model 716 on the training data 710 to generate the trained ML model 718 which is then evaluated 720. This ML training process is repeated for a number of epochs until a final trained ML model 718’ is obtained. Stopping of the training process may be based on one or more of an accuracy criterion, a predetermined number of training epochs, or the improvement or difference compared to one or more previous epochs. Some ML models only require a single training epoch, in which case, hyperparameter adjustment is not performed. Typically, deep learning and neural network based models are trained over many epochs (e.g.,1O’s to 100’s of epochs). As additional data becomes available, ML models may be retrained on a new or expanded dataset (e.g., including the additional data). In some examples, a third blind validation dataset may also be split from the pre-processed dataset 706 in data split 708. The training:test:validate split may be 60:30:10 or 70:20:10. The performance of the final trained ML model 718’ is then evaluated 720 against this validation dataset. This is also referred to as a blind dataset as unlike the test dataset, the data is completely withheld from the training process and is only used to validate the final model 718’ (e.g., determine accuracy on a completely unseen dataset).

[0098] Figure 7B is an architectural diagram 730 of a deep neural network with one flatten layer 732, multiple dense layers 734 and an output layer 736 according to an example. In this example, the neural network model architecture included one flatten layer (FL) at the beginning, multiple hidden dense layers (HDL) in the middle, and an output layer at the end. In figure 7B nk indicates the number of neurons in the kth HDL. In each neuron, the input quantities are first aggregated through summation and then an activation function (fA) is applied, such as rectified linear unit (ReLU). The output layer 736 then processes the last layer of the hidden dense layers to generate an output, such as an object detection or classification. The object detection may be a boundary such as boundary box, a location in the image, and probability the object is within the boundary. Hyperparameter adjustment is typically performed based on a loss function, an optimization function and a learning rate and may include adjusted model and training parameters such as the learning rate, the number of neurons within a dense layer, number of dense layers, activation function, number of epochs, etc.

[0099] Training uses the training data to adjust and configure the model architecture, such as by determining the appropriate number, configuration and weights between neurons, so as to enable the model to robustly classify a material or determine a match to a reference shape. The training process thus allows the model to learn wear patterns or a complex geometrical shape of a GET or load container. Whilst it may be difficult to identify exactly how a model makes a decision, the effectiveness of the training can be assessed using a metric such as accuracy from applying the model to a test and / or validation set.

[0100] A range of machine learning algorithms may be used for material classification and shape analysis could be used including K-nearest neighbor (KNN), random forest, decision trees, neural networks (including convolutional neural networks (CNN)), support vector machine, Naive Bayes, and logistic regression algorithms. For example, a random forest classifier could be used for material classification, whilst a CNN model could be trained to perform shape analysis. Training of ML models may be performed using software libraries and packages such as Keras, TensorFlow, NumPy, Sklearn python library, and Pandas, and analysis software may be written in computer languages such as Python, R, C++, etc. In addition to training of machine learning models, these libraries and computer languages may also be used to performmathematical computations, provide data visualization, manage databases, generate reports and user interfaces.

[0101] In some examples scaling and normalization may be performed on the sets of spectral point clouds, or material point cloud datasets. This may be performed on spatial data and / or reflectance data to ensure all measured data and reference data is on the same scale.

[0102] The physical scaling issue is illustrated in Figures 8A and 8B. Figure 8A shows a side view of the end portion of the mining shovel 102 showing the location of the spectral lidar apparatus 201 , and the geometrical orientation of the handle 125 and load container 154 in the positions corresponding to Figures 1A and 1 B. In position P1, corresponding to Figure 1A, the load container 154 is lowered and the tip of the GETs is a distance D1 from the scene scanner of the spectral lidar apparatus. In position P2, corresponding to Figure 1 B, the load container 154 is raised and the tip of the GETs is a distance D2 from the scene scanner of the spectral lidar apparatus (where D2 is less than D1). As illustrated in Figure 8A, the spectral lidar apparatus 210 fires laser pulses into the field of view according to a scan pattern. The pointing direction of each laser pulse subtends a fixed solid angle. As can be seen in Figure 8A as the laser pulse travels further from the scene scanner (e.g., increasing depth) the physical distance between adjacent light pulses increases. The distance (arc lengths) between adjacent laser scans at depths D1 and D2 are shown in Figure 8A by arcs R1 and R2. As can be seen in Figure 8A, the arc length between adjacent scans along arc R1 is greater than the corresponding arc length along arc R2. Thus, as the size of the load container is fixed, this means that the number of scan points required to capture the entire load container will decrease with depth. This is further shown in Figure 8B, which is an orthogonal view of figure 8A to show the view along the tip of the load container (e.g., into the page of Figure 8A). A set of adjacent laser pulses are shown in Figure 8B along with the tip of the load container at positions P1 and P2, along with arcs R1 and R2. In Figure 8B it can be seen that when the load container is more distant from the spectral lidar apparatus 210 (position P1) the tip of the load container spans fewer scan lines of the laser.

[0103] Thus, to account for the variation in apparent size of the load container due to the depth of the load container in the field of view, the spectral point cloud dataset may be scaled to a common reference scale or size, for example the size as at a predetermined depth in the field of view. This is illustrated in Figure 8C which shows a scale transformation of a point cloud dataset. In Figure 8C the size 296 of the load container material point cloud dataset when the load container is in position P1. This is then scaled to a reference size 297 corresponding to the size of the load container at position P2. The reference spatial model of the load container may also be scaled to the size at the reference depth (D2) size. Scaling of the spectral point cloud dataset, and / or material point cloud cases may be performed prior to shape analysis. This ensures assessment of shapes and wear are performed on the same scales. In examples where trained machine learning models are used, this assists in ensuring the input data is always of the same scale as any training data used to train the machine learning model.

[0104] Scaling and normalization may also be performed on the reflectance estimates or intensity measurements, so that comparisons between measured values and the reference spectral values are on the same scale.

[0105] The monitoring method may be configured to analyze single sets of spectral point cloud datasets, for example captured at different time points during the operational cycle. The monitoring method may be used to generate alerts, for example, on detection of a missing or worn or broken GET so mining operations can be stopped in the case of a potential safety hazard. In other examples, the monitoring method may combine multiple sets of spectral point cloud datasets, or multiple sets of material point cloud datasets, collected at different time points. This provides a method to compensate for missing or obscured spatial datapoints in any one set of spectral point cloud images (or material point cloud datasets). For example, a GET 150 may be transiently obscured by earthen material at one spatial datapoint in time, but as mining operations are continued the obscuring material may be dislodged revealing the previously hidden datapoint on the GET. Combining spatial point cloud datasets or material point cloud datasets collected at different times may thus allow positive identification of the GET. Various mathematical and statistical methods may be used to combine datapoints, and in particular nearby or overlapping datapoints. For example, datapoints within a threshold distance of each other could be replaced with a single datapoint with an average intensity or spectral value. Additionally, data cleaning may be performed to identify redundant data points, outlier or inconsistent data points, which can be removed or ignored so that they do not affect the shape analysis.

[0106] The capture and use of spectral and spatial data by a spectral lidar apparatus allows material classification to be performed on individual spatial datapoints. This in turn enables more accurate, precise and robust estimation of GET edges and shapes, as well as carryback volumes, load container volumes and payload. Thus, examples of the methods, apparatus and systems described herein provide significant advantages over lidar and camera based systems.

[0107] Those of skill in the art would understand that information and signals may be represented using any of a variety of technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips may be referenced throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0108] Those of skill in the art would further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the examples disclosed herein may be implemented as electronic hardware, computer software or instructions, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on theoverall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

[0109] The steps of a method or algorithm described in connection with the examples disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. For a hardware implementation, processing may be implemented within one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, micro-controllers, microprocessors, other electronic units designed to perform the functions described herein, or a combination thereof. Software modules, also known as computer programs, computer codes, or instructions, may contain a number of source code or object code segments or instructions, and may reside in any computer readable medium such as a RAM memory, flash memory, ROM memory, EPROM memory, registers, hard disk, a removable disk, a CD-ROM, a DVD-ROM, a Blu-ray disc, or any other form of computer readable medium. In some aspects, the computer- readable media may include non-transitory computer-readable media (e.g., tangible media). In addition, for other aspects computer-readable media may include transitory computer- readable media (e.g., a signal). Combinations of the above should also be included within the scope of computer-readable media. In another aspect, the computer readable medium may be integral to the processor. The processor and the computer readable medium may reside in an ASIC or related device. The software codes may be stored in a memory unit and the processor may be configured to execute them. The memory unit may be implemented within the processor or external to the processor, in which case it can be communicatively coupled to the processor via various means as is known in the art.

[0110] Further, it should be appreciated that modules and / or other appropriate means for performing the methods and techniques described herein can be downloaded and / or otherwise obtained by computing device. For example, such a device can be coupled to a server to facilitate the transfer of means for performing the methods described herein. Alternatively, various methods described herein can be provided via storage means (e.g., RAM, ROM, a physical storage medium such as a compact disc (CD) or floppy disk, etc.), such that a computing device can obtain the various methods upon coupling or providing the storage means to the device. Moreover, any other suitable technique for providing the methods and techniques described herein to a device can be utilized.

[0111] In one form, the disclosure may include a computer program product for performing the method or operations presented herein. For example, such a computer program product may include a computer (or processor) readable medium having instructions stored (and / or encoded) thereon, the instructions being executable by one or more processors toperform the operations described herein. For certain aspects, the computer program product may include packaging material.

[0112] The methods disclosed herein include one or more steps or actions for achieving the described method. The method steps and / or actions may be interchanged with one another without departing from the scope of the claims. In other words, unless a specific order of steps or actions is specified, the order and / or use of specific steps and / or actions may be modified without departing from the scope of the claims.

[0113] As used herein, the terms “determining”, “obtaining” and “estimating” encompass a wide variety of actions. For example, “determining”, “obtaining” and “estimating” may include calculating, computing, processing, deriving, investigating, looking up (e.g., looking up in a table, a database or another data structure), ascertaining and the like. Also, “determining”, and “obtaining” may include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory) and the like. Also, “determining”, “obtaining” and “estimating” may include resolving, selecting, choosing, establishing and the like.

[0114] The reference to any prior art in this specification is not, and should not be taken as, an acknowledgement or any form of suggestion that such prior art forms part of the common general knowledge. It will be understood that the terms “comprise” and “include” and any of their derivatives (e.g., comprises, comprising, includes, including) as used in this specification, and the claims that follow, is to be taken to be inclusive of features to which the term refers, and is not meant to exclude the presence of any additional features unless otherwise stated or implied.

[0115] In some cases, a single example may, for succinctness and / or to assist in understanding the scope of the disclosure, combine multiple features. It is to be understood that in such a case, these multiple features may be provided separately (in separate examples), or in any other suitable combination. Alternatively, where separate features are described in separate examples, these separate features may be combined into a single example unless otherwise stated or implied. This also applies to the claims which can be recombined in any combination. That is a claim may be amended to include a feature, or part of a feature, defined in any other claim or any other part of the specification. That is the specification and claims are to be read broadly such that any combination or part combination of features may be recombined, and all alleged “intermediate generalizations” are possible unless explicitly disclaimed. Further a phrase referring to “at least one of” a list of items refers to any combination of those items, including single members. As an example, “at least one of: a, b, or c” is intended to cover: a, b, c, a-b, a- c, b-c, and a-b-c.

[0116] It will be appreciated by those skilled in the art that the disclosure is not restricted in its use to the particular application or applications described. Neither is the present disclosure restricted in its preferred example with regard to the particular elements and / or features described or depicted herein. It will be appreciated that the disclosure is not limited to the example orexamples disclosed, but is capable of numerous rearrangements, modifications and substitutions without departing from the scope as set forth and defined by the following claims. Features in dependent claims may be combined with features in other dependent claims, and the following claim dependencies should not be interpreted as an implication that a feature in one dependent claims cannot be combined with a feature in another dependent claim.

Claims

CLAIMS:

1. A spectral light detection and ranging (Lidar) spectral lidar monitoring system to monitor an earthen working equipment including a load container having a plurality of ground engaging tools (GETs), comprising: a. a spectral Lidar apparatus configured to: i. emit a plurality of pulses of light into a field of view at a plurality of emitted wavelengths, the plurality of emitted wavelengths including a plurality of detection wavelengths, and ii. capture spatial data and spectral data at the plurality of detection wavelengths, the spectral data including an intensity of a return signal at each of the plurality of detection wavelengths; and b. a computing apparatus including at least one processor and at least one memory and a communications interface configured to operatively connect to the spectral Lidar apparatus, and the at least one memory includes computer-executable instructions for configuring the at least one processor to: i. trigger the spectral Lidar apparatus to emit the plurality of pulses of light into a field of view (FOV) at a plurality of emitted wavelengths, the plurality of emitted wavelengths including a plurality of detection wavelengths, and to capture spatial data and spectral data at the plurality of detection wavelengths, the spectral data including an intensity of a return signal at each of the plurality of detection wavelengths collected at a plurality of spatial datapoints in the field of view; ii. receive and process the spatial data and spectral data from the spectral lidar apparatus to generate one or more spectral point cloud datasets for the FOV, wherein the one or more spectral point cloud datasets include the plurality of spatial datapoints within the FOV wherein each of the plurality of spatial datapoints includes an estimate of a reflectance at at least one detection wavelength for each spatial datapoint within the FOV; iii. classify each spatial datapoint in the one or more spectral point cloud datasets for the FOV to determine if one or more materials of interest are located at the respective spatial datapoint using the estimate of the reflectance at each of the detection wavelengths for the respective spatial datapoint, wherein the one or more materials of interest includes at least one load container material;iv. filtering the spatial datapoints to generate a material point cloud dataset for each of the one or more materials of interest, wherein each material point cloud dataset includes the set of spatial datapoints in a field of view classified as the respective material of interest; and v. performing a shape analysis on the one or more material point cloud datasets using one or more reference shapes for at least a portion of the load container to determine one or more of a missing GET, a worn GET, a carryback volume, a load container volume, or a payload.

2. The system of claim 1 , wherein the computing apparatus is further configured to generate an alert or electronic report based on the shape analysis.

3. The system of claim 2, further comprising an alert device communicatively coupled to the computing apparatus to receive a GET status signal indicating a missing GET or worn GET and provide an audible and / or visible alert based on receiving the GET status signal.

4. The system of claim 1 , wherein the spectral lidar apparatus is further configured to be mounted on an earthen working equipment having a load container with a plurality of ground engaging tools (GETs) such that the field of view is orientated to capture at least a portion of the load container including the GETs during at least part of an operational cycle.

5. The system of claim 1 , wherein the plurality of pulses of light are either a plurality of single pulses of light each including the plurality of emitter wavelengths, or are a plurality of sets of single wavelength pulses where each pulse of light in a set is emitted at one of the plurality of emitter wavelengths.

6. The system of claim 1 , wherein the one or more spectral point cloud datasets for the FOV are a single spectral point cloud for the FOV that includes a plurality of estimates of reflectance for each of the plurality of detection wavelengths at each of the plurality of spatial datapoints within the FOV .

7. The system of claim 1 , wherein the one or more spectral point cloud datasets for the FOV are a plurality of spectral point clouds for each of the detection wavelengths for the FOV wherein each spectral point cloud includes the plurality of spatial datapoints for the FOV and an estimate of a reflectance at the respective detection wavelength for each spatial datapoint within the FOV.

8. The system of claim 1 , wherein each spatial datapoint is a three dimensional location within the field of view and is determined using a pointing direction of an emitted pulse oflight and a time of flight of the return signal, and wherein the estimate of a reflectance at each spatial datapoint is determined using the measurement of the intensity of the return signal at the respective detection wavelength and a measurement of the intensity of the emitted signal at a reference wavelength.

9. The system of claim 1 , wherein the at least one memory stores a trained material machine learning classifier model trained to identify each of the one or more materials of interest and classifying each spatial datapoint is performed using the trained machine learning classifier model, wherein the trained material machine learning classifier model is trained using a training dataset including multiple estimates of a reflectance for each detection wavelength for each of the one or more materials of interest, and / or multiple spectral point clouds collected from a field of view including the one or more materials of interest.

10. The system as claimed in claim 9, wherein the training dataset is a labelled training dataset including multiple spectral point cloud datasets in which each spatial datapoint is labelled with a material class.

11. The system of claim 1 , wherein the at least one memory stores a reflectance signature for each material of interest based on a reference reflectance of the respective material at each of the detection wavelengths and classifying each spatial datapoint is performed by comparing the estimate of the reflectance at the plurality of detection wavelength with one or more of the reference reflectance signatures and based on the comparison determining if the respective material of interest is located at the spatial datapoint.

12. The system of claim 1 , wherein the one or more reference shapes includes one or more GET shapes and the at least one memory stores a trained machine learning shape analysis model, and the shape analysis is performed using the trained machine learning shape analysis model, wherein the trained machine learning shape analysis model is trained using a training dataset including multiple estimates of the GET shapes to determine one or more missing GETs or the worn GETs.

13. The system of claim 1 , wherein the at least one memory stores a trained machine learning shape analysis model, and the shape analysis is performed using the trained machine learning shape analysis model, wherein the trained machine learning shape analysis model is trained using a training dataset including multiple estimates of at least a portion of the load container to determine one or more of the carryback volume, the load container volume, or the payload.

14. The system of claim 1 , wherein the at least one load container material incudes at least one GET material, at least one memory stores a reference shape for each GET of theload container and the shape analysis includes at least comparing the reference shape of each GET with the each material point cloud dataset for the at least one GET material to determine one or more missing GETs or worn GETs.

15. The system of claim 1 , wherein the one or more materials of interest further includes earthen material, and filtering the spatial datapoints generates at least one material point cloud for the at least one load container material and at least one material point cloud for the at least one earthen material, and the shape analysis includes at least comparing the reference shape of the load container with each of the at least one material point cloud dataset for the at least one load container material and using each material point cloud dataset for the earthen material point cloud dataset to determine one or more of the carryback volume, the load container volume, or the payload.

16. The system of claim 15, wherein filtering the spatial datapoints further includes filtering spatial datapoints classified as earthen material based on a depth threshold determined using a depth of one or more spatial datapoints classified as load container material.

17. The system of claim 1 , wherein the type of ground engaging tools includes at least one of the following: a point, a tip, a shroud, a ripper tooth, a wing shroud, and a button.

18. The system of claim 1 , wherein the computer executable instructions further include using a scale transformation to scale each spectral point cloud at each of the detection wavelengths or a material point cloud dataset to a predetermined reference size.

19. The system of claim 1 , wherein the spectral Lidar apparatus includes a spectral source, a scene scanner, a spectral receiver, and one or more mounts, and the computer apparatus further includes a Lidar controller having of at least one processor and at least one memory, and the Lidar controller is operatively connected to the computer apparatus, wherein: a. the spectral source is configured to generate the plurality of pulses of light and each pulse of light includes the plurality of emission wavelengths; b. the scene scanner is configured to emit, for each generated pulse of light, the respective pulse of light at a pointing direction within the field of view, and for each emitted pulse of light the scene scanner captures a return signal; c. the spectral receiver is configured to measure, for each return signal, an intensity of the return signal at each of a plurality of detection wavelengths, and a time of flight; andd. the one or more mounts configured to mount the spectral Lidar apparatus on an earthen material and earth moving equipment including a load container having a plurality of ground engaging tools (GETs), wherein one of the one or more mounts is configured to mount the scene scanner such that the field of view is directed at a target location, such that when the earthen material and earth moving equipment is in operational use, the target location includes at least a portion of the load container including the plurality of GETs; and e. the at least one memory of the Lidar controller includes instructions to control the spectral Lidar apparatus to trigger the spectral Lidar apparatus to generate and emit the plurality of pulses of light at the plurality of wavelengths into the field of view, and generate the one or more spectral point cloud datasets from the measurements of the spectral receiver and the pointing direction of the scene scanner for each for each pulse of light, and to send the generated one or more spectral point cloud datasets to the computer apparatus.

20. The system of claim 19, wherein the spectral receiver is configured to measure an intensity of the emitted pulse of light at at least one reference wavelength.

21. The system of claim 19, wherein the spectral source includes a super continuous laser.

22. The system of claim 19, wherein the spectral receiver is a hyperspectral receiver included of a plurality of channels, wherein the plurality of detection wavelengths corresponds to selected channels in the plurality of channels.

23. The system of claim 19, wherein the spectral Lidar receiver is a multispectral receiver including a detection channel for each detection wavelength.

24. The system of claim 19, wherein the spectral source includes a band pass filter selected to block emission of all wavelengths except the emission wavelengths.

25. The system of claim 19, wherein the spectral source includes one or more filters selected to block emission of all wavelengths except the detection wavelengths.

26. The system of claim 1 , wherein the plurality of detection wavelengths is determined based on the one or more materials of interest.

27. A computer implemented method to monitor an earthen material and earth moving equipment including a load container having a plurality of ground engaging tools (GETs) using a spectral light detection and ranging (Lidar) apparatus, the method comprising:a. generating, one or more spectral point cloud datasets for a Field of view (FOV) from spatial data and spectral data generated by a spectral lidar apparatus, wherein the one or more spectral point cloud datasets include a plurality of spatial datapoints within the FOV and each of the plurality of spatial datapoints includes an estimate of a reflectance at at least one detection wavelength for each spatial datapoint within the FOV; b. classifying each spatial datapoint in the one or more spectral point cloud datasets for the FOV to determine if one or more materials of interest are located at the respective spatial datapoint using the estimate of the reflectance at each of the detection wavelengths for the respective spatial datapoint, wherein the one or more materials of interest include at least one load container material; c. filtering the spatial datapoints to generate a material point cloud dataset for each of the one or more materials of interest, wherein each material point cloud dataset is included of the set of spatial datapoints in a field of view classified as the respective material of interest; and d. performing a shape analysis on the one or more material point cloud datasets using one or more reference shapes for at least a portion of the load container to determine one or more of a missing GET, a worn or broken GET, a carryback volume, a load container volume, or a payload.

28. The computer implemented method of claim 26, further including generating an alert or electronic report based on the shape analysis.

29. The computer implemented method of claim 26, wherein the one or more spectral point cloud datasets for the FOV are a single spectral point cloud for the FOV that includes a plurality of estimates of reflectance for each of the plurality of detection wavelengths at each of the plurality of spatial datapoints within the FOV.

30. The computer implemented method of claim 26, wherein the one or more spectral point cloud datasets for the FOV are a plurality of spectral point clouds for each of the detection wavelengths for the FOV wherein each spectral point cloud includes the plurality of spatial datapoints for the FOV and an estimate of a reflectance at the respective detection wavelength for each spatial datapoint within the FOV.31 . The computer implemented method of claim 27, wherein each spatial datapoint is a three dimensional location within the field of view and is determined using a pointing direction of an emitted pulse of light and a time of flight of the return signal, and wherein the estimate of a reflectance at each spatial datapoint is determined using a measurement ofthe intensity of the return signal at the respective detection wavelength and a measurement of the intensity of the emitted signal at a reference wavelength.

32. The computer implemented method of claim 27, wherein classifying each spatial datapoint is performed using a trained machine learning classifier model trained to identify each of the one or more materials of interest and is trained using a training dataset including multiple estimates of a reflectance for each detection wavelength for each of the one or more materials of interest and / or multiple spectral point clouds collected from a field of view including the one or more materials of interest.

33. The computer implemented method of claim 32, wherein the training dataset is a labelled training dataset including multiple spectral point cloud datasets in which each spatial datapoint is labelled with a material class.

34. The computer implemented method of claim 27, wherein a reference spectral signature is generated for each material of interest based on the reflectance of the respective material at each detection wavelength, and classifying each spatial datapoint is performed by comparing the estimate of the reflectance at the plurality of detection wavelengths with one or more of the reference spectral signatures and based on the comparison determining if the respective material of interest is located at the spatial datapoint.

35. The computer implemented method of claim 27, wherein the one or more reference shapes includes one or more GET shapes, and the shape analysis is performed using a trained machine learning shape analysis model, wherein the trained machine learning shape analysis model is trained using a training dataset including multiple estimates of the GET shapes to determine one or more missing GETs or the worn GETs.

36. The computer implemented method of claim 27, wherein the shape analysis is performed using a trained machine learning shape analysis model, wherein the trained machine learning shape analysis model is trained using a training dataset including multiple estimates of at least a portion of the load container to determine one or more of the carryback volume, the load container volume, or the payload.

37. The computer implemented method of claim 27, wherein the at least one load container material incudes at least one GET material and the one or more reference shapes includes a reference shape for each GET of the load container and performing the shape analysis includes at least comparing the reference shape of each GET with each material point cloud dataset for the at least one GET material to determine one or more missing GETs or worn GETs.

38. The computer implemented method of claim 36, wherein the one or more materials of interest further includes earthen material, and filtering the spatial datapoints generates at least one material point cloud for the at least one load container material and at least one material point cloud for the at least one earthen material, and the shape analysis includes at least comparing the reference shape of the load container with each of the at least one material point cloud dataset for the at least one load container material and using each material point cloud dataset for the earthen material point cloud dataset to determine one or more of the carryback volume, the load container volume, or the payload.

39. The computer implemented method of claim 37, wherein filtering the spatial datapoints further includes filtering spatial datapoints classified as earthen material based on a depth threshold determined using a depth of one or more spatial datapoint classified as load container material.

40. The computer implemented method of claim 27, further including using a scale transformation to scale each spectral point cloud at each of the detection wavelengths or a material point cloud dataset to a predetermined reference size.

Images