System and method for detecting wear and loss of ground-engaging tools on a work machine

Abstract

The exemplary wear detection system (110) receives a plurality of images from a plurality of sensors (126, 128) associated with the work machine (100). Each sensor of the plurality of sensors has a respective field of view (127, 129) that is different from other sensors of the plurality of sensors. The wear detection system identifies a first region of interest (550) and a second region of interest (550) associated with the at least one GET. The wear detection system determines a first set of image points (920) and a second set of image points (920) for the at least one GET based on geometric parameters (535) associated with the GET. The wear detection system determines a wear level or loss of the at least one GET based on the GET measurements.

Description

[Technical Field]

[0001] The present disclosure relates to systems and methods for detecting wear on an object over time, and more particularly to systems and methods for detecting wear or loss of one or more ground engaging tools (GETs) over time using imaging techniques. [Background technology]

[0002] Machines may be used to perform a variety of tasks at a worksite. For example, machines may be used to excavate, move, shape, contour, and / or remove materials present at the worksite, such as gravel, concrete, asphalt, soil, and / or other materials. These machines may include buckets used to collect such materials, which may include a set of GETs, such as teeth, to loosen the material. GETs may also include shrouds attached to the bucket between the teeth to protect the bucket's edges. Over time, GETs wear and reduce in size, reducing their effectiveness and making it more difficult for the bucket to collect materials at the worksite. GETs may break off from the bucket. If GET breakage is not detected, the GETs may mix with worksite materials and cause damage to downstream processing equipment, such as crushers and pulverizers. Work machines may utilize wear detection systems to identify worn or broken GETs before damage occurs to downstream equipment.

[0003] An attempt to provide a wear detection system is described in International Publication No. 2020 / 237324 ("the '324 publication"), published on December 2, 2020. The '324 publication describes a system for monitoring the condition of a GET, including one or more sensors attached to a work device and pointed at the GET. The system receives data from the one or more sensors, generates a three-dimensional (3D) representation of at least a portion of the GET, and compares a currently generated 3D representation of the GET with a previously generated 3D representation of the GET. The system determines wear or loss of the GET based on the comparison.

[0004] However, relying solely on 3D representations to detect GET wear or loss, as described in the '324 publication, has drawbacks. 3D representations of GETs can be overly dense, and processing them can consume significant computational resources. In environments with a large number of sensors, environments where the GET must be scanned multiple times during a work machine's dig-dump cycle, or environments where speed of GET wear detection is important, processing the 3D representations is impractical due to computational complexity. As a result, the system described in the '324 publication may not be suitable for some environments. The systems and methods described herein are intended to solve one or more of these problems. Summary of the Invention

[0005] A ground engaging tool (GET) wear detection method according to a first aspect includes receiving a plurality of images from a plurality of sensors associated with a work machine, each sensor of the plurality of sensors having a respective field of view different from other sensors of the plurality of sensors, the plurality of images including an image of at least one GET of the work machine. The method also includes identifying a first region of interest associated with the at least one GET and included in a first image of the plurality of images, the first region of interest including first data characterizing the at least one GET. The method also includes identifying a second region of interest associated with the at least one GET and included in a second image of the plurality of images, the second region of interest including second data characterizing the at least one GET. The method also includes determining a first set of image points for the at least one GET, the first set of image points being determined based at least in part on geometric parameters associated with the at least one GET and a first edge of the at least one GET detected in the first data. The method also includes determining a second set of image points for the at least one GET, the second set of image points being determined based at least in part on the geometric parameters associated with the at least one GET and a second edge of the at least one GET detected in the second data, determining a GET measurement for the at least one GET based at least in part on the first set of image points and the second set of image points, and determining a wear level or loss of the at least one GET based on the GET measurement.

[0006] According to a further aspect, a GET wear detection system includes one or more processors and a plurality of sensors associated with a work machine, the plurality of sensors having respective fields of view oriented to capture a plurality of images of at least one GET of the work machine. The GET wear detection system also includes a non-transitory computer-readable medium storing executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including identifying a first region of interest associated with the at least one GET and included in a first image of the plurality of images, the first region of interest including first data characterizing the at least one GET. The operations also include identifying a second region of interest associated with the at least one GET and included in a second image of the plurality of images, the second region of interest including second data characterizing the at least one GET. The operations also include determining a first set of image points for the at least one GET, the first set of image points being determined based at least in part on geometric parameters associated with the at least one GET and a first edge of the at least one GET detected in the first data. The operations also include determining a second set of image points for the at least one GET, the second set of image points being determined based at least in part on the geometric parameters associated with the at least one GET and a second edge of the at least one GET detected in the second data, determining a GET measurement value for the at least one GET based at least in part on the first set of image points and the second set of image points, and determining a wear level or loss of the at least one GET based on the GET measurement value.

[0007] According to another aspect, a work machine includes a bucket including at least one GET, a stereo camera including a left image sensor and a right image sensor, an infrared sensor, one or more processors, and a non-transitory computer-readable medium storing executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including receiving a left image of the at least one GET captured by the left image sensor, receiving a right image of the at least one GET captured by the right image sensor, and generating a disparity map based on the left image and the right image, identifying a first region of interest associated with the at least one GET based on the disparity map, receiving infrared image data from the infrared sensor, and identifying a second region of interest associated with the at least one GET within the infrared image data. The operations also include determining a first set of image points for the at least one GET based at least in part on geometric parameters associated with the at least one GET and a first edge of the at least one GET detected in the disparity map, and determining a second set of image points for the at least one GET determined based at least in part on the geometric parameters associated with the at least one GET and a second edge of the at least one GET detected in the second data, determining a GET measurement for the at least one GET based at least in part on the first set of image points and the second set of image points, and determining a wear level or loss of the at least one GET based on the GET measurement. [Brief explanation of the drawings]

[0008] Specific embodiments will now be described with reference to the accompanying drawings, in which: In each figure, the leftmost digit(s) of a reference number identifies the figure in which the reference number first appears; the same reference numbers in different figures indicate similar or identical items. [Figure 1]FIG. 1 is a block diagram that schematically illustrates an example machine including an example system for detecting wear in a GET. [Figure 2] 1 is a schematic side view illustrating an example environment having an example machine including an example system for detecting wear in a GET. [Figure 3] FIG. 1 is a schematic side view illustrating another example environment having an example machine including an example system for detecting wear in a GET. [Figure 4] FIG. 1 illustrates a schematic side view of another exemplary environment having an exemplary machine including an exemplary system for detecting wear in a GET. [Figure 5] FIG. 1 is an image data flow diagram showing an example of the flow of image data for region of interest detection processing using computer vision technology. [Figure 6] FIG. 1 is an image data flow diagram showing an example of the flow of image data for region of interest detection processing using deep learning techniques. [Figure 7] FIG. 1 is an image data flow diagram showing an example of the flow of image data for region of interest detection processing using deep learning techniques. [Figure 8] FIG. 10 is an image data flow diagram showing an example of the flow of image data for region of interest detection processing using a LiDAR sensor and imaging data. [Figure 9] FIG. 10 is an image data flow diagram showing an example of image data flow for wear detection processing using image points. [Figure 10] 1 is an exemplary process for detecting wear in an exemplary environment. [Figure 11] 1 is an exemplary process for detecting wear in an exemplary environment. [Figure 12] 1 is an exemplary user interface of an exemplary system for detecting wear on a GET. DETAILED DESCRIPTION OF THE INVENTION

[0009] The present disclosure is generally directed to systems and methods for detecting wear on a work machine component in an environment, such as a work site, using one or more sensors. The one or more sensors may include an imaging sensor (which may be part of a stereoscopic camera or “stereo camera”), a LiDAR sensor, an infrared (IR) sensor, a sonar sensor, a temperature sensor, or a radar sensor that can capture imaging data related to the component. The imaging data may include, but is not limited to, video, images, LiDAR imaging data, IR imaging data, audio-based imaging data, or radar data. The imaging data is analyzed by a wear detection computer system associated with the work machine, which may be located on the work machine, in a stereo camera, in one or more sensors on the work machine, or external thereto or external to the work machine, to detect wear on the component. For example, the component may be one or more getters on the work machine bucket. The one or more sensors on the work machine may each have a different field of view and may generate slightly different imaging data for the component. The different fields of view may reduce errors related to poor lighting conditions, shadows, or debris that may adversely affect imaging of the component. The wear detection system may determine image points from the captured imaging data that are associated with marker points of the GET (e.g., edges, corners, or visual indicators on the component) and use the image points to determine measurements of the component. The wear detection system may determine the wear level or loss of the component based on the determined measurements and / or historical or baseline measurements. The wear detection system may receive imaging data from one or more sensors at various points in the work machine's dig-dump cycle and determine whether to generate an alert or capture additional imaging data based on the placement of the points in the dig-dump cycle.

[0010] FIG. 1 is a schematic block diagram of an example work machine 100 including an example wear detection computer system 110. In FIG. 1, work machine 100 is depicted as a hydraulic mining shovel; however, in other examples, work machine 100 may include any machine that moves, carves, excavates, or removes material, such as soil, rock, or minerals. As shown in FIG. 1, work machine 100 may include a bucket 120 attached to an arm 122. Bucket 120 may include one or more ground engaging tools (GETs) 125, such as teeth, that assist work machine 100 in loosening material. In the examples provided herein, GETs 125 are generally referred to as teeth, although other types of GETs are considered within the scope of embodiments provided by this disclosure. For example, GETs may include lip shrouds, edge guards, adapters, ripper protectors, cutting edges, sidebar protectors, tips, or other tools associated with the work machine that wear over time due to friction with worksite materials.

[0011] Work machine 100 also includes one or more sensors having respective fields of view, such as sensor 126 having field of view 127 and stereo camera 128 having field of view 129. Field of view 127 and field of view 129 are both directed toward bucket 120 and GET 125. As shown in FIGURE 1, field of view 127 and field of view 129 are different, although they overlap. Sensor 126 may include an image sensor, a LiDAR sensor, an IR sensor, a sonar sensor, or a radar sensor, just to name a few examples.

[0012] As used in this disclosure, the term “imaging data” refers to data generated by sensor 126 or stereo camera 128 and received by wear detection computer system 110 that can be interpreted or processed to reflect the size, shape, or appearance of GET 125. While this disclosure refers to a singular sensor 126, in some embodiments, work machine 100 includes two or more sensors 126, in addition to stereo camera 128, typically each with its own field of view 127. For example, work machine 100 may include camera 128 with field of view 129, a LiDAR sensor, an additional imaging sensor, and an IR sensor, all of which may generate imaging data that is processed by wear detection computer systems according to embodiments of the present disclosure.

[0013] In some embodiments, sensor 126 includes an adaptive scanning LiDAR sensor, i.e., a LiDAR sensor whose resolution and field of view can be commanded, controlled, and configured. For example, sensor 126 can include an AEYE 4Sight M™. In some embodiments, field of view 127 may start from a baseline of 60 degrees by 30 degrees (representing a “low” resolution range scan) and then be adjusted in 0.1 degree increments to a high-definition region of interest spanning 0.025 degrees, although other fields of view and angular resolutions may exist in other embodiments. Sensor 126 can be configured to collect up to 1600 points per square degree at a frequency of 100 Hz. The accuracy of sensor 126 is a function of the angular resolution of field of view 127 and the distance between sensor 126 and GET 125. For example, if GET 125 is approximately 6 meters away from sensor 126 and field of view 127 is configured at 60 degrees by 30 degrees, scanning 1600 points per square degree will produce LiDAR hits within an imaging rectangle of approximately 7.2 meters by 3.2 meters. By refocusing the field of view, LiDAR hits can be recorded as wide as 2.6 millimeters horizontally and vertically. While an example sensor 126 is described above, different LiDAR sensors capable of adaptive scanning can be used in various embodiments.

[0014] The sensor 126 may also include an infrared sensor or a sensor capable of detecting the thermal signature of the GET 125. For example, the sensor 126 may include a long-wave FLIR® infrared camera with a 640x512 resolution, a 9Hz refresh rate, and a 75-degree field of view. The infrared sensor 126 may complement the camera 128 in low light environments or environments where debris may adhere to the GET 125 during operation. Other examples of the sensor 126 may include a sonar or radar sensor.

[0015] The stereo camera 128 includes left and right image sensors spaced apart to capture stereo images of objects within a field of view 129, such as the bucket 120 and the GET 125. In some embodiments, the left and right image sensors capture monochrome images. The stereo camera 128 may further include a color image sensor to capture color images of objects within the field of view 129. In some embodiments, the camera 128 outputs digital images, or the work machine 100 may include an analog-to-digital converter disposed between the camera 128 and the wear detection computer system 110 to convert the analog images to digital images before the wear detection computer system 110 receives them. While this disclosure refers to the stereo camera 128 having a single field of view 129 for ease of discussion, one skilled in the art will understand that each image sensor (e.g., left, right, color) of the camera 128 has a respective field of view to generate stereo images from which the GET 125 may be measured in accordance with embodiments of the present disclosure.

[0016] In some embodiments, one or more sensors of work machine 100, such as sensor 126 and camera 128, may include a lens cleaning device for removing debris, mist, or other obstructions from the lens surface (or screen) of the one or more sensors. The lens cleaning device may include, for example, a nozzle that sprays compressed air, a cleaning solvent, or a cleaning antifreeze solution. The lens cleaning device may further include a movable wiper configured to contact and wipe the surface of the lens, pushing debris or other obstructions from the lens surface. In some embodiments, the cover of the lens of one or more sensors may include an actuator that rotates the lens screen (in the case of a cylindrical lens screen) or slides the lens screen (in the case of a planar lens screen) to contact one or more wipers and remove debris from the screen.

[0017] As work machine 100 operates within a work site, it may move arm 122 to position bucket 120 to move or excavate material within the work site as part of a dig-dump cycle. Work machine 100 may move bucket 120 in and out of field of view 127 and field of view 129 as it positions bucket 120 through a dig-dump cycle. Sensor 126 and camera 128 may be positioned to provide an unobstructed view of GET 125 during a dig-dump cycle. For example, sensor 126 and camera 128 may be positioned on work machine 100 to view bucket 120 and GET 125 at the moment the bucket 120 empties material within a dig-dump cycle. As another example, sensor 126 and camera 128 may be positioned to provide a view of bucket 120 and GET 125 when arm 122 is fully extended or retracted during a dig-dump cycle. As described below with respect to Figures 2-4, the locations of sensor 126 and camera 128 (and their respective fields of view 127 and 129) may vary depending on the type of work machine 100 and the specifics of the work site.

[0018] In some embodiments, field of view 127 and field of view 129 may capture image data for bucket 120 and GET 125 at different points in a dig-dump cycle. For example, sensor 126 may capture image data for GET 125 early in the dig-dump cycle (e.g., closer to the beginning of the cycle than the end of the cycle), and camera 128 may capture image data for GET 125 later in the dig-dump cycle (e.g., closer to the end of the cycle than the beginning of the cycle). In some embodiments, sensor 126 and / or camera 128 may adjust their respective fields of view 127, 129 to collect image data for GET 125 at different points in the dig-dump cycle. For example, in some embodiments, both sensor 126 and camera 128 may capture image data for GET 125 early in the dig-dump cycle and then adjust their fields of view 127, 129 to capture image data for GET 125 later in the dig-dump cycle.

[0019] According to some embodiments, the work machine 100 includes an operator control panel 130. The operator control panel 130 may include a display 133 that generates output for the operator of the work machine 100 so that the operator can receive status or alerts related to the wear detection computer system 110. The display 133 may include a liquid crystal display (LCD), a light emitting diode display (LED), a cathode ray tube (CRT) display, or other types of displays known in the art. In some examples, the display 133 includes an audio output, such as a speaker or a port for headphones or peripheral speakers. The display 133 may also include an audio input device, such as a microphone or a port for a peripheral microphone. In some embodiments, the display 133 includes a touch-sensitive display screen that also functions as an input device.

[0020] The display 133 may display information regarding the wear level or loss of the GET 125 rendered by the wear computer detection system 110. For example, the display 133 may display a calculated measurement value of the GET 125. The calculated measurement value, in some embodiments, may be color-coded to reflect the health of the GET 125. For example, the calculated measurement value may be displayed against a green background if the GET 125 is deemed to have an acceptable level of wear, against a yellow background if the GET 125 is close to needing replacement, or in red if the GET 125 has broken or worn to the point where it requires replacement. The display 133 may also display an image of the bucket 120, an image of the GET 125, or an image of an area of interest within the field of view 127, 129 associated with the GET 125 rendered by the wear computer detection system 110.

[0021] In some embodiments, operator control panel 130 also includes keyboard 137. Keyboard 137 provides input capability for wear detection computer system 110. Keyboard 137 includes a plurality of keys that allow an operator of work machine 100 to provide input to wear detection computer system 110. For example, according to examples of the present disclosure, the operator may press keys on keyboard 137 to select or input the type of work machine 100, bucket 120, and / or GET 125. Keyboard 137 may be non-virtual (e.g., including those that allow physical keys to be pressed), or keyboard 137 may be a virtual keyboard that is displayed on a touch-sensitive embodiment of display 133.

[0022] As shown in FIG. 1 , the wear detection computer system 110 includes one or more processors 140. The processor 140 may include one or more of a central processing unit (CPU), a graphics processing unit (GPU), a field programmable gate array (FPGA), some combination of a CPU, GPU, or FPGA, or any other type of processing unit. The processor 140 may have multiple arithmetic logic units (ALUs) that perform arithmetic and logical operations, and one or more control units (CUs) that retrieve instructions and memory contents from processor cache memory and then invoke the ALUs to execute instructions as needed during program execution. The processor 140 may also be responsible for executing drivers and other computer-executable instructions for applications, routines, or processes stored in memory 143, which may be associated with common types of volatile (RAM) and / or non-volatile (ROM) memory.

[0023] The wear detection computer system 110 further includes memory 143. The memory 143 can include system memory, which can be volatile (e.g., RAM), nonvolatile (e.g., ROM, flash memory, etc.), or any combination thereof. The memory 143 can also include non-transitory computer-readable media, such as volatile and non-volatile, removable and non-removable media, implemented by any method or technology for storage of information, such as computer-readable instructions, data structures, program modules, or other data. System memory, removable storage, and non-removable storage are examples of non-transitory computer-readable media. Examples of non-transitory computer-readable media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital multifunction disk (DVD) or other optical storage, cassette tape, magnetic tape, magnetic disk storage or other magnetic storage, or other non-transitory media used to store the necessary information and which can be accessed by the wear detection computer system 110.

[0024] The memory 143 stores data including computer-executable instructions for the wear detection computer system 110, as described herein. For example, the memory 143 may store one or more components of the wear detection computer system 110, such as the physical parameter library 145, the image analyzer 150, the wear analyzer 153, the alert manager 155, and the GET wear level storage 157. The memory 143 may also store additional components, modules, or other code executable by the processor 140 to enable operation of the wear detection computer system 110. For example, the memory 143 may include code related to input / output functions, software drivers, an operating system, or other components.

[0025] According to some embodiments, various aspects of the wear detection computer system 110 may be located within the camera 128. For example, the camera 128 may include one or more of the processor 140 and / or the memory 143. Similarly, various aspects of the wear detection computer system 110 may be located within the sensor 126. Additionally, or alternatively, various aspects of the wear detection computer system 110 may be located on the work machine 100 and external to the sensor 126 or the camera 128.

[0026] Physical parameter library 145 includes sets of physical parameters related to work machine 100, bucket 120, GET 125, sensor 126, and / or camera 128. For example, physical parameter library 145 may include measurement data related to the size of bucket 120, the shape of bucket 120, the size of GET 125, the shape of GET 125, and the spatial relationship between GET 125 and bucket 120, and / or the spatial relationship between sensor 126 and camera 128, to name a few examples. Physical parameter library 145 may further include parameters related to the size and shape of GET 125 in a new or unworn condition, and parameters related to the size and shape of GET 125 when maximum wear is reached.

[0027] In some embodiments, physical parameter library 145 may include geometric parameters related to marker points on bucket 120 or GET 125. The marker points relate to aspects or reference points on bucket 120 and GET 125 that may be used by wear analyzer 153 to measure wear or loss on GET 125. For example, the marker points may include corners of GET 125, edges of bucket 120 where GET 125 engages bucket 120, corners between the edges of bucket 120 and GET 125, or physical markers applied to GET 125, such as welds, paint, grooves, reflective tape, or bar codes. Physical parameter library 145 may include physical parameter sets with information regarding the relative positions of the marker points with respect to GET 125 or bucket 120. For example, the physical parameter sets may include the angle of a corner of GET 125, the relative position of a physical marker on GET 125 with respect to the edge of GET 125 or the edge of bucket 120. Embodiments of the wear detection computer system 110 may include physical parameter sets beyond the specific examples described herein, and those skilled in the art will understand that other methods of detecting marker points on the bucket 120 or GET 125 may be used, and that the physical parameter library 145 may include physical parameter sets that may be used by the wear analyzer 153 to identify marker points in image data collected by the sensor 126 and / or camera 128.

[0028] The physical parameter library 145 may also include templates or reference images (e.g., bucket tool templates) for the bucket 120 and GET 125 combination. For example, for the work machine 100, one of the templates stored in the physical parameter library 145 may include an image of the bucket 120 having the GET 125 as the bucket 120 is expected to be located within the field of view 127 and / or the field of view 129. The bucket tool template may represent an unworn GET 125 (e.g., an unworn or expected edge) or a GET 125 that has reached maximum wear (e.g., a threshold edge). The physical parameter library 145 may also include other information regarding the wear on the GET 125 to help the wear analyzer 153 determine when the GET has worn enough to require replacement. The wear data for the GET 125 may be in an actual measured format (e.g., metric or imperial size) or in the form of pixel values, as some examples.

[0029] As another example, the physical parameter library 145 may include a CAD-based model of the GET 125. The CAD-based model may be a reference GET 125 model developed using a computer-aided design program such as AutoCAD®, Autodesk®, SolidWorks®, or other well-known CAD programs. The wear detection computer system 110 may compare the observed size and shape of the GET 125 with a model of the same type, a standard, or an unworn GET, using the CAD-based model as a reference point, to determine wear or loss of the GET 125. In some embodiments, the CAD-based model may include the location, orientation, and / or relative positioning of points on the GET 125.

[0030] Physical parameter library 145 may include multiple physical parameter sets, each corresponding to a work machine, a bucket, a GET, or a combination thereof. During operation, an operator may use operator control panel 130 to select a physical parameter set from physical parameter library 145 that is compatible with bucket 120 and GET 125 or work machine 100. For example, if work machine 100 is a hydraulic mining shovel having model number "6015B," the operator may enter model number "6015B" using operator control panel 130, and wear detection computer system 110 may load a physical parameter set corresponding to the hydraulic mining shovel model "6015B" from physical parameter library 145 into memory 143. In some examples, upon power-on or reset operation of wear detection computer system 110, a list of templates available in physical parameter library 145 may be displayed on display 133, and the operator may select one physical parameter set from the list to operate depending on the model number of work machine 100, the bucket type of bucket 120, or the type of GET 125.

[0031] In some embodiments, an operator can position bucket 120 and GET 125 within field of view 129 of camera 128 at the start of a work shift and have wear detection computer system 110 take images of bucket 120 and GET 125 using inputs on operator control panel 130. Wear detection computer system 110 can then perform an image matching process to match bucket 120 and GET 125 with a set of physical parameters and configure itself for the wear detection and image processing processes disclosed herein based on the matching physical parameter sets. In some embodiments, wear detection computer system 110 can use sensor 126 and field of view 127 in the configuration process instead of camera 128 and field of view 129.

[0032] Image analyzer 150 can be configured to analyze imaging data captured by either sensor 126 or camera 128 to identify GET 125 within field of view 127 and field of view 129 and measure wear on GET 125 based on processing of the imaging data. For example, image analyzer 150 can receive stereoscopic images from camera 128 in the form of a left-rectified image (captured by the left image sensor of camera 128) and a right-rectified image (captured by the right image sensor of camera 128). Image analyzer 150 can perform various computer vision techniques on the left-rectified image and the right-rectified image to identify or determine an area of interest corresponding to GET 125. As another example, image analyzer 150 can receive imaging data captured by sensor 126 that can be used to identify an area of interest corresponding to GET 125. In the disclosed embodiment, image analyzer 150 receives data from sensor 126 to determine wear or loss on GET 125, as described in more detail below.

[0033] In some embodiments, image analyzer 150 processes two sets of imaging data when wear or loss of GET 125 is detected. The first set of imaging data is captured to identify an area of interest within field of view 127 or field of view 129. The area of interest corresponds to the relative position of GET 125 within field of view 127 or field of view 129. The first set of imaging data for detecting an area of interest is captured of broad, low-resolution imaging data, sometimes referred to as a “coarse scan,” designed to locate a general area of interest on GET 125. In some embodiments, the first set of imaging data may be captured using camera 128, and image analyzer 150 determines the area of interest using computer vision or machine learning techniques. In other embodiments, the first set of imaging data may be captured using sensor 126 at a relatively wide, first, low resolution (e.g., 60 degrees by 30 degrees in a LiDAR embodiment of sensor 126). In some embodiments, image analyzer 150 receives the first set of imaging data from sensor 126 and camera 128.

[0034] Once the image analyzer 150 identifies a region of interest corresponding to the GET 125, it may control the sensor 126 to focus on the region of interest and perform a higher resolution scan, or in some embodiments, a "fine scan." For example, the image analyzer 150 may communicate with the application programming interface (API) of the sensor 126 to narrow the field of view 127 to focus on the identified region of interest. The sensor 126 then performs another scan of the GET 125 to collect a second set of imaging data. The second set of imaging data captured by the sensor 126 with the narrow field of view 127 has higher resolution than the first set of imaging data captured by either the sensor 126 (if set to a wide field of view) or the camera 128.

[0035] In some embodiments, the coarse scan can be performed by either the sensor 126 or the camera 128, and the fine scan can be performed by the other of the camera 128 or the sensor 126. Alternatively, both the sensor 126 and the camera 128 may perform both the coarse scan and the fine scan.

[0036] In one embodiment, the image analyzer 150 creates a high-density stereo disparity map based on the left-rectified and right-rectified images received from the camera 128. The image analyzer can segment the high-density stereo disparity map to identify regions of interest. Additionally, the image analyzer 150 can create a 3D point cloud based on the high-density stereo disparity map and segment the 3D point cloud to identify regions of interest.

[0037] In addition to or instead of using computer vision techniques, the image analyzer 150 can use deep learning or machine learning techniques to identify regions of interest in the left-rectified image and / or the right-rectified image captured by the camera 128. For example, the image analyzer 150 can use a deep learning GET detection algorithm that uses a neural network trained to identify regions of interest based on a corpus of images labeled with individual GETs, groups of GETs, or combinations of GETs and buckets. The image analyzer 150 can also use a deep learning GET location algorithm that employs a neural network trained to identify the location of GETs in an image. The GET location algorithm can be trained using a corpus of images labeled with individual GETs. Once the GET location algorithm identifies an individual GET in an image, it outputs the corresponding location of the GET. For example, the GET location algorithm can output a pixel location or bounding box output for the GET location.

[0038] In some embodiments, the deep learning GET detection algorithm includes a neural network trained to identify regions of interest based on disparity maps. For example, a corpus of training data for the deep learning GET detection algorithm can include disparity images between left-rectified and right-rectified images labeled with individual GETs, groups of GETs, or combinations of GETs and buckets.

[0039] As described above, once image analyzer 150 identifies a region of interest that includes GET 125, image analyzer 150 may instruct and control sensor 126 to focus field of view 127 or may instruct and control camera 128 to focus field of view 129 at the region of interest. In some embodiments, image analyzer 150 uses spatial relationship data between sensor 126 and camera 128 to instruct sensor 126 to change field of view 127 over the region of interest. For example, in a LiDAR embodiment, upon receiving a command to change its field of view, sensor 126 may change the configuration of its MEMS (micro-electro-mechanical systems) mirrors to a narrow field of view 127 to capture higher resolution imaging data for GET 125.

[0040] The image analyzer 150 can create a 3D point cloud corresponding to the GET 125 from the captured higher resolution imaging data. Each point in the 3D point cloud corresponds to a "hit" or detection point captured by the sensor 126, such as a LiDAR hit or an infrared thermal signature. In some embodiments, the actual distance between points may be as small as 1 mm. In embodiments with sufficiently high resolution (i.e., when the actual distance between points is less than about 2.5 mm), the image analyzer 150 sends the 3D point cloud data to the wear analyzer 153 for wear detection analysis. In other embodiments, the image analyzer 150 may perform additional processing to further refine the 3D point cloud data for wear analysis.

[0041] For example, in some embodiments, the image analyzer 150 can convert the 3D point cloud to a dense mesh surface. The image analyzer 150 can further convert the dense mesh surface to a sparse mesh surface before communicating the GET imaging data to the wear analyzer 153. The conversion from the 3D point cloud to a dense mesh surface and then to a sparse mesh surface may be desirable to reduce computational costs when comparing imaging data captured by the sensor 126 to a CAD-based GET model. The conversion from the 3D point cloud to a dense mesh surface and then to a sparse mesh surface can also remove noise that may be present in the imaging data due to oversampling.

[0042] In some embodiments, the wear analyzer 153 fuses the first received imaging data, which has a lower resolution, from the camera 128 with the second received imaging data, which has a higher resolution, from the sensor 126 to provide confidence in the observational measurements of the GET 125. In such embodiments, the image analyzer 150 performs additional processing on the left and right images captured by the camera 128. For example, once the image analyzer 150 has identified the regions of interest, it may further process them to create a left edge digital image corresponding to the left-corrected image and a right edge digital image corresponding to the right-corrected image. The image analyzer 150 may employ gradient magnitude search-based edge detection, although other edge detection techniques employed in the field of computer vision (e.g., zero-crossing based edge detection techniques) may be employed in other embodiments to create the left edge digital image and the right edge digital image.

[0043] In some examples, the image analyzer 150 can refine the edge estimate of the GET 125 and / or identify individual GETs 125 by using the expected location of the GET 125 within the captured image. For example, the image analyzer 150 can know the expected location of the GET 125 relative to the bucket 120 based on a set of physical parameters stored in the physical parameter library 145 corresponding to the type of bucket 120 and GET 125 being used. Using this information, the image analyzer 150 can reach the expected location within the selected image and image the pixel region closest to the tooth. The pixel region can then be used to further identify the tooth based on computer vision techniques, such as applying a convolution filter, segmentation analysis, edge detection, or pixel intensity / darkness analysis within the pixel region. In some embodiments, the image analyzer 150 can apply individual tooth templates to the pixel region and further refine the tooth location using computer vision techniques. The image analyzer 150 may further refine the edges using dynamic programming techniques. Dynamic programming techniques may include smoothing based on edge strength, whether the edge is near a hole or an uncertain region in the dense stereo disparity map, or other edge detection optimization techniques. The image analyzer 150 may also use the output of the GET location algorithm to gain confidence in determining the location of the GET and further refine the edge estimation based on the output of the GET location algorithm.

[0044] Image analyzer 150 can also create sparse stereo disparity that is provided to wear analyzer 153, which can be used with higher resolution imaging data captured by sensor 126 to determine wear or loss in GET 125. In some embodiments, image analyzer 150 creates sparse stereo disparity between a left edge digital image (associated with the left modified image) and a right edge digital image (associated with the right modified image), and this disparity is used by wear analyzer 153. Alternatively, image analyzer 150 can calculate sparse stereo disparity from a first region of interest image (associated with the left modified image) and a second region of interest image (associated with the right modified image) and detect edges from the sparse stereo disparity image.

[0045] Wear analyzer 153 can be configured to analyze the sparse stereo disparity generated by image analyzer 150 for wear. For example, the physical parameter set associated with bucket 120 and GET 125 can include expected data related to the unworn GET 125 or a set of unworn GETs 125 calibrated based on expected image capture of camera 128. As some examples, the expected data can be in the form of pixels, actual measurements, a CAD-based model of GET 125, or edge images for the unworn GET. Upon receiving the sparse stereo disparity, wear analyzer 153 can fuse and correlate the sparse stereo disparity with a three-dimensional point cloud of higher-resolution imaging data captured by sensor 126 (or, in some embodiments, a dense or sparse mesh surface determined based on the three-dimensional point cloud) to determine measurement data for GET 125. The determined measurement data can then be compared to expected data corresponding to the unworn version of GET 125 to determine the wear level or loss of GET 125.

[0046] In some embodiments, the image analyzer 150 identifies marker points in the image data collected from the sensor 126 and / or the camera 128. As described above, the marker points may refer to corners of the GET 125, edges of the bucket 120 that fit on the GET 125, or physical or visual markers on the bucket 120 or the GET 125. For example, if the image analyzer 150 identifies corners of the GET 125 as marker points, the image analyzer 150 may perform corner detection techniques known in the field of computer vision, such as Moravec, Harris & Stephens, Shi-Tomasi, Forstner, Multiscale Harris, Laplacian of Gaussian, Wang and Brady, SUSAN (Single-Valued Segment Assimilation Kernel), Trajkovic, Hedley, and / or Hessian techniques, to name a few. In addition to performing segmentation, pattern matching, or template matching analysis, other corner detection methods may be used to identify marker points based on the image data collected by the sensor 126 and / or the camera 128.

[0047] In some embodiments, image analyzer 150 may use deep learning or machine learning techniques in addition to or as an alternative to computer vision techniques to identify marker points within the region of interest. For example, image analyzer 150 may deploy a marker point detection algorithm trained using a data corpus in which marker points (e.g., corners, edges, markers) of GET 125 are labeled. The marker point detection algorithm may be trained using normalized monochrome images (e.g., similar to imaging data provided by left and right image sensors of camera 128), color images (e.g., similar to imaging data provided by color image sensors of camera 128), disparity maps (e.g., similar to disparity generated based on imaging data provided by left and right image sensors of camera 128), LiDAR point cloud imaging data, and / or infrared imaging data, as some examples. The training data for the marker point detection algorithm may vary and correspond to the embodiment of sensors 126 and cameras 128 disposed on the embodiment of work machine 100.

[0048] Image analyzer 150 also maps image points in the imaging data across image data sources. For example, image analyzer 150 may determine image points associated with the end tip of GET 125 associated with imaging data captured by sensors 126 and 128. Image analyzer 150 may do this to determine errors in the imaging data that may be caused by debris or poor lighting conditions. For example, image analyzer 150 may ignore image point data from one set of imaging data captured by sensor 126 if it appears to be an outlier when compared to image point data from another set of imaging data captured by camera 128, or vice versa. In some embodiments, image analyzer 150 may compare image point data determined from recently captured imaging data with previous image point data and ignore image point data that does not match the previous image point data.

[0049] In some embodiments, pixel counts associated with sparse stereo disparity can be used to measure wear or loss of a GET. Pixel counts include, but are not limited to, area (e.g., total pixels of the GET), height of the GET in pixels, width of the GET in pixels, and the sum of height and width of the GET. The method for determining pixel count can vary depending on the shape and style of the GET. For example, for a GET whose length is much greater than its width, height pixel counts can be used, and for a GET whose width is much greater than its length, width pixel counts can be used. Various methods for determining pixel counts can be used without departing from the spirit and scope of this disclosure.

[0050] In some embodiments, the wear analyzer 153 can calculate a similarity score between the determined measurement data and expected data corresponding to an unworn GET 125. The similarity score can reflect a measure of how well the determined measurement data of the GET 125 matches the expected data for the physical parameter set. For example, the similarity score can include using a union intersection or Jaccard index method to detect similarity. In some embodiments, the similarity score can be determined using a Dice coefficient or F1 score method to detect similarity. The similarity score can also include a value reflecting the percentage of overlap of pixels in the sparse stereo disparity with the expected edge image. In some embodiments, the similarity score can be scaled or normalized from 0 to 100.

[0051] The similarity score can provide an indication of wear on the GET125. For example, a low score (e.g., in the range of 0 to 20) may indicate that one of the GET125 is broken or missing, indicating tooth loss. A high score (e.g., in the range of 80 to 100) may indicate good dental health and no need for replacement. Scores between low and high scores can indicate the level of tooth wear, with higher scores indicating a longer lead time for tooth replacement than lower scores.

[0052] In some embodiments, the wear analyzer 153 may collect measurement data about the GET 125 over time and use the collected measurement data to determine the wear level of the GET 125 and the wear trend of the GET 125. The wear analyzer 153 may store the collected measurement data in the GET wear level storage 157. For example, the work machine 100 may continue to operate at a work site for several days for a job. As the work machine 100 moves material during the job, the camera 128 provides stereo images of the bucket 120 and the GET 125 to the wear detection computer system 110, and the image analyzer 150 creates a sparse stereo disparity for the GET 125. The wear analyzer 153 may map the measurement data (e.g., pixel count, metric measurements, imperial measurements) associated with the GET 125 at several points over the duration of the job. As the bucket 120 and the GET 125 engage material at the work site, the GET 125 is expected to decrease in size due to wear. Thus, the measurement data associated with the GET 125 also decreases over time, and the pixel count over time reflects the wear trend. The wear trend at a particular point in time can be used by the wear analyzer 153 to determine the wear level of the GET 125 at that particular point in time. The wear level of the GET 125 can indicate the need for replacement of the GET 125 or the loss of one or more pieces of the GET 125. In some embodiments, the measurement data associated with the GET 125 can be stored in memory 143 and can apply to multiple jobs and multiple work sites, and the wear trend can apply to the life of the GET 125. In such embodiments, when the bucket 120 or the GET 125 is replaced, the pixel count associated with the GET 125 imaged by the wear analyzer 153 can be reset, and the wear analyzer 153 can resume collecting the pixel count of the GET 125 from time zero.

[0053] Because the wear analyzer 153 determines wear trends based on measurement data measured by the GET 125 over time, the wear analyzer 153 can also make predictions of when the GET 125 may need to be replaced. For example, if the wear analyzer 153 determines that measurement data associated with the GET 125 indicates that the GET 125 loses 1% of its life every 10 working hours (because the measurement data decreases by 1% every 10 working hours) and the GET 125 has been used for 800 working hours, the wear analyzer 153 can determine that the GET 125 needs to be replaced within 200 hours.

[0054] In some embodiments, the wear detection computer system 110 may include an alert manager 155. The alert manager 155 may communicate with the wear analyzer 153 and may monitor wear trends and wear levels identified by the wear analyzer 153. The alert manager 155 may provide a message alert to the operator control panel 130 based on information identified by the wear analyzer 153. For example, if the wear level reaches a wear threshold, the alert manager 155 may generate an alert shown on the display 133 of the operator control panel 130. The threshold may correspond to a value indicative of excessive GET wear, or possibly complete GET loss. The alert may indicate to the operator of the work machine 100 that one or more GETs 125 require replacement. The wear threshold may vary depending on the embodiment and may depend on the type of GET 125 and the worksite materials with which the GET 125 engages.

[0055] The alert manager 155 can also provide an alert that the GET 125 may need to be replaced at some point in the future, such as that the GET 125 needs to be replaced within two weeks. The replacement alert can include information regarding a predicted wear trend for the GET 125. For example, the replacement alert can include a quantification of the wear trend (e.g., the GET 125 is wearing 2% per day), the number of hours the tooth has been used, or the date or time that the GET 125 is expected to reach a wear threshold based on usage data.

[0056] In some embodiments, the warning manager 155 can monitor wear trends identified by the wear analyzer 153 and provide wear level values to the display 133 to notify the operator of the work machine 100 of the current wear level. For example, if the wear trend indicates that the GET 125 is 60% worn, the warning manager 155 can provide an indication that, based on the wear trend, the GET 125 has 40% of its life remaining before it will need to be replaced. The display 133 can also notify the operator of tooth breakage, indicating tooth loss (e.g., when one or more of the GETs 125 has less than 20% of its life remaining).

[0057] In some embodiments, the alert manager 155 may generate instructions that cause the wear level to be rendered on the display 133 indicating a measurement of the wear level or GET 125. For example, if the wear analyzer 153 determines, based on the processed imaging data, that one of the GETs 125 is currently 325 mm, the alert manager 155 may generate instructions that, when executed by the processor, cause the display 133 to indicate that the one GET is currently 325 mm.

[0058] In some embodiments, the wear detection computer system 110 can be in communication with a remote monitoring computer system 160. The remote monitoring computer system 160 can include a remote display 163, one or more remote processors 165, and a remote memory 170. The remote monitoring computer system 160 can be located at a work site where one or more work machines 100 operate and can communicate with relevant wear detection computer systems 110 at the work site. The remote monitoring computer system 160 can be configured to display GET wear levels for multiple work machines 100 and store GET wear information to facilitate monitoring of the health of GETs across a work site. For example, the remote display 163 can be configured to display user interfaces corresponding to one or more work machines 100 to display the wear level of each GET 125 in one location and facilitate monitoring. The remote monitoring computer system 160 can be implemented as a laptop, desktop computer system, or mobile device.

[0059] The remote monitoring computer system 160 includes one or more remote processors 165. The remote processor 165 may include one or more of a central processing unit (CPU), a graphics processing unit (GPU), a field programmable gate array (FPGA), some combination of a CPU, GPU, or FPGA, or any other type of processing unit. The remote processor 165 may have multiple arithmetic logic units (ALUs) that perform arithmetic and logical operations, and one or more control units (CUs) that retrieve instructions and memory contents from processor cache memory and then invoke the ALUs to execute instructions as needed during program execution. The remote processor 165 may also be responsible for executing drivers and other computer-executable instructions for applications, routines, or processes stored in remote memory 170, which may be associated with common types of volatile (RAM) and / or non-volatile (ROM) memory.

[0060] The remote monitoring computer system 160 also includes remote memory 170. The remote memory 170 can include system memory, which can be volatile (e.g., RAM), nonvolatile (e.g., ROM, flash memory, etc.), or any combination thereof. The remote memory 170 can also include non-transitory computer-readable media, such as volatile and non-volatile, removable and non-removable media, implemented by any method or technology for storage of information, such as computer-readable instructions, data structures, program modules, or other data. System memory, removable storage, and non-removable storage are examples of non-transitory computer-readable media. Examples of non-transitory computer-readable media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital multifunction disk (DVD) or other optical storage, cassette tape, magnetic tape, magnetic disk storage or other magnetic storage, or other non-transitory media used to store necessary information and which can be accessed by the remote monitoring computer system.

[0061] Remote memory 170 stores data, including computer-executable instructions, in remote monitoring computer system 160 for commanding and controlling remote display 163 and for implementing remote log manager 175. Remote memory 170 may also store additional components, modules, or other code executable by remote processor 165 to enable operation of remote monitoring computer system 160. For example, remote memory 170 may include code for input / output functions, software drivers, an operating system, or other components.

[0062] The remote monitoring computer system 160 generates monitoring outputs for a work site manager related to the health of the GETs and includes a remote display 163 that receives warnings and alerts related to the GET wear levels of one or more work machines 100 at the work site. The remote display 163 may include a liquid crystal display (LCD), a light emitting diode display (LED), a cathode ray tube (CRT) display, or other types of displays known in the art. In some examples, the remote display 163 includes an audio output, such as a speaker or a port for headphones or peripheral speakers. The remote display 163 may also include an audio input device, such as a microphone or a port for a peripheral microphone. In some embodiments, the remote display 133 includes a touch-sensitive display that also functions as an input device.

[0063] Similar to the display 133 (on the work machine 100), the remote display 163 of the remote monitoring computer system 160 may display information regarding the wear level or loss of the GET 125 rendered by the wear computer detection system 110 at the work site. For example, the remote display 163 may display calculated measurements of the GET 125 of one or more work machines 100. The calculated measurements, in some embodiments, may be color-coded to reflect the health of the GET 125. For example, the calculated measurements may be displayed against a green background if the GET 125 is deemed to have an acceptable level of wear, against a yellow background if the GET 125 is close to needing replacement, or in red if the GET 125 has broken or worn to the point where it requires replacement. The display 133 may also show images of the bucket 120, the GET 125, or an area of interest within the field of view 127, 129 associated with the GET 125, rendered by each wear computer detection system 110 of the work machines 100 at the work site.

[0064] Wear detection computer system 110 enables the operator of work machine 100 to be notified when GET 125 needs replacement due to extensive wear or is damaged. The processes employed by wear detection computer system 110 and described in more detail below provide highly accurate and precise measurements of GET wear on a sub-5 mm scale, allowing the operator to shut down operation of work machine 100 in the event of excessive GET wear or GET loss. The processes and techniques deployed by wear detection computer system 110 can be used with a variety of work machines.

[0065] For example, Figure 2 illustrates a schematic side view of an exemplary environment 200 in which a wheel loader work machine 201 operates. The wheel loader work machine 201 may include a bucket 220 and one or more buckets 225. As shown in Figure 2, the sensor 226 and camera 228 are positioned such that the GET 225 and bucket 220 are within a field of view 227 (of the sensor 226) and a field of view 229 (of the camera 228) during the dump end of a dig-dump cycle. As a result, in such an embodiment, the LiDAR sensor 226 and camera 228 may be configured to capture imaging data when the bucket 220 is stationary at the dump end of the dig-dump cycle.

[0066] 3 is a diagram illustrating a schematic side view of an exemplary environment 300 in which a hydraulic mining shovel work machine 301 is operating. The hydraulic mining shovel work machine 301 may include a bucket 320 and one or more buckets 325. In contrast to the location of the sensor 226 and the camera 228 of the wheel loader work machine 201, the sensor 326 and the camera 328 are positioned such that the GET 325 is within the field of view 327 (of the sensor 326) and the field of view 329 (of the camera 328) during the digging end of the dig-dump cycle. In such an embodiment, the sensor 326 and the camera 328 may be configured to capture imaging data while the bucket 320 is stopped at the digging end of the dig-dump cycle.

[0067] In yet another example, Figure 4 illustrates a schematic side view of an example environment 400 in which an electric rope shovel work machine 401 is operating. The electric rope shovel work machine 401 may include a bucket 420, one or more GETs 425, a sensor 426, and a camera 428. As shown in Figure 4, the GET 425 may be located within a field of view 427 (of the sensor 426) and a field of view 429 (of the camera 428) at a midpoint of a dig-dump cycle, but when the bucket 420 is relatively close to the sensor 426 and the camera 428. In such an embodiment, the sensor 426 and the camera 428 may be configured to capture imaging data when the bucket 420 enters a positional range associated with the field of view 427 and the field of view 429.

[0068] 2-4 illustrate only example positions of particular work machines and sensors / cameras, it should be noted that sensors 226, 326, 426 and cameras 228, 328, 428 may be positioned such that their respective fields of view 227, 327, 427, 229, 329, 429 capture image data at any point in the dig-dump cycle of work machines 201, 301, 401. Furthermore, the described positioning of sensors 226, 326, 426 and cameras 228, 328, 428 may be combined in some embodiments. For example, the present disclosure contemplates embodiments of a wheel loader work machine 201, a hydraulic mining shovel work machine 301, and an electric rope shovel work machine 401 having sensors and cameras with fields of view aimed at the beginning, middle, and / or end of the dig-dump cycle.

[0069] 5 shows an image data flow diagram 500 illustrating an example of imaging data flow for region of interest detection processing using computer vision techniques. Image data flow diagram 500 includes images received, processed, and generated by image analyzer 150 when a region of interest is detected in imaging data captured by camera 128 for GET 125. Image data flow diagram 500 includes left image 510 and right image 520 captured by camera 128. Left image 510 may be a rectified image captured by the left image sensor of camera 128. Right image 520 may be a rectified image captured by the right image sensor of camera 128. Both left image 510 and right image 520 include images of bucket 120 and GET 125.

[0070] Image analyzer 150 can process left image 510 and right image 520 to create a disparity map 530. Disparity map 530 may be a dense stereo disparity map that indicates the disparity between each pixel in left image 510 and each pixel in right image 520. Using disparity map 530 and a physical parameter set 535 obtained from physical parameter library 145 and associated with bucket 120, GET 125, and / or work machine 100, image analyzer 150 can construct a three-dimensional point cloud 540. 3D point cloud 540 indicates the disparity between left image 510 and right image 520 in three dimensions. Image analyzer 150 can then perform a segmentation analysis on 3D point cloud 540 to identify a region of interest 550 containing GET 125 in left image 510, right image 520, or both. In some embodiments, the image analyzer 150 can use the region of interest 550 to instruct and control the sensor 126 to capture higher resolution imaging data for the GET 125 .

[0071] 6 shows an image data flow diagram 600 illustrating an example of imaging data flow for region of interest detection processing using deep learning techniques. Similar to image data flow diagram 500 described above, the output of the region of interest detection processing is a region of interest 550 corresponding to GET 125 that image analyzer 150 uses to further analyze the image data. However, unlike image data flow diagram 500, image analyzer 150 utilizes deep learning techniques to detect region of interest 550 in image data flow diagram 600.

[0072] Image data flow diagram 600 includes an image 610 captured by camera 128. Image 610 may be a retouched image captured by either the left or right image sensor of camera 128, or may be image data captured from sensor 126. Image analyzer 150 may apply a deep learning GET detection algorithm to image 610. The deep learning GET detection algorithm may employ a neural network trained on an image data corpus in which GETs are individually identified and labeled and / or groups of GETs are individually identified and labeled. Upon applying the deep learning GET detection algorithm to image 610, image analyzer 150 may identify multiple individual GET bounding boxes 620 for the image that include individual GETs 125. In some embodiments, image analyzer 150 may also identify a GET group bounding box 630 that includes the individual GET bounding boxes 620. Once image analyzer 150 identifies a GET group bounding box 630, image analyzer 150 may extract pixels therein as regions of interest 550. 6 illustrates detection of region of interest 550 from image 610, which may represent image data captured by the left image sensor of camera 128, the right image sensor of camera 128, the color image sensor of camera 128, or sensor 126, and in some embodiments, image data flow diagram 600 may be applied to two or more of these at a particular time. For example, image analyzer 150 may detect a region of interest in imaging data from two or more of the left image sensor of camera 128, the right image sensor of camera 128, the color image sensor of camera 128, and / or image data captured by sensor 126 at a particular time.

[0073] 7 illustrates an image data flow diagram 700 showing an example of imaging data flow for region of interest detection processing using deep learning techniques. Similar to image data flow diagram 600 described above, the output of the region of interest detection processing is a region of interest 550 corresponding to a GET 125 that is used by image analyzer 150 to further analyze the image data. However, unlike image data flow diagram 500, the deep learning GET detection algorithm described with respect to FIG. 7 is trained using a corpus of data including disparity maps, and regions of interest are detected using disparity maps generated from two or more image sensors (e.g., left and right image sensors of camera 128).

[0074] Image data flow diagram 700 includes a first image 710 and a second image 720 captured by camera 128 or sensor 126. As just one example, first image 710 may be a rectified image captured by the left image sensor of camera 128, and second image 720 may be a rectified image captured by the right image sensor of camera 128, although first image 710 and second image 720 do not necessarily have to be received from camera 128. As another example, either first image 710 or second image 720 may be infrared image data captured from sensor 126. As yet another example, first image 710 may be captured by sensor 126, while second image 720 may be captured by camera 128, or vice versa. Both first image 710 and second image 720 include images of bucket 120 and GET 125. The image analyzer 150 may process the first image 710 and the second image 720 to create a disparity map 730. The disparity map 730 may be a dense stereo disparity map that indicates the disparity between each pixel of the first image 710 and each pixel of the second image 720.

[0075] The image analyzer 150 can apply a deep learning GET detection algorithm to the disparity map 730. The deep learning GET detection algorithm may employ a neural network trained on a corpus of image data in which GETs are individually identified and labeled, and / or on groups of GETs individually identified and labeled in the disparity map corresponding to the types of buckets 120 and GETs 125 captured in the first image 710 and the second image 720. Upon applying the deep learning GET detection algorithm to the disparity map 730, the image analyzer 150 can identify multiple individual GET bounding boxes 740 for the image that include the individual GETs 125. In some embodiments, the image analyzer 150 can also identify a GET group bounding box 750 that includes the individual GET bounding boxes 740. Once the image analyzer 150 identifies the GET group bounding box 750, the image analyzer 150 can extract pixels therein as a region of interest 550.

[0076] 8 illustrates an image data flow diagram 800 illustrating an example of imaging data flow for region of interest detection processing of imaging data captured by sensor 126. The following discussion of FIG. 8 uses a LiDAR embodiment of sensor 126 as an example, although other examples are contemplated. The imaging data captured by sensor 126 according to image data flow diagram 800 substantially corresponds to the field of view shown in image 810. As shown, the field of view includes bucket 120 and GET 125.

[0077] When sensor 126 detects an object surface, such as a surface corresponding to either bucket 120 or GET 125, sensor 126 captures LiDAR data including multiple LiDAR "hits." The LiDAR hits may be represented as a three-dimensional point cloud 820, with each point in the three-dimensional point cloud 820 corresponding to a LiDAR hit. Image analyzer 150 determines region of interest 510 based on the three-dimensional point cloud 820 by performing segmentation analysis or other object recognition analysis techniques. In some embodiments, image analyzer 150 can identify region of interest 550 using physical parameter set 535. For example, image analyzer 150 can identify region of interest 550 within the three-dimensional point cloud 820 using a bucket tooth template, a CAD-based model of GET 125, or pattern matching techniques.

[0078] 9 illustrates an image data flow diagram 900 showing an example of imaging data flow for wear detection processing using marker point identification. As described above, features of the GET 125 can be considered marker points, and the wear detection computer system 110 can determine measurements of the GET 125 based on identifying the marker points in the image data or regions of interest in the image data corresponding to the GET 125. Marker points can include, for example, edges, corners, or particular angles of the GET 125. While data flow diagram 900 illustrates data flow for identifying marker points using one region of interest (e.g., region of interest 910), in operation, the wear detection computer system 110 can process imaging data according to data flow diagram 900 for multiple sets of image data received from the sensor 126 and / or camera 128 at different time points.

[0079] Once the image analyzer 150 has identified the region of interest 910, it references the physical parameter set 535 to identify marker points 920 in the region of interest 910 that correspond to the GET 125 to assist in identifying marker points 920 in the region of interest 910. For example, the physical parameter set 535 may include the size or shape of the angle of each corner of the GET 125 that the image analyzer may use when performing a corner detection analysis on the region of interest 910. As another example, the physical parameter set 535 may include templates that match marker points (e.g., images of corners, marks, welds, or other identifiers) in the GET 125 that the image analyzer 150 applies in a segmentation analysis into the region of interest 910. In an embodiment in which the region of interest 910 corresponds to image data captured by an IR camera, the physical parameter set 535 may include an expected temperature value of the GET 125 and temperature information related to the work environment in which the GET 125 is used.

[0080] Once the image analyzer 150 has determined the marker points 920, it can identify and / or otherwise determine the GET ends 930. The GET ends 930 correspond to the length of the GET 125, with one end corresponding to the bucket-side edge of the GET 125 (e.g., where the GET 125 contacts the bucket 120) and the other end corresponding to the leading or engaging edge of the GET 125 (e.g., where the GET 125 engages the ground). Based on the determination of the GET ends 930, the wear analyzer 153 can determine GET measurements 940 for the GETs 125 that are within the region of interest 910. For example, the wear analyzer 153 may determine the number of pixels between the GET ends 930 for a particular GET 125 and then convert the number of pixels to a distance measurement (e.g., a metric or imperial real-world distance measurement).

[0081] FIG. 10 shows a flowchart depicting an exemplary wear detection process 1000 for detecting wear on the GET 125. In some embodiments, the process 1000 may be performed by the image analyzer 150 and the wear analyzer 153. The process 1000 generally follows the image data flow of FIGS. 5-9 and should be interpreted consistent with the descriptions of these figures and the description of the image analyzer 150 and the wear analyzer 153 described above with respect to FIG. 1. The following discussion describes aspects of the process 1000 performed by the image analyzer 150 or the wear analyzer 153, although other components of the wear detection computer system 110 may perform one or more blocks of the process 1000 without departing from the spirit and scope of the present disclosure.

[0082] Process 1000 begins at step 1010, where image analyzer 150 receives imaging data from multiple sensors associated with work machine 100. The multiple sensors may include, for example, left monochrome, right monochrome, or color image sensors of sensor 126 and camera 128. In some embodiments, each of the multiple sensors has a different field of view. For example, the left monochrome image sensor of camera 128 and the right monochrome image sensor of camera 128 may have slightly different fields of view to create the parallax required for stereo imaging. As another example, sensor 126 and camera 128 may have different orientations on work machine 100 to capture image data associated with GET 125 from different angles. Image analyzer 150 receives imaging data from each of the multiple sensors during a dig-dump cycle of the work machine, and image analyzer 150 may correlate the imaging data from each of the multiple sensors for analysis purposes. For example, for one iteration of process 1000, image analyzer 150 may receive first image data from one of the multiple sensors and second image data from another of the multiple sensors and process them together to determine the GET wear measurements for one dig-dump cycle.

[0083] In step 1020, the image analyzer 150 identifies each region of interest within the image data received from the multiple sensors in step 1010. For example, the image analyzer 150 may use computer vision techniques to identify a first region of interest within the first image data received from a first sensor of the multiple sensors and a second region of interest within the second image data received from a second sensor of the multiple sensors (see, for example, FIG. 5). The image analyzer 150 may use deep learning techniques to identify a first region of interest within the first image data received from a first sensor of the multiple sensors and a second region of interest within the second image data received from a second sensor of the multiple sensors (see, for example, FIGS. 6 and 7). If one of the multiple sensors is a LiDAR sensor, the image analyzer 150 may use a point cloud analysis method to determine the region of interest within the image data (see, for example, FIG. 8).

[0084] In step 1030, image analyzer 150 may also identify one or more image points within the region of interest (see, e.g., FIG. 9). In some examples, the image points identified by image analyzer 150 in step 1030 are associated with edges of GET 125, as described above with respect to FIGS. 1 and 9. In some embodiments, image analyzer 150 determines the image points using geometric parameters that describe GET 125. The geometric parameters may describe corners of GET 125 (e.g., the relative length of the leading edge to the side edges, the angle of the corners, the shape of the corners), or may describe other physical aspects of GET 125, such as the overall size, shape, or thickness.

[0085] In step 1040, the wear analyzer 153 determines a GET measurement based on the image points identified in step 1030. The wear analyzer 153 may correlate the number of pixels between the image points with the distance measurement (e.g., see FIG. 9 ) to determine the GET measurement. In some embodiments, the wear analyzer 153 may track and record the GET measurement in pixels. Based on the GET measurement determined in step 1040, the wear analyzer 153 determines the wear level or loss of the GET in step 1050. The wear level or loss can be quantified in real-world measurements (e.g., millimeters), in pixels, or as a percentage of the expected size (e.g., a CAD-based model of the GET 125). As described above, the wear analyzer 153 can use a CAD-based model of the GET 125 in an unworn state and compare it to the observed GET 125 measurements to determine the GET wear level or loss. The wear analyzer 153 can also use historical measurement data of the GET to determine wear level over time or to determine wear level trends to predict when the GET 125 needs to be replaced. In some embodiments, the wear analyzer 153 may be configured to determine loss when wear exceeds a threshold. For example, the wear analyzer may determine loss of a GET if its size decreases by 50% or more, or if it decreases by a certain measured amount (e.g., 5 cm in length). The wear analyzer 153 may issue an alert when wear on a GET reaches or exceeds a threshold.

[0086] In some embodiments, the image analyzer 150 and the wear analyzer 153 execute the process 1000 several times within a single dig-dump cycle. In such embodiments, the wear analyzer 153 may compare the GET measurement determined in the current dig-dump cycle (step 1040) with one or more GET measurements determined previously (or later) within the same dig-dump cycle, or with historical measurements, to determine whether the currently determined GET measurement matches. If the measurements are inconsistent (e.g., differ from other GET measurements within the same dig-dump cycle by some threshold), the wear analyzer 153 may discard the current measurement as noise or erroneous data. The threshold may be configured based on the environment, the work machine, or the type of GET. For example, the threshold may be set low (e.g., less than 10%) for a work machine excavating soft materials with long GETs, and higher (e.g., greater than 20%) for a work machine excavating hard materials with short GETs. In some embodiments, the image analyzer 150 and the wear analyzer 153 may also consider the point in time within the dig-dump cycle when performing the process 1000 to determine whether the determined GET measurement is noise.

[0087] FIG. 11 shows a flowchart depicting an exemplary wear detection process 1100 for detecting wear on the GET 125. In some embodiments, the process 1100 may be performed by the image analyzer 150 and the wear analyzer 153. The process 1100 generally follows the image data flow of FIGS. 5-9 and should be interpreted consistent with the descriptions of these figures, as well as the description of the image analyzer 150 and the wear analyzer 153 described above with respect to FIG. 1, and the description of the wear detection process 1000. The following discussion describes aspects of the process 1100 performed by the image analyzer 150 or the wear analyzer 153, although other components of the wear detection computer system 110 may perform one or more blocks of the process 1100 without departing from the spirit and scope of the present disclosure.

[0088] Process 1100 may be performed within one dig-dump cycle of work machine 100 or within / across multiple dig-dump cycles of work machine 100. In some embodiments, image analyzer 150 and wear analyzer 153 perform process 1100 several times within a single dig-dump cycle.

[0089] After the start of the dig-dump cycle (block 1105), in step 1110, the image analyzer 150 receives first image data from the sensor 126 or camera 128. After processing the captured first image data (using process 1000 just as one non-limiting example), the wear analyzer 153 determines a first wear measurement for the GET 125 in block 1115 and then determines a first wear level for the GET 125. The wear analyzer 153 may determine the first wear measurement and first wear level using the techniques described above with respect to Figures 1, 9, and 10, just as some examples.

[0090] After the wear analyzer 153 determines the first wear level of the GET 125, the wear analyzer 153 determines whether the first wear level indicates a GET replacement condition, e.g., whether the wear analyzer 153 should generate a warning notifying the operator of the work machine 100 that the GET needs to be replaced. Determining whether the first wear level indicates a GET replacement condition may include two determinations in some embodiments. One determination is whether the wear level indicates the need for a warning (step 1127). If the wear level does not indicate the need for a warning (step 1127: NO), the process 1100 ends and may be repeated at another point in the same dig-dump cycle (block 1195) consistent with embodiments of the present disclosure. However, if the wear level indicates the need for a warning (step 1127: YES), the process proceeds to another determination (step 1129) to determine whether the first image data was captured later in the dig-dump cycle. In some embodiments, the other determination may be based on a “late point” in the dig-dump cycle. If the first image data is received after the delay point, the wear analyzer 153 may determine that the dig-dump cycle is late (step 1129: YES) and process 1100 proceeds to step 1135, where the wear analyzer 153 generates a warning. If the first image data is received before the delay point, the wear analyzer 153 may determine that the dig-dump cycle is not late (step 1129: NO) and process 1100 proceeds to step 1140. In some embodiments, the delay point may be set by the operator of the work machine 100, or the delay point may have a default setting. For example, the default setting is the midpoint or 50% point of the dig-dump cycle. Image data captured near the end of the dig-dump cycle will be captured later in the cycle, while image data captured near the beginning of the dig-dump cycle will be captured earlier in the dig-dump cycle (not late).

[0091] If the wear analyzer 153 determines that the first GET wear level indicates the need for an alert and did not receive first image data later in the dig-dump cycle, a second set of image data for the GET 125 is collected to confirm the need for an alert. In step 1140, the image analyzer 150 receives second image data from the sensor 126 or camera 128. After processing the second image data (using process 1000 as just one non-limiting example), the wear analyzer 153 determines a second wear measurement for the GET 125 in block 1145 and then determines a second wear level for the GET 125 in a manner similar to how it determined the first wear level for the GET 125 described above (step 1150). The wear analyzer 153 then determines whether a GET replacement condition is met. If both the first wear level and the second wear level indicate the need for an alert (step 1155: YES), the wear analyzer generates an alert in step 1135. However, if the second wear level does not indicate the need for an alert (step 1155: NO), the GET replacement condition is not met and the process 1100 ends for the current dig-dump cycle (block 1195).

[0092] In some embodiments, the wear analyzer 153 may log the first GET wear level or the second GET wear level (if calculated) for each iteration of the process 1100. Logging may include storing the GET wear measurements, the GET wear levels, and / or storing the captured image data. The wear analyzer 153 may record this information in the GET wear level storage 157 at the wear detection computer system 110 or provide it to the remote monitoring computer system for storage in the remote log manager 175.

[0093] FIG. 12 illustrates an example wear detection user interface 1200 that may be rendered on the display 133 or remote display 163. The user interface 1200 may include a captured image 1210 of the bucket 120 and the GET 125. The captured image 1210 may, in some embodiments, be a still image or a real-time video image of the bucket 120 and the GET 125. The user interface 1200 may also include a GET 125 status user interface element 1220 that displays the current status of the GET 125. The status user interface element 1220 may display, for example, a current measurement value of the GET 125 (as shown in FIG. 12 ), a wear rate of the GET 125, or both. In some embodiments, the status user interface may include an indicator of GET wear, such as color coding, to provide information to the operator of the work machine 100. For example, if the GET 125 is in good condition without significant wear, the status user interface element 1220 may be rendered with a health indicator 1230. Health indicator 1230 may be color-coded (e.g., green) to indicate that no action needs to be taken. As another example, if GET 125 is partially worn and close to needing replacement, status user interface element 1220 may render with a partial wear indicator 1240. Partial wear indicator 1240 may be color-coded (e.g., yellow), in a special font, or highlighted within user interface 1200. As another example, if GET 125 is worn to the point where replacement is required or is broken, status user interface element 1220 may render with a full wear indicator 1250. Full wear indicator 1250 may be highlighted within user interface 1200 or color-coded with a special font (e.g., red) to consistently flash and alert the operator.

[0094] The user interface 1200 may also include a close-up view 1260 of the GET 125. The close-up view 1260 may correspond to an identified region of interest within the image data, for example. In some embodiments, in addition to the full wear indicator 1250, the user interface 1200 may include a caution notice 1270 to alert the operator to the GET replacement condition. The partial wear indicator 1240, the full wear indicator 1250, and / or the caution notice 1270 may also be accompanied by an audio prompt or alarm.

[0095] Throughout the above description, certain components of the wear detection computer system 110 are described as performing certain operations. However, in some embodiments of the wear detection computer system 110, other components may perform these operations. Furthermore, the wear detection computer system 110 may include additional or fewer components than those described above in the example embodiments. Those skilled in the art will understand that the wear detection computer system 110 need not be limited to the specific embodiments disclosed above. [Industrial Applicability]

[0096] The disclosed systems and methods can be used in connection with the operation of work machines at a work site that excavate, move, shape, contour, and / or remove materials, such as soil, rock, and minerals. These work machines may be equipped with buckets used to scoop, dig, and dump materials at the work site. The buckets may include one or more buckets to help loosen the material during operation. The work machines may also include a system having a processor and memory configured to execute a wear detection method according to the examples described herein. The systems and methods can detect wear or loss in work machine components, such as GETs, so that operators of such work machines can take corrective action before a failure occurs that could damage downstream processing equipment.

[0097] In some examples, the systems and methods may capture imaging data associated with the GET from one or more sensors on the work machine and process the imaging data to determine wear or loss of the GET. The one or more sensors may include a LiDAR sensor, an image sensor, and / or a stereo camera.

[0098] In some examples, systems and methods use image data from multiple sensors to more accurately determine GET wear. The systems and methods detect image points in the image data and correlate GET measurements across multiple image sensors. The use of image points can accurately simplify processing across different types of sensors. Additionally, the use of image points can reduce the computational complexity of GET wear measurements by reducing the amount of data required to determine measurements. The processes described in this disclosure provide highly accurate measurements of GET while providing processing efficiency.

[0099] While aspects of the present disclosure have been particularly shown and described with reference to the above examples, it will be understood by those skilled in the art that various additional embodiments may be contemplated by modifying the apparatus, systems, and methods of the present disclosure without departing from the spirit and scope of the present disclosure. Such embodiments should be understood to be within the scope of the present disclosure as determined by the claims and any equivalents thereof.

Claims

1. receiving a plurality of images from a plurality of sensors (126, 128) associated with a work machine (100), the plurality of images including an image of at least one ground engaging tool (GET) (125) of the work machine, the at least one GET being included in a bucket, each sensor of the plurality of sensors having a respective field of view (127, 129) that is different from other sensors of the plurality of sensors, and capturing at least one image when the bucket enters a position range associated with its respective field of view; identifying a first region of interest (550) associated with the at least one GET and included in a first image of the plurality of images, the first region of interest including first data characterizing the at least one GET; identifying a second region of interest (550) associated with the at least one GET and included in a second image of the plurality of images, the second region of interest including second data characterizing the at least one GET; determining a first set of image points (920) for the at least one GET, the first set of image points (920) being determined based at least in part on geometric parameters (535) associated with the at least one GET and a first edge of the at least one GET detected in the first data; determining a second set of image points (920) for the at least one GET, the second set being determined based at least in part on the geometric parameters (535) associated with the at least one GET and a second edge of the at least one GET detected in the second data; determining a GET measurement (940) for the at least one GET based at least in part on the first set of image points and the second set of image points; determining a wear level or loss of the at least one GET based on the GET measurements; 1. A computer-implemented method comprising:

2. the plurality of sensors include a left image sensor and a right image sensor of a stereoscopic camera; The plurality of images are a left image (510) of the at least one GET captured by the left image sensor; a right image (520) of the at least one GET captured by the right image sensor.

3. The computer-implemented method of claim 2 , wherein identifying the first region of interest includes generating a disparity map (530) based on the left image and the right image.

4. 2. The computer-implemented method of claim 1, wherein identifying the first region of interest comprises applying a deep learning GET detection algorithm to the first image, or wherein identifying the second region of interest comprises applying a deep learning GET detection algorithm to the second image.

5. The computer-implemented method of claim 1 , further comprising generating an alert when the wear level or loss indicates a GET replacement condition.

6. a plurality of sensors (126, 128) associated with the work machine (100), each sensor of the plurality of sensors having a respective field of view (127, 129) distinct from other sensors of the plurality of sensors, the plurality of sensors capturing a plurality of images of at least one ground engaging tool (GET) (125) of the work machine when a bucket of the work machine enters a position range associated with its respective field of view; one or more processors (140); a non-transitory computer-readable medium (143) storing executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including: The operation is identifying a first region of interest (550) associated with the at least one GET and included in a first image of the plurality of images, the first region of interest including first data characterizing the at least one GET; identifying a second region of interest (550) associated with the at least one GET and included in a second image of the plurality of images, the second region of interest including second data characterizing the at least one GET; determining a first set of image points (920) for the at least one GET, the first set of image points (920) being determined based at least in part on geometric parameters (535) associated with the at least one GET and a first edge of the at least one GET detected in the first data; determining a second set of image points (920) for the at least one GET, the second set being determined based at least in part on the geometric parameters associated with the at least one GET and a second edge of the at least one GET detected in the second data; determining a GET measurement (940) for the at least one GET based at least in part on the first set of image points and the second set of image points; determining a wear level or loss of the at least one GET based on the GET measurements; Including, the system.

7. the plurality of sensors include a left image sensor and a right image sensor of a stereoscopic camera; The plurality of images are a left image (510) of the at least one GET captured by the left image sensor; a right image (520) of the at least one GET captured by the right image sensor.

8. The system of claim 7 , wherein identifying the first region of interest includes generating a dense stereo disparity map (530) based on the left image and the right image.

9. 7. The system of claim 6, wherein identifying the first region of interest comprises applying a deep learning GET detection algorithm to the first image, or wherein identifying the second region of interest comprises applying a deep learning GET detection algorithm to the second image.

10. The system of claim 6 , further comprising generating an alert when the wear level or loss indicates a GET replacement condition.

Images