Systems and methods for identifying and locating missing or damaged ground engagement tools

Abstract

A method for monitoring and confirming damaged or missing ground engaging tools (212) at a construction site includes receiving an alert associated with a damaged or missing ground engaging tool (212) of an earthmoving machine (102, 104), receiving confirmation for the damaged or missing tool (212), and generating a containment area (310, 320) representing one or more locations where at least a portion of the ground engaging tool (212) may be located.The method also includes determining that an additional machine (102, 104, 106, 108) was present in the containment area, and taking an action for the additional machine (102, 104, 106, 108) based on the presence of the machine (102, 104, 106, 108) in the containment area (310, 320), wherein the action includes generating a notification or controlling an action of the additional machine (102, 104, 106, 108).

Description

Technical area

[0001] This disclosure relates generally to monitoring one or more aspects of a construction site, and more particularly to systems and methods for monitoring ground engaging tools associated with machines operating at a construction site. State of the art

[0002] Construction sites, such as mines, operate most efficiently when multiple machines perform work functions simultaneously. In surface and underground mining environments, machine functions include digging, scraping, and lifting material, placing the material in a bucket, loading the material into an off-highway truck, transporting the material with the off-highway truck, and processing the material. In some types of mines, large pieces of material may be processed using a crusher, which breaks the material into finer fragments or particles.

[0003] To facilitate material gathering and reduce maintenance costs, earthmoving equipment such as mining buckets, loaders, and others incorporate work tools equipped with ground-engaging tools (GETs). These GETs are constructed of extremely hard and wear-resistant materials. Although GET failures resulting in partial or complete separation from the machine are rare, these failures can result in significant costs and downtime. For example, a detached GET can damage a crusher on the construction site, significantly impacting overall productivity and potentially requiring costly repairs.

[0004] While monitoring systems can help prevent these rare but serious damages to processing machines, these monitoring systems come with other disadvantages. For example, some systems are associated with frequent false alarms, which can lead to repeated, unnecessary shutdowns of a site. These false alarms, in turn, can lead to lost productivity. Frequent false alarms are typically ignored by operators, thus missing the benefits of the monitoring devices. Additionally, while monitoring systems can identify a missing or damaged GET on one machine, they may not consider the entire mine operation when confirming and locating the missing or damaged GET.

[0005] An exemplary system for monitoring a mining bucket tooth or adapter is described in U.S. Patent Application No. 2021 / 0262204 A1 ("the '204 publication") by Tafazoli Bilandi et al. The monitoring system described in the '204 publication uses image analysis using a neural network to determine when a work tool tooth is missing. The monitoring system can identify a missing tooth by checking whether visually identified teeth are evenly spaced and comparing the number of identified teeth to the expected number of teeth on the work tool. Although the system described in the '204 publication may be useful for identifying a missing tooth, it does not address the potential for false positives or unnecessary downtime associated with false positives or other detection errors.

[0006] The techniques of this disclosure may solve one or more of the problems set forth above and / or other problems of the prior art. However, the scope of the current disclosure is defined by the appended claims, rather than by the ability to solve any specific problem. Brief description

[0007] In one aspect, a method for monitoring and acknowledging damaged or missing ground engaging tools at a jobsite may include receiving an alert associated with a damaged or missing ground engaging tool of an earthmoving machine, receiving acknowledgment for the damaged or missing tool, and generating a containment area representing one or more locations where at least a portion of the ground engaging tool may be located. The method may also include determining that an additional machine was present in the containment area and taking an action for the additional machine based on the presence of the machine in the containment area, wherein the action comprises generating a notification or controlling an action of the additional machine.

[0008] In another aspect, a method for reducing false alarms associated with damaged or missing tools may include receiving an alert associated with a potentially damaged or missing tool of an earthmoving machine, requesting acknowledgment of the potentially damaged or missing tool, and continuing operation of the earthmoving machine after receiving the alert. The method may also include receiving acknowledgment of the damaged or missing tool and generating a signal to disconnect or stop the earthmoving machine or a connected machine, or receiving acknowledgment that the potentially damaged or missing tool is not damaged or missing and allowing the earthmoving machine or connected machine to continue operating.

[0009] In yet another aspect, a monitoring controller for monitoring and acknowledging a missing or damaged tool for an earthmoving machine may include at least one memory storing instructions and at least one processor operably connected to the memory and configured to execute the instructions to perform operations. The operations may include receiving an alert associated with a damaged or missing tool of a machine, in response to receiving the alert, receiving machine location information associated with the machine, generating a request to acknowledge the missing or damaged tool, and generating a containment area based on the machine location information and receiving an acknowledgement of the missing or damaged tool.

[0010] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosed embodiments as claimed. Short description of the drawings

[0011] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate various exemplary embodiments and, together with the description, serve to explain the principles of the disclosure. Fig. 1 is a schematic view of an exemplary environment in which a ground engaging tool monitoring system may be used in accordance with aspects of this disclosure. Fig. 2 is a block diagram illustrating exemplary components of the ground engaging tool monitoring system of Fig. 1 represents. Fig. 3 is a view illustrating an exemplary view of a displayed image for locating a missing or corrupted GET. Fig. 4 illustrates a flow diagram of an exemplary method for monitoring ground engaging tools according to aspects of the disclosure. Fig. Figure 5 is a block diagram illustrating an example of a computing device. Detailed description

[0012] Both the foregoing general description and the following detailed description are exemplary and explanatory only and do not limit the features as claimed. As used herein, the terms "comprises," "comprising," "having," "including," or other variations thereof are intended to cover non-exclusive inclusion such that a method or apparatus comprising a list of elements may include not only those elements, but may also include other elements not expressly listed or inherent in such method or apparatus. In this disclosure, relative terms such as "approximately," "substantially," "generally," or "about" are used to indicate a possible deviation of ±10% from the stated value or feature.

[0013] Fig. 1 is a schematic diagram illustrating an environment 110 in which a ground-engaging tool monitoring system 100 may identify and confirm missing or damaged ground-engaging tools. System 100 may be configured to enable continued operations on the jobsite and avoid false alarms, while simultaneously taking action to contain machines that have been operating in an area (e.g., a "containment area," as described below) where a ground-engaging tool may have been damaged or lost. System 100 may include a plurality of machines, a vision system, including vision devices 112 distributed among a plurality of machines and / or at other locations within environment 110.System 100 may also include a supervisory controller 160 and a communications system that enables communication between the machines and the supervisory controller 160. If desired, the communications system may also enable monitoring of these machines with a remote monitoring system 180 that communicates with the machines and / or the supervisory controller 160.

[0014] The machines of system 100 may include one or more earthmoving machines 102 and 104, such as a mining shovel (earthmoving machine 102), a loader (earthmoving machine 104), or other types of work-performing machines. System 100 may include one or more transport machines 106, for example, off-road trucks. If environment 110 is located on or includes a mining construction site, the machines of system 100 may include one or more processing machines 108, such as a conveyor and / or a crusher, configured to crush the raw materials delivered via machines 102, 104, and 106. While machines 102, 104, 106, and 108 are described herein as being located in a mining environment 110 (e.g.,While illustrated and described as being used in a variety of applications (e.g., open-pit mining), it should be understood that these machines and other components of system 100 may also include other types of machines and / or perform tasks on a different type of construction site (e.g., paving, construction, forestry, etc.). If necessary, one or more drones 130 or other systems may assist in monitoring the ground intervention tools.

[0015] Earthmoving machines 102 and 104 may include a ground-engaging work tool 118, including a bucket, ripper, blade, scraper, etc., movable with one or more work tool linkages 116. Linkage 116 may include, for example, a hydraulically movable boom and / or stick. A ground-engaging tool (GET) 120 connected to the work tool 118. Each work tool 118 may be equipped with a plurality of GETs 120 or a single GET 120. GET 120 may include teeth and / or adapters, as well as other components that may be attached to a work tool 118, such as protective devices for the sides or edges of the work tool 118, including lip guards, side guards, and others.As used herein, the term "missing GET" is interchangeable with "missing or damaged GET" and includes ground engaging tools that have been damaged but not significantly separated from a work tool, tools that have been completely separated from a work tool (which may or may not be damaged), and tools that have been partially separated from a work tool. Machines 102 and 104 may also include one or more machine vision devices 112, communication or networking devices 114, a machine controller 122, and an inertial measurement unit (IMU) 124, as described below.

[0016] Transport machines 106 may each include a loading platform configured to receive and transport the material 126 fed by machines 102 and 104. Transport machines 106 may be configured for manual (local or remote) control, fully autonomous control, or semi-autonomous control. Transport machines 106 may include one or more vision devices 112 and one or more network devices 114 to enable transport machines 106 to identify or confirm a missing or damaged GET, locate the GET, and determine when machine 106 has entered a containment area, as described below. Each transport machine 106 may include a machine controller 122 configured to communicate with other machine controllers 122 and supervisory controller 160.The transport machine 106, when used in a mining environment 110, may have a capacity suitable for receiving separated material 126 (e.g., rock, ore, or other raw materials) from the material source 150.

[0017] Processing machine 108 may be configured to receive material 126 from machines 106. This received material 140 may, as one example, be processed by crushing the material with machine 108. Thus, machine 108 may be a crusher that includes material crushing components that exert large forces on material 140. In some aspects, this material crushing machine may be susceptible to damage if a GET is present in material 140. Processing machine 108 may include an image processing device 112, a network device 114, and a machine controller 122. Like machines 102, 104, and 106, machine 108 may be manually controlled, fully autonomously controlled, or semi-autonomously controlled. For example, when machine 108 is within a containment area (described below), machine 108 may cease operation autonomously or automatically.In contrast, one or more other machines 102, 104, 106 of the environment 110 may continue to perform at least some functions when they are within a containment area.

[0018] The machines of system 100, including machines 102, 104, 106, and 108, may be configured to collectively identify, confirm, and locate a missing GET 120 and establish containment areas where work processes can be modified until a missing GET 120 is located. An image processing system may include image processing devices 112 present on a plurality of these machines. If desired, image processing devices 112 may also be present at locations external to machines 102, 104, 106, and 108, such as at stationary locations within environment 110 and / or on a mobile land-based or airborne drone 130.

[0019] Supervisory controller 160 may include one or more systems located within environment 110. In particular, supervisory controller 160 may be located at a construction site, such as a mine. Supervisory controller 160 may be configured to communicate with one or more of machines 102, 104, 106, and 108 via network 170 or another communication method. Supervisory controller 160 may enable one or more users (e.g., supervisors) to monitor progress on the construction site, monitor current and past geographic locations and actions (e.g., lifting, tipping, transporting, etc.) of one or more machines, monitor past, present, and future tasks of the machines, monitor the status of the machines, and monitor other types of information in real-time or near-real-time.If a missing GET is detected, the supervisory controller 160 may enable a visual indication of the work device 118 with the missing GET, a visual indication of a request to confirm that the GET is missing, and / or a display of a containment area, which may include a probability map (or heat map) or other form of notification for illustrating possible locations of the missing GET, as well as the identity and / or locations of machines that should be isolated after the completion of one or more cycles or immediately. Supervisory controller 160 may be configured to receive information from one or more machines 102, 104, 106, and 108 in response to receiving a notification of a potentially missing GET 120. For example, in response to receiving this notification, the supervisory controller 160 may begin displaying visual data (e.g.,including visual information for confirming a missing GET 120), motion data (e.g., forces experienced by a machine), location data (e.g., a geographic location), operation data (e.g., an operation count), and other data transmitted from machines 102, 104, 106, and 108. If necessary, at least some of this information may also be transmitted from a monitoring system such as one or more drones 130. In some aspects, receiving and / or storing information in response to identifying a damaged or missing GET may conserve computing resources, including memory, network capacity, and processing capability.

[0020] Supervisory controller 160 may be configured to transmit a notification, such as a warning, to a supervisor, a machine operator, or another user to draw attention to a potentially missing GET 120. Supervisory controller 160 may further be configured to request acknowledgment of the potentially missing GET 120. This acknowledgment request may be presented on supervisory controller 160 itself, on displays of machines 102, 104, 106, and 108 (e.g., via machine controller 122), on personal computing devices associated with the machine operators, and / or on remote monitoring system 180.

[0021] In configurations where one or more of machines 102, 104, 106, and 108 can operate semi-autonomously or fully autonomously, supervisory controller 160 can be configured to suspend operation or initiate one or more desired actions for a machine associated with a missing GET 120, including machines other than the machine with the missing GET 120. These actions can be determined based on the machine or machines that were or are within a containment area. Supervisory controller 160 can be further configured to establish the containment area, including the location and size of the area, based on the likelihood that a missing GET 120 is located at a particular location.

[0022] The machine vision system of system 100 may include a plurality of image processing devices 112. Devices 112 may be configured to capture images that provide range information (e.g., stereo cameras) and may include infrared cameras, optical cameras, or a variety of these types of cameras. Image processing devices 112 are not limited to human-visible light or specifically to images or videos, but may also include devices such as laser-based systems (LIDAR) or other types of devices that enable a controller to assess terrain or material, measure the expected location of GETs, etc.

[0023] A first subset of image processing devices 112 may be positioned to enable initial identification of a missing or damaged GET. For example, image processing devices 112 present on earthmoving machines 102 and 104 may be directed toward work device 118. In the Fig. 1, vision devices 112 for machines 102 and 104 are present on respective work tool linkages 116. In some aspects, these vision devices 112 may be secured to a boom of the machine and directed toward the missing or damaged GET 120 of the work tool 118. It should be understood that at certain times, these vision devices 112 may have an unobstructed view of the missing or damaged GET 120, while at other times, they may be unable to view the missing or damaged GET 120 due to the material present in the work tool 118 or the position of the work tool 118. Other vision devices 112 may be positioned to enable confirmation of a missing or damaged GET and / or to locate a GET that has become disconnected from the machines 102 or 104.These imaging devices 112 may include devices 112 on transport machines 106, wherein these devices 112 are directed toward a loading area of ​​the corresponding transport machine 106. If necessary, machine 108 may include an imaging device 112 directed toward the material 140.

[0024] A second subset of vision devices 112 on machines 106 and 108 may be positioned toward machines 102 and 104 to enable confirmation of a missing or damaged GET 120. In some aspects, one or more of the first subset of devices 112 (e.g., on machine 104) may function as part of the second subset. This may occur, for example, when supervisory controller 160 generates a command that causes machine 104 to move autonomously within environment 110 such that vision device 112 on machine 104 faces a location associated with a missing GET 120, such as work device 118 of machine 102. This may be useful for confirming that GET 120 is missing or for locating the missing GET 120 if it is near machine 102.Additionally or alternatively, supervisory control 160 may generate a notification for an operator to manually move the machine to enable vision devices 112 to confirm the absence of a GET 120 or potentially locate the missing GET 120. Devices 112 on machines 106 and 108 may be positioned such that a line of sight extends toward the locations of the environment 110 proximate the earthmoving machine and / or the material source 150 on which the machine was working. These locations of the environment 110 may be determined based on one or more likely locations of a missing GET 120 (e.g., within a containment area), as determined with supervisory control 160, as described below.

[0025] A communication system of system 100 may include a plurality of network devices 114. As in Fig. 1, each machine 102, 104, 106, and 108 may be equipped with a corresponding network device 114. Network devices 114 may enable communication over one or more networks 170, where network 170 is a local area network for machines 102, 104, 106, 108, drone 130, and supervisory controller 160 that is not accessible from locations outside a worksite. Alternatively, network 170 may be accessible from one or more remote systems 180 and / or other systems remote from environment 110 by enabling communication, for example, via the Internet, over network 170. In particular, network 170 may comprise a wide area network ("WAN"), a local area network ("LAN"), a personal area network ("PAN"), or the like.If network 170 is a local area network, one or more operators or supervisors located on-site in environment 110 may receive and transmit information with machines 102, 104, 106, 108 and supervisory controller 160 using a personal computing system (e.g., a stationary computing system, a mobile computing system, a mobile device, a tablet, etc. with a display). If network 170 is remotely accessible, remote systems, including remote monitoring systems 180 and / or personal computing systems, may also be configured to receive and transmit information with machines 102, 104, 106, 108 and supervisory controller 160.

[0026] Fig. 2 is a block diagram illustrating communication, data sharing, and, if necessary, transmission of commands between supervisory control 160, machines 102, 104, 106, 108, remote monitoring system 180, and, if necessary, one or more drones 130. As in Fig. 2, supervisory controller 160 may receive as inputs information from machine controllers 122 of one or more machines 102, 104, 106, and 108. In some aspects, at least some of the information received by supervisory controller 160 may be collected and / or analyzed in response to the initial identification of a potentially missing or damaged GET 120. However, under normal operating conditions, when no missing GET 120 has been identified, some or all of the information may be received by supervisory controller 160.

[0027] Supervisory controller 160 may include one or more modules (e.g., memory programmed therewith) that enable supervisory controller 160 to perform the functions described herein, including the operations described below with respect to method 400. In particular, supervisory controller 160 may include a GET monitor 220 and a machine learning (ML) model 230.

[0028] GET monitor 220 may monitor or track GETs 120 of a plurality of machines using visual and other data. GET monitor 220 may enable functions including identifying potentially missing or corrupted GETs 120, confirming a missing or corrupted GET 120, defining a containment area solely or based on outputs of machine learning model 230, controlling one or more machines that are or were present in the containment area, and generating notifications for confirming a missing GET 120, as described herein.GET monitor 220 may receive any of the inputs transmitted from machines 102, 104, 106, and 108, drone 130 to supervisory controller 160 and generate outputs that cause notifications to be displayed with machines 102, 104, 106, and 108, with supervisory controller 160 itself, with remote monitoring system 180, or with computer systems associated with operators or supervisors associated with machines 102, 104, 106, and 108.

[0029] Machine learning (ML) model 230 may include a trained machine learning model configured to identify and output one or more possible locations for a missing GET 120. In some aspects, ML model 230 may be configured to output a confidence level or confirmation for a missing or damaged GET 120 in response to initially identifying a missing or damaged GET 120. ML model 230 may receive as inputs one or more of visual data 210, motion data 214, location data 216, and operation counts 218 from machine 102 or other machines 104, 106, 108, or drone 130.In some aspects, model 230 may be configured to output one or more possible locations of a missing GET 120 to monitor 220 for use in defining containment area(s) useful for generating notifications and / or controlling the operation of machines 102, 104, 106, and 108 that were present within the containment area(s).

[0030] Visual data 210 may, for example, be generated with image processing devices 112 and transmitted with network devices 114. Visual data 210 may include real-time videos and images or time-shifted videos or images. As shown in Fig. As shown in Figure 2, an image or video from image processing devices 112 may provide an image including an image of the potentially missing or damaged GET 212. This image 212 may be a thermal image, a stereo image, etc.

[0031] Motion data 214 may include force and / or motion information acquired with IMU 124 of machine 102. Motion data 214 may include information associated with one or more rods of a work tool system of earthmoving machine 102 and earthmoving machine 104. This information may be useful for determining, with model 230 or another component of supervisory control 160, one or more actions performed by machine 102. These actions may be associated with tasks that have a relatively higher probability of damage to or loss of GET 120, to generate one or more containment areas and / or create a probability map for the containment areas. For example, ML model 230 may be configured to use training data to determine events (lifting, dumping, etc.).) associated with a relatively high probability of damage to or loss of GET 120, thereby enabling GET monitor 220 to receive the determined events and generate corresponding containment areas. In other examples, motion data 214 may enable GET monitor 220 to generate a containment area based on forces measured by IMU 124 that exceed a predetermined threshold. In some aspects, monitor 220 may assign a higher probability of finding a missing GET 120 based on increasing forces or impacts measured by motion data 214 from IMU 124.

[0032] Location data 216 may include information about the geographic location of machine 102 generated using a global navigation satellite system (GNSS), such as a global positioning system receiver (not shown), an IMU 124 with a gyroscope for determining orientation, or other device. Location data 216 may be useful for identifying a location of machine 102 when a missing GET 120 is partially or completely separated from work tool 118. Location data 216 may also enable the use of historical location data to identify previous locations where GET 120 may have been lost or damaged. Location data 216 may include a geographic location (e.g., a position of the machine on the jobsite), a heading of the machine, or other information.

[0033] Operation count 218 may be generated with an operation counter or other suitable module of the machine controller 122. The counter 218 may monitor a number of tasks performed by the machine 102. For example, the earthmoving machine 102 may be configured to drive the work tool 118 into the material source 150 and to draw material from the source 150 ( Fig. 1) onto a bed of haulage machine 106 to complete a single excavation operation. A operation may include a series of task steps performed by earthmoving machine 102, including gathering material from material source 150 and depositing the material onto a bed of haulage machine 106. Each time material is introduced into a particular machine 106, the operation count 218 may increment. Count 218 may be compared to a total number of operations required to substantially fill the bed of haulage machine 106 without overloading.

[0034] Visual data 210, movement data 214, location data 216, and / or operation count 218 may be correlated with time information to enable supervisory controller 160 to associate one or more identified actions or events with a particular time period. For example, time periods may include a time period prior to the detection of a missing or damaged GET 120, a time period corresponding to the initial detection of a potentially missing or damaged GET 120, and a time period (e.g., a delay time described below) following that initial detection. In some aspects, supervisory controller 160 may be configured to receive and analyze the above-described inputs from machine controller 122 in response to an initial detection of a potentially missing or damaged GET made with machine controller 122.Machine controller 122 may be programmed to identify any potentially missing or damaged GET 212 based on visual data 210. Based on this identification, machine controller 122 may transmit a notification to supervisory controller 160 to enable confirmation of the missing GET based on an analysis performed by supervisory controller 160 and / or a confirmation generated by machines 104, 106, 108, drone 130, supervisory controller 160, remote monitoring system 180, or the computer systems associated with operators or supervisors.

[0035] As in Fig. 2, each machine 102, 104, 106, and 108, each drone 130, and each remote system 180 may be in communication with supervisory controller 160. However, in at least some embodiments, one or more of these systems may be in communication with each other directly or indirectly without involvement of supervisory controller 160. Additionally, although motion data 214, location data 216, and operation count 218 are associated with machine 102, as will be understood, one or more of these types of data may be provided to supervisory controller 160 from another machine (e.g., one or more of machines 104, 106, and 108). As an example, operation count 218 could be communicated with a transport machine 106 that receives each material operation, or with another machine (e.g.,Machine 104) having an image processing device 112 or other mechanism capable of displaying operations performed by other machines.

[0036] Fig. 5 is a simplified functional block diagram of a computer system 500 that may be configured as a device for performing method 400 (described below) according to exemplary embodiments of the present disclosure. System 500 represents an exemplary configuration of machine controller 122, supervisory controller 160, remote system 180, a mobile or personal computer system carried by personnel (e.g., operators) within environment 110, and / or another system according to this disclosure.

[0037] Computer system 500 may include a central processing unit ("CPU") 502 in the form of one or more processors for executing program instructions. Computer system 500 may include an internal communications bus 508 and a storage device 506 (e.g., ROM, HDD, SDD, etc.) that may store data on a computer-readable medium 522, although computer system 500 may receive programming and data via network communications. Computer 500 may also include a memory 504 (e.g., RAM) that stores instructions 524 for carrying out the techniques presented herein, although the instructions 524 may be temporarily or permanently stored within other modules of computer 500 (e.g., processor 502 and / or computer-readable medium 522). Computer 500 may also include input and output ports 512 and / or a display 510 to connect to input and output devices such as keyboards, mice, touchscreens, monitors, displays, etc.The various system functions can be implemented on a variety of similar platforms to distribute the processing load. Alternatively, the systems can be implemented by appropriately programming a computer hardware platform. Industrial applicability

[0038] System 100 may be useful in various environments where a ground-engaging tool may be damaged or lost, including mining environments, paving environments, construction environments, forestry environments, and others. System 100 may be usable with various types of machines, including machines suited to a particular environment. In the example of mining applications, earthmoving machines, haulage machines, and processing machines may be present, whereas in construction or paving environments, cold planers, compactors, bulldozers, and other types of machines may be present instead of or in addition to the earthmoving, haulage, and processing machines. System 100 may be configured to monitor tools that may detach from a machine, such as ground-engaging tools for buckets or blades.Under at least some circumstances, system 100 may be configured to confirm damage (e.g., breakage), loss, or wear of a GET 212 while allowing continued operation of machines in environment 110, including machines within a containment area.

[0039] Fig. 3 illustrates an exemplary displayed image 300 that may enable locating a missing GET 120 during operation of the system 100. One or more elements of image 300 may be presented on a display associated with machine 102, 104, 106, 108, supervisory control 160, remote monitoring system 180, or a computer system associated with an operator or supervisor. Each Fig. 3 or described with respect to image 300 may represent a visualization of calculations or determinations performed with machine controller 122, supervisory controller 160, or remote monitoring system 180. In some embodiments, including embodiments in which image 300 is not displayed, the elements shown in Fig. However, the elements illustrated in Figure 3 and described with respect to image 300 represent determinations determined by supervisory controller 160, remote monitoring system 180, etc., that are not necessarily displayed with a display device. For example, a containment area 310 may be displayed to a user if desired for a particular configuration, while in other configurations, the containment area 310 may be generated or otherwise determined by a computer system, such as supervisory controller 160, without being visually represented on a display device.

[0040] Image 300 may include one or more graphical elements representing current or previous locations of machines 102, 104, 106, and 108, locations where work is currently being performed or where work was previously performed (e.g., material source 150), and one or more containment areas 310, 320 representing locations where a damaged or missing GET 120 may be present. As in Fig. 3, each containment area 310, 320 covers multiple locations (e.g., locations contained within each corresponding oval). However, in other configurations, a containment area 310, 320 may instead be a single point (e.g., a "pin" or other symbol associated with a specific location, such as a geographic location on a construction site).

[0041] Machines 102, 104, 106, and 108 may be represented by machine images 302, 304, 306, and 308, respectively. Images 302, 304, 306, and 308 may be static or moving (e.g., real-time) representations of current locations of these machines determined based on motion data 214. These images may also represent locations of machines 102, 104, 106, and 108 at a time when a potentially missing or damaged GET was first identified.

[0042] As shown by ovals in Fig. 3, containment areas 310 and 320 may be determined by the supervisory controller 160 based on each identified location where a missing GET 212 may be present. In some aspects, a single containment area (e.g., area 310) may be created to encompass an entire region that should be isolated due to the possible presence of a missing GET 212. The size and shape of the containment area 310 may therefore be based on: receipt of an acknowledgment of the missing GET 120, a time at which the missing GET 212 was first identified (e.g.,by image processing devices 112), a time period between the initial identification and receipt of the acknowledgment ("delay time"), a distance traveled by the earthmoving machine 104 associated with the missing GET 120 during or after the delay time, a number of operations performed during or after the delay time, and other factors. The shape of the containment areas need not be oval or circular, but may be square, rectangular, or other regular or irregular shapes. In some aspects, the terrain of the environment 110 may be considered to generate a containment area.

[0043] In some aspects, the containment region 310 may form a probability map or a so-called "heat map" (an image depicting areas where the missing GET 120 is more likely to be found compared to others). The regions with a higher probability may be depicted in any suitable form, such as with colors, shading, symbols, etc. A single containment region 310 may include a plurality of regions with different probabilities of containing the missing GET 120. Additionally, multiple containment regions 310 and 320 may exist simultaneously, with different regions 310 and 320 having different probabilities of containing the missing GET 120. Although these different regions 310 and 320 may overlap, they may also be completely separate from each other, as in Fig. 3. In addition, while one or more regions 310 may be stationary (e.g., centered around a location where GET 120 may have been lost), other regions 320 may move by connecting or "attaching" the region 320 to a particular machine and / or a particular material accumulation 126.

[0044] Other information may also be displayed in image 300, such as an operation count, a task performed by a particular machine, a task performed by a machine prior to identification of a missing or damaged GET 120, etc. Other images 300 include, for example, images of a work device 118 taken by the machine associated with the work device (e.g., image “missing GET” 212 of Fig. 2 to confirm a missing or damaged GET 120), images of the work tool 118 taken by another machine, images of the drone 130 or others.

[0045] Any machine that was present in or otherwise connected to a containment area 310, 320 may be highlighted in image 300. Machines connected to a containment area 310, 320 may be isolated, prevented from traveling to processing machine 108, or electronically fenced (e.g., controlled to remain in a specific area). This isolation or containment may be effected by generating commands for these machines via supervisory controller 160 and / or remote monitoring system 180. For example, controller 160, system 180, or both may generate commands for autonomously isolating one or more machines that have entered containment area 310. In some aspects, these commands may cause a machine to travel to a specific isolated location and release material 126. This material may be accessed by personnel (e.g.,Operators) in the vicinity of this machine are searched, with these operators being informed of the missing GET 212 in the material 126 through a notification displayed on a machine display or a mobile computing system. Containment areas 310 and 320 may be static or dynamic. For example, containment area 310 may be static based on a geographic location of the machine 104 when controller 160 first determined the possibility of a missing GET 120. Containment area 320 may move based on the travel of the machine 106 transporting material 126 in which a missing GET 120 may be present.

[0046] Fig. 4 is a flowchart illustrating an exemplary method 400 for monitoring and confirming a damaged or missing tool at a worksite, such as environment 110. While method 400 is described below from the perspective of supervisory controller 160, as understood, other systems included in system 100, such as remote monitoring system 180, may perform at least one and possibly all of the steps of method 400.

[0047] In a step 402, supervisory controller 160 may receive an alert for a potentially missing or damaged GET 120. The terms "receiving an alert" and "receiving an alert" include alerts generated by supervisory controller 160 itself based on visual data 210 and / or other received information (and presented on a display, transmitted to another system, or prompting another action). For example, this alert may be generated according to an analysis of visual data 210 including a "missing GET" image 212. Receiving an alert may also include receiving notification of a potentially missing GET 120 from a machine or other system in communication with controller 160.

[0048] In response to this received warning, supervisory controller 160 may generate a notification requesting acknowledgment via GET monitor 220. Additionally, supervisory controller 160 may begin collecting machine information. Collecting information may include one or more of: receiving information, recording information, or receiving an increased amount of information from machines 102, 104, 106, and / or 108. This information may include visual data 210, motion data 214, location data 216, and operation count 218. The information may be received from the machine associated with the missing GET 120 or from machines within a predetermined distance or line of sight to the machine with the missing GET 120 or the possible location of the missing GET 120.

[0049] The request for confirmation may be presented on a display of supervisory controller 160 and / or one or more other devices of system 100, as described above. The request may be presented on a display within a cab of a machine by machine controller 122 for the machine with the missing GET 120, or on a display of one or more machines that have a view of the machine with the missing GET or that are within a predetermined range of the machine with the missing GET. As described above, the request may also be presented on one or more mobile computing systems (e.g., cellular phones) associated with these machines.

[0050] The confirmation request may include visual data 210 captured by vision device 122 depicting work equipment 118 or an area (e.g., material 126 on a loading platform of a transport machine 106). The image may be a real-time video, a current image, or an image 212 and / or a video from the time the missing GET 120 was first detected.

[0051] Based on the request, one or more responses may be received in step 404, including a response acknowledging the missing GET 120. The acknowledgment may be based on the visual data 210 transmitted or presented during step 402. Thus, the acknowledgment may identify the machine (e.g., machine 104) with the missing GET 120. The acknowledgment may be generated automatically or manually by an operator interacting with an input device of the machines 102, 104, 106, or 108, a mobile device of an operator or other employee, the controller 160, or the system 180.

[0052] The confirmation in step 404 may be influenced by the manner in which the missing GET 120 was confirmed. For example, confirmation may be received from personnel who view image 212 and provide confirmation that image 212 represents a missing or damaged GET 120. This image 212 may be an image generated after the missing GET 120 was initially identified and may be an image captured by the image processing device 122 of a machine other than the machine connected to the GET 120. In other aspects, the image used for confirmation may be the original (i.e., first) image in which a potentially missing GET 120 was identified. Additionally, confirmation may be performed automatically, e.g., through image analysis performed with the controller 160 or another system described herein.Finally, the confirmation may be based on the personal observation of the work tool 118 with the missing or damaged GET 120 by an operator or other personnel.

[0053] A step 406 may include identifying one or more possible locations of the missing GET 120 with the GET monitor 220. For example, the GET monitor 220, model 230, or both may analyze visual data 210, motion data 214, and location data 216. Visual data 210 may be useful for determining an initial time at which the missing GET 120 was first identified and may include previous images or videos to determine the last known time at which the GET 120 was still present and undamaged.Movement data 214 may be useful for identifying one or more actions performed by a machine associated with the missing GET 120, including machine 102 performing an action that may have caused GET 120 to become separated from work tool 118, and one or more actions by a transport machine 106 picking up material from machine 102 that may have resulted in transport of GET 120. GET monitor 220 may use computer vision and GPS data or other location information to enable one or more of these determinations. In conjunction with GET monitor 220, or independently, ML model 230 may use a trained model (e.g., a model trained based on previous incidents in which a GET was lost or damaged) to identify possible locations of GET 120 based on previous data.

[0054] A step 408 may include determining one or more containment areas, such as areas 310 and 320 ( Fig. 3), with GET monitor 220. The possible locations of GET 120 identified in step 406 may be used to determine one or more containment areas. For example, a location, size, and shape of containment area 310 may be based on the current location of machine 104 associated with the missing GET 120, previous locations of machine 104 after the last positive identification of all GETs at machine 104, and interactions of machine 104 with other machines, such as transport machines 106. As described above, motion data 214 may also be used to determine these characteristics of each containment area.

[0055] In some configurations, a first containment area 310 may be established based on the machine 104 that lost a GET 120, and a second containment area 320 may be established based on a transport machine 106 transporting material 126 that may also include at least a portion of the missing GET 120. The second containment area 320 may therefore be generated based on the actions of the machines 104 and 106 that occurred within the first containment area 310.

[0056] In configurations where containment regions 310 and 320 are depicted in image 300 during step 408, the corresponding regions 310 and 320 may be displayed with different colors, shades, or other indications of relative confidence that the missing GET 120 is present within a particular containment region. If desired, each containment region 310 and 320 may form a probability map, as described above, in which portions of each region 310 and 320 indicate sub-regions or portions that have a higher probability of containing the missing GET 120. Additionally, the size, shape, or heat map portions of the regions 310 and 320 may be based on the delay time described above between the initial identification of a potentially missing GET 120 (e.g., in step 402) and the receipt of the confirmation (e.g.,in step 404) and movements of one or more machines that were present in a containment area during those times.

[0057] A step 410 may include taking one or more actions based on the containment areas 310 and 320. In particular, step 410 may include generating commands for autonomously controlling machines that are currently located in a containment area or were previously located in a containment area. As a first example, step 410 may include generating a command for a transport machine 106 (e.g., machine 106 connected to area 320) to travel to an isolation area and unload material 126 to enable automated or manual search of that area by personnel, drones 130, vision devices 112 onboard or external to machines, etc. Commands may also be issued to align thermal imaging devices (e.g.,one or more devices 112) toward the possible location(s) of a missing GET 120 that may retain sufficient heat to enable thermal identification. Additionally, transport machine 106 may be electronically fenced and prevented from entering a prohibited area into which transport machine 106 would otherwise be permitted to enter by issuing autonomous commands or notifications. For example, transport machine 106 may be prevented from entering an area within a predetermined distance of processing machine 108.

[0058] In a second example, step 410 may include monitoring machines within a containment area but allowing at least some continued operations. For example, a haulage machine 106 partially or completely within containment area 310 may continue to receive material from an earthmoving machine 102 that does not have a missing or damaged GET. In some configurations, an earthmoving machine 104 with a potentially missing GET 120 may also be allowed to continue loading material into that haulage machine 106. This may continue until pass count 218 reaches the maximum number of passes associated with a fully loaded haulage machine 106.At this time, operation of the transport machine 106 may be interrupted until the missing GET 120 is located or until supervisory controller 160 receives notification that the potentially missing GET 120 is still present on work device 118.

[0059] If a potentially missing GET 120 is identified as undamaged and / or still present on work equipment 118, or if the missing GET 120 is located as described above, remedial actions such as isolation, electronic fencing, etc., may be aborted. For example, an off-highway truck 106 that has been isolated and / or prevented from traveling to processing machine 108 may receive commands (autonomous commands and / or manual commands) to resume operation by traveling toward processing machine 108. In examples where processing machine 108 itself stops or suspends operation, that operation may also resume. An earthmoving machine 102 or 104 may resume operation by moving material 150 and / or loading haulage machines 106.Finally, any containment area may be ignored or deleted, and systems that received a request for acknowledgment of the missing GET 120 may receive a second notification that the GET 120 has been located or has not been lost.

[0060] System 100 and method 400 may be useful for accurately identifying a potentially missing ground-engaging tool while enabling continued operation of a worksite, such as a mining environment. Confirming a missing tool may avoid false alarms, thereby avoiding an operator's tendency to ignore notification of a missing ground-engaging tool. Additionally, capturing or recording information in response to the initial determination of a missing tool may reduce storage and processing requirements. Additionally, using automation for confirming a missing tool and / or for locating such a tool, including machine learning, machine vision, etc., may at least enable continued operation (e.g., continued loading cycles).Thus, at least some operations can continue while preventing machines that might unknowingly transport a missing tool from introducing material into a machine for further processing. This can improve jobsite efficiency and prevent damage to the machines. Additionally, visualizations illustrating possible locations of a missing tool can facilitate locating the tool. These visualizations can enable the operation of manually controlled machines in a manner that prevents the introduction of a missing or damaged tool into a processing machine. System 100 and method 400 can generate a map or other identification of a probability that a missing tool is located at a particular location.

[0061] It will be apparent to those skilled in the art that various modifications and variations may be made to the disclosed system and method without departing from the scope of the disclosure. Other embodiments of the system and method will be apparent to those skilled in the art in light of the specification and method and system disclosed herein. It is intended that the description and examples be considered exemplary only, with a true scope of the disclosure being indicated by the following claims and their equivalents. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] US 2021 / 0262204 A1

[0005]

Claims

[1] A method for monitoring and confirming damaged or missing ground engaging tools (212) on a construction site, the method comprising: Receiving an alert associated with a damaged or missing ground engaging tool (212) of an earthmoving machine (102, 104); Receiving an acknowledgment for the damaged or missing tool (212); Creating a containment area (310, 320) representing one or more locations where at least a portion of the ground engaging tool (212) may be located; Determining that an additional machine (102, 104, 106, 108) was present in the containment area; and Taking an action for the additional machine (102, 104, 106, 108) based on the presence of the machine (102, 104, 106, 108) in the containment area (310, 320), the action comprising generating a notification or controlling an action of the additional machine (102, 104, 106, 108). [2] The method of claim 1, wherein the additional machine (106) is an off-road truck having a bed that has received material (126) from the earthmoving machine (102, 104). [3] The method of claim 2, further comprising causing the off-road truck (106) to stop operating or travel to an isolation area after picking up the material (126) from the earthmoving machine. [4] The method of claim 3, wherein at least a portion of the material (126) is picked up by the off-road truck after receiving the warning associated with the damaged or missing ground engaging tool (212). [5] A method according to claim 3 or claim 4, wherein the action comprises preventing the additional machine from traveling to a processing machine (108) after receiving the acknowledgment. [6] A method according to any preceding claim, wherein the acknowledgment is received from a machine or system other than the earthmoving machine (102, 104) associated with the damaged or missing ground engaging tool (212). [7] A method according to any one of the preceding claims, wherein the action comprises displaying the containment area (310, 320) as the notification. [8] The method of claim 7, wherein the containment area (310, 320) is represented as a probability map showing a plurality of different possible locations of the damaged or missing tool (212). [9] A monitoring control (160) for monitoring and confirming a missing or damaged tool (212) for an earthmoving machine (102, 104), the control comprising: at least one memory (504) storing instructions (524); and at least one processor (502) operatively connected to the memory (504) and configured to execute the instructions (524) to perform operations comprising: Receiving an alert associated with a damaged or missing tool (212) of a machine (102, 104); in response to receiving the alert, receiving machine location information (216) associated with the machine (102, 104); Generating a request for confirming the missing or damaged tool (212); and Creating a containment area (310, 320) based on the machine location information and receiving confirmation of the missing or damaged tool (212). [10] The monitoring controller (160) of claim 9, wherein the controller (160) further comprises a machine learning model (230) configured to determine a confidence level associated with the damaged or missing tool (212).

Images