Identifying residues in unknown truck beds

Abstract

A system for detecting residues within the dump body of a transport truck may include at least one camera and at least one controller. The at least one camera is configured to generate a scan of the inner surface of the dump body. The at least one controller is configured to receive a scan of the inner surface of the dump body and determine the type of the dump body by comparing the scan of the inner surface of the dump body with at least one scan of the inner surface of a known dump body.

Description

[Technical Field]

[0001] FIELD OF THE DISCLOSURE The present disclosure relates to haul trucks, and more particularly, to detecting and calculating residue within a dump body of a haul truck. [Background technology]

[0002] Machines such as haul trucks or hauling machines are used in various industries to transport or move material from one location to another. When haul trucks are used to transport material, under certain conditions, some of the material may adhere or stick to the interior surface of the dump body of the haul truck after each dumping operation. The material remaining within the dump body may be referred to by different terms, such as residue, residual load, or dead bed.

[0003] Residual material remaining within the dump body is undesirable because it reduces machine productivity. More specifically, the residual material reduces the dump body's effective capacity (e.g., volume), thus requiring a greater number of haul cycles to move the desired amount of material from the loading site to the dump site, or misleading the amount of material being hauled. Furthermore, if a haul truck is loaded based on its volumetric carrying capacity, the truck may become overloaded. The increased weight of the dump body due to the residual material may overload the haul truck, increase axle loads on the road, and / or reduce the fuel efficiency of the haul truck. Each of the above-mentioned problems reduces the efficiency of the material-moving process and can cause damage or excessive wear to the road or the haul truck itself.

[0004] U.S. Patent Application No. 20180179732A1 to Barsch et al., filed December 22, 2016, discloses a toll cargo optimization system including one or more visual sensors coupled to a transport machine and configured to scan and generate a toll cargo body dataset. The system may further include a loading machine including a toll cargo bucket configured to load toll cargo onto the toll cargo body. Furthermore, a loading system controller may be communicatively coupled to each of the transport machine and the loading machine and configured to identify the transport machine and the loading machine using a set of machine identifiers. Furthermore, the controller may receive the toll cargo body dataset from the one or more visual sensors, generate a toll cargo body map, and program a loading sequence for the toll cargo bodies based on the toll cargo body map. The loading system controller may transmit and display a loading sequence configured to guide a loading cycle between the transport machine and the loading machine. Summary of the Invention

[0005] In one example, a system for detecting residue within a dump body of a haul truck may include at least one camera and at least one controller. The at least one camera may be configured to generate a scan of an interior surface of the dump body. The at least one controller may be configured to receive the scan of the interior surface of the dump body and determine a type of the dump body by comparing the scan of the interior surface of the dump body to at least one scan of the interior surface of a known dump body.

[0006] In another example, a method for calculating residue of a dump body of a haul truck may include comparing a scan of an interior surface of the dump body taken by a camera with at least one scan of a known dump body. The method may also include determining a type of dump body captured by the camera when the scan of the interior surface of the dump body matches one of the scans of the interior surface of the known dump body. [Brief explanation of the drawings]

[0007] The drawings are not necessarily drawn to scale, and like numerals may represent similar components in different views. Like numerals with different letter suffixes may represent different instances of similar components. The drawings illustrate generally, by way of example, but not by way of limitation, various embodiments discussed herein.

[0008] [Figure 1] FIG. 1 is a perspective view of a system for scanning the dump body of a haul truck. [Figure 2] FIG. 2 is a perspective view of a system for scanning the dump body of a haul truck. [Figure 3] FIG. 3 is a perspective view of the dump body of the haul truck. [Figure 4] FIG. 4 is a schematic diagram of a system for detecting residue within a dump body of a haul truck. [Figure 5] FIG. 5 is a flow chart illustrating a method for configuring a new dump body. [Figure 6] FIG. 6 is a flow chart illustrating a method for detecting residue within a dump body. DETAILED DESCRIPTION OF THE INVENTION

[0009] FIG. 1 is a perspective view of a system 100 for scanning a dump body 54 of a haul truck 50. A work site 10 may include multiple locations designated for specific purposes. For example, the work site 10 may include a loading location (not shown) where at least one excavator 20 (hereinafter "excavator 20") can load material onto one or more haul trucks 50 (hereinafter haul trucks 50). The work site 10 may also include one or more dump locations (not shown) where a haul truck 50, with or without the aid of an excavator 20, can unload material delivered by the haul truck 50. In another example, a haul truck may be loaded or unloaded at a location not on the work site 10.

[0010] Excavator 20 may be configured to load or unload material from a haul truck 50. Excavator 20 may include a platform 22, at least one ground engaging unit (hereinafter "ground engaging unit 24"), and a dig and drop system 25. In one or more embodiments, excavator 20 may be any type of machine used to excavate material at a work site.

[0011] Platform 22 may be configured to support an operator of excavator 20 to control excavator 20. As shown in FIG. 1 , platform 22 may extend longitudinally between and away from ground engaging units 24. Ground engaging units 24 and dig and drop system 25 may be mounted to platform 22.

[0012] The ground-engaging unit 24 may be configured to move the excavator 20 back and forth along the ground. As shown in Figure 1, the ground-engaging unit 24 may be a tracked assembly or a crawler. In another example, the ground-engaging unit 24 may be a wheel, such as an inflatable or rigid tire, or any other ground-engaging device used to operate a construction vehicle.

[0013] The digging and dropping system 25 may be configured to dig and drop materials at the worksite 10. The digging and dropping system 25 may include a boom 26, a stick member 30, a bucket 34, and a bucket cylinder 38.

[0014] The boom 26 may be attached to and extend from the platform 22. The boom 26 may mechanically couple the platform 22 and the stick member 30. The boom 26 may include at least one boom cylinder 28 (hereinafter, "boom cylinder 28"). The boom cylinder 28 may be attached to the boom 26 at one end and to the platform 22 at the other end. The boom cylinder 28 may extend and retract to move the boom 26 relative to the platform 22.

[0015] The stick member 30 may be pivotally attached to the boom 26. The stick member 30 may extend from the boom 26. The stick member 30 may mechanically couple the boom 26 and the bucket 34. The stick member 30 may include at least one stick member cylinder 32 (hereinafter, "stick member cylinder 32"). The stick member cylinder 32 may be attached at one end to the stick member 30 and at the other end to the boom 26. The stick member cylinder 32 may expand and contract to move the stick member 30 relative to the boom 26.

[0016] The bucket 34 may be configured to penetrate a surface and pick up materials at the worksite 10. The bucket 34 may be pivotally attached to the stick member 30 opposite where the stick member 30 is attached to the boom 26. The bucket 34 may include at least one bucket cylinder 38 (hereinafter “bucket cylinder 38”). The bucket cylinder 38 may be attached at one end to the stick member 30 and at the other end to the bucket 34. The bucket cylinder 38 may extend and retract to move the bucket relative to the stick member 30.

[0017] The boom 26, boom cylinder 28, stick member 30, stick member cylinder 32, bucket 34, and bucket cylinder 36 may be controlled by an operator using an operator controller (not shown) to move the position of the bucket 34 and to pick up and drop material.

[0018] The platform 22 may include a power source 40. The power source 40 may come in any number of different forms, including, but not limited to, an internal combustion engine, an electric motor, a hybrid engine, or any power source used to power construction equipment. Power from the power source 40 may be transmitted to various components and systems of the excavator 20, such as the ground-engaging unit 24 or the dig and drop system 25.

[0019] Excavator 20 may be controlled by one or more controllers (hereinafter "controllers 42"). Controllers 42 may include one or more processors, microprocessors, microcontrollers, electronic control modules (ECMs), electronic control units (ECUs), programmable logic controllers (PLCs), or any other suitable means for electronically controlling the functions of excavator 20.

[0020] The haul truck 50 may be configured to transport and load / unload materials on or off the work site 10. The haul truck 50 may include a frame 52, a dump body 54, a platform 56, and a ground-engaging unit 58.

[0021] The frame 52 may be configured to provide structural support and rigidity to other components of the haul truck 50. For example, the dump body 54, the platform 56, and the ground engaging units 58 may all be mounted to the frame 52. As such, the frame 52 may be subjected to many stresses and strains while the haul truck 50 is loaded, while the haul truck 50 is traveling around or off the job site 10, and while the haul truck 50 is being loaded and unloaded.

[0022] The dump body 54 may be configured to receive material from the excavator 20 or any other equipment capable of loading material into the dump body 54. The dump body 54 may be attached to at least one transport body cylinder (not shown). The dump body 54 may be pivotally attached to the frame 52. The transport body cylinder may be attached to the frame 52 and the dump body 54 such that when the transport body cylinder extends, the dump body 54 can rotate about its connection to the frame 52 to tilt the dump body 54 upward, and when the transport body cylinder retracts, the dump body 54 can rotate about its connection to the frame 52 to level the dump body 54 with the frame 52.

[0023] The ground engaging units 58 may be configured to move the haul truck 50 back and forth along the ground. As shown in FIG. 1, the ground engaging units 58 may be wheels, such as inflatable or rigid tires. In another example, the ground engaging units 58 may be tracked assemblies or crawlers, or any other ground engaging device used to operate a construction vehicle.

[0024] Platform 56 may be configured to hold an operator of haul truck 50 and controls (e.g., a human-machine interface) that control haul truck 50. As shown in FIG. 1 , platform 56 may extend longitudinally between and away from ground-engaging units 58. Platform 56 may include a power supply 70 and a controller 72.

[0025] The power source 70 may be configured to power various components of the haul truck 50. The power source 70 may come in any number of different forms, including, but not limited to, an internal combustion engine, an electric motor, a hybrid engine, or any power source used to power construction equipment. Power from the power source 70 may be transmitted to various components and systems of the haul truck 50, such as the ground-engaging unit 58 or the haul body cylinder 60.

[0026] The controller 72 may be configured to control various components of the haul truck 50. The controller 72 may include one or more processors, microprocessors, microcontrollers, electronic control modules (ECMs), electronic control units (ECUs), programmable logic controllers (PLCs), or any other suitable means for electronically controlling the functions of the haul truck 50.

[0027] As shown in FIG. 1 , a system for measuring residue (hereinafter “system 100”) may be used at the work site 10 or other locations, such as between a loading location and a dumping location. The system 100 may be configured to identify the type of dump bed, calculate the amount of residue in the dump bed, and update an accounting of material moved to the work site 10. The system 100 may include at least one scanning device 102, such as a 3D scanner, camera, or other device, for capturing three-dimensional information about the interior of the dump body. As shown in FIG. 1 , the scanning device 102 may be mounted on the boom 26 of the excavator 20 such that the scanning device 102 has the dump body 54 within its field of view 104. In one or more embodiments, the scanning device 102 may be mounted on the stick member 30 of the excavator 20 such that the scanning device 102 has the dump body 54 within its field of view 104. In yet another embodiment, the scanning device 102 may be mounted anywhere else on the excavator 20 such that the scanning device 102 has the dump body 54 within its field of view 104. The scanning device 102 and field of view 104 are described in more detail below in connection with FIG.

[0028] FIG. 2 is a perspective view of a scanning device 102 of a system 100 for scanning a dump body 54 of a haul truck 50. As shown in FIG. 2, the scanning device 102 may be mounted on a post 170 (e.g., in addition to or instead of a scanning device mounted on an excavator). In one or more embodiments, the post 170 may be sufficiently tall so that when a haul truck 50 parks under, drives, or passes under or beside the scanning device 102, the scanning device 102 may have the dump body 54 of the haul truck 50 within its field of view 104. In one or more embodiments, the post 170 may be located at the work site 10. In another embodiment, the post 170 may be located outside of the work site 10, anywhere a haul truck 50 can operate. For example, the post 170 may be located on the side of a road or other path for one or more haul trucks. In one or more embodiments, post 170 may be positioned a set distance away from work site 10 so that a driver can have his or her truck scanned by scanning device 102 and system 100 can alert the work site that a haul truck (e.g., haul truck 50) is near (e.g., moving away from or approaching) work site 10.

[0029] FIG. 3 is a perspective view of the dump body 54 of the haul truck 50. The dump body 54 may include an interior surface 62, at least one sidewall 64 (hereinafter, “sidewall 64”), a floor 66, and a track wall 68. As shown in FIG. 3 , each of the sidewalls 64 may extend from opposite ends of the floor 66. The track wall 68 may extend from the floor at an end closest to the haul truck (not shown). Thus, the track wall 68 may be configured to extend away from the floor 66 until it reaches the height of the sidewall 64, and then extend away from the sidewall 64 to help protect the platform 56 of the haul truck 50. The interior surface 62 is defined by the sidewall 64, the floor 66, and the track wall 68. The interior surface 62 may define a volume V of the dump body 54.

[0030] The dump body 54 shown in FIG. 3 is merely one example of a dump body. However, dump bodies are provided in any number of different shapes and sizes. In one or more embodiments, the dump body can have higher or lower sidewalls. In one or more embodiments, the dump body can have higher or lower track walls. In one or more embodiments, the dump body can have a longer or shorter floor. In one or more embodiments, the dump body can combine any combination of higher or lower sidewalls, higher or lower track walls, or longer or shorter floors. Therefore, as one or more of these dump body parameters change, the interior surface and volume of the dump body may also change. Accordingly, the system 100 (shown in FIGS. 1 and 2 and discussed in further detail below) can detect the dump body type and determine its volume V. Still further, while a haul truck dump body is contemplated and illustrated, residue may occur in a variety of other devices and systems.

[0031] In one or more embodiments, the system 100 may be configured to identify any type of truck, define a set of criteria to determine the geometric parameters of the dump body and compare them to one or more stored scans, and locate residue within the dump body.

[0032] FIG. 4 is a schematic diagram of a system 100 used to detect residue within a dump body 54 (FIGS. 1-3) of a haul truck 50 (FIGS. 1-2). The system 100 may include a first controller 200, a second controller 208, and a computer 231. In one or more embodiments, the first controller 200, the second controller 208, and the computer 231 may be combined in various combinations. For example, the first controller 200 and the second controller 208 may be combined and interact with the computer 231. In another embodiment, the first controller 200 may be combined with the computer 231 and interact with the second controller 208. In yet another embodiment, the second controller 208 may be combined with the computer 231 and interact with the first controller 200.

[0033] First controller 200 may include one or more processors, microprocessors, microcontrollers, electronic control modules (ECMs), electronic control units (ECUs), programmable logic controllers (PLCs), or any other suitable means for processing image or three-dimensional data captured by scanning device 102 and communicated to second controller 208 and computer 231. First controller 200 may include a storage medium or memory accessible by controller 200, which may be in the form of a physical, non-transitory storage medium, for example, a floppy disk, a hard drive, optical media, random access memory (RAM), read-only memory (ROM), or any other suitable computer-readable storage medium commonly used in the art (each referred to as a "database"). As shown in FIGS. 1 and 2, first controller 200 may be located within system 100. In one or more embodiments, first controller 200 may be a controller (e.g., controller 42 (shown in FIG. 1)) onboard excavator 20. If the scanning device 102 is disposed on a post 170, the controller 200 may reside on the post 170, for example, or may reside in a back office location, for example.

[0034] The first controller 200 can be in electronic communication with one or more scanning devices 102. In one or more embodiments, the first controller 200 can be in electronic communication with two scanning devices 102. The scanning devices 102 can be configured to capture images (moving images (i.e., continuous video images), still images (i.e., photographs taken at a set frequency)) or three-dimensional data within the field of view 104 and retransmit those images and / or data to the first controller 200. In one or more embodiments, the scanning devices 102 can be stereo cameras (i.e., two monochrome and one color camera modules). In another embodiment, the scanning devices 102 can be any other type of camera that can be used to detect residue within the dump body of a dump truck. In yet other embodiments, the scanning devices 102 can include a three-dimensional scanner or other surface capture system.

[0035] Each scanning device 102 can define a field of view 104. To increase or decrease the field of view 104, the scanning devices 102 can be adjusted (e.g., moved farther or closer to the object) to increase the total field of view 104 of the system 100, or more scanning devices 102 can be added. In many embodiments, the scanning devices 102 can be adjusted to increase the field of view 104. For example, a lens (not shown) of the scanning device 102 can be added (or changed) to the scanning device 102 to increase the width of the field of view 104. There are many other modifications (zoom, focus adjustment, etc.) that can be made to the scanning device 102 to improve the field of view 104 and / or adjust the clarity, precision, and / or accuracy of the captured image or data.

[0036] In the embodiment shown in FIGS. 1 and 2 , one scanning device 102 may be mounted on a boom (e.g., boom 26) of an excavator (e.g., excavator 20). In another embodiment, one of the scanning devices 102 may be mounted on a boom (e.g., boom 26) and another of the scanning devices 102 may be mounted on a stick member (e.g., stick member 30) of an excavator (e.g., excavator 20). In yet another embodiment, one or two of the scanning devices 102 may be mounted on the boom and / or stick member of the excavator. In one or more embodiments, any number of scanning devices 102 may be added to the boom or stick member of the excavator to provide image capture quality and improve the field of view 104. Additionally or alternatively, one or more scanning devices 102 may be mounted on one or more posts 170.

[0037] The first controller 200 may also include a vision processing electronic control module 202. The vision processing electronic control module 202 may be configured to receive captured images or data from any and / or all of the scanning devices 102 and process the captured images or data for transmission to the second controller 208 and the computer 231. The vision processing electronic control module 202 may communicate with any of the scanning devices 102 of the system 100 to receive and aggregate captured images or data from the various scanning devices 102. The vision processing electronic control module 202 may also be electrically connected to the gateway electronic control module 204.

[0038] The gateway electronic control module 204 may enable communication of the aggregated captured images or data from the vision processing electronic control module 202 to the modem 206. Thus, the gateway electronic control module 204 may enable wireless communication of the aggregated captured images or data via the modem 206 to either the second controller 208 and the computer 231.

[0039] The second controller 208 may include one or more processors, microprocessors, microcontrollers, electronic control modules (ECMs), electronic control units (ECUs), programmable logic controllers (PLCs), or any other suitable means for processing images or three-dimensional data communicated from the first controller 200 or the computer 231. The second controller 208 may include a storage medium or memory accessible by the second controller 208, which may be in the form of a physical, non-transitory storage medium, for example, a floppy disk, a hard drive, optical media, random access memory (RAM), read-only memory (ROM), or any other suitable computer-readable storage medium commonly used in the art (each referred to as a “database”). The second controller 208 may be configured to at least receive aggregated captured images or data from the first controller 200, process the aggregated captured images or data, identify the type of dump body captured in the images, complete accounting based on the processed aggregated captured images or data, and communicate with the first controller 200 and the computer 231. The second controller 208 can process the aggregated captured images from the first controller 200 during a known scan of a known dump body of a known haul truck. As described above, the second controller 208 can include a computer-readable storage medium. The computer-readable storage medium can store various types of information, including a baseline model repository 210, a truck ID list 212, a driver ID list 214, a driver app location 216, an excavator ID 218, trigger threshold data 220, a scan comparator algorithm 222, a truck type list 224, a three-dimensional computer-aided design model of a truck type (hereinafter, "truck type 3D CAD model 226"), and an inventory control program 228. The second controller 208 can also include a modem 230.

[0040] The baseline model repository 210 may be configured to store known scans of known truck body types. Aggregated scans of known dump bodies and haul trucks from the second controller 208 may be stored in the baseline model repository 210 and used as reference scans for other operations performed by the second controller 208. Additionally, the second controller 208 may update the scans stored in the baseline model repository 210 if the known dump body has a scan that results in a higher or lower volume than the previous baseline scan. Each scan stored in the baseline model repository 210 may be identified by a truck identification pin 211 or a driver identification pin 213.

[0041] The truck ID list 212 may be configured to store information about one or more trucks operating at the work site 10 (FIG. 1). The truck's truck identification pin 211 may be stored in the truck ID list 212. As the truck travels through the work site 10, the system 100 may store all information in the second controller 208 (e.g., the location of the truck, the amount of material moved by the truck, the amount of time the truck was at the work site, or any other information related to the truck on the work site).

[0042] The driver ID list 214 may be configured to store information about one or more drivers operating at the worksite 10. The driver's driver identification pin 213 may be stored in the driver ID list 214. As the drivers operate the worksite 10, the system 100 may store all information (e.g., the driver's location, the amount of material moved by the driver, the amount of time the driver was at the worksite, or any other information related to one or more drivers on the worksite) on the second controller 208.

[0043] Driver app location 216 may be configured to store the driver's location on or off worksite 10. In one or more embodiments, the driver's location may be shared with second controller 208 via computer 231. In another embodiment, the driver's location may be shared with second controller 208 by one or more controllers on the haul truck or other work machine (e.g., controller 72 on haul truck 50). Second controller 208 may use the driver's location to automatically alert operators of other equipment (e.g., the operator of excavator 20) when the haul truck is ready for loading or unloading.

[0044] Excavator ID 218 may be configured to store information about one or more excavators on worksite 10. Information stored in excavator ID 218 may include payload capacity (e.g., boom and bucket size) and the location of the excavator.

[0045] Trigger threshold data 220 may be collected and stored for each type of dump body to signal the second controller 208 when residue is detected, when a new scan of the dump body should be performed, or when a new baseline should be saved to the baseline model repository 210. The trigger threshold data 220 may include various shapes of the dump body. For example, the trigger threshold data 220 may be the floor length or sidewall height of the dump body. In another example, the trigger threshold data 220 may be the volume of an interior surface (e.g., the interior surface 62 of the dump body 54). In yet another example, the trigger threshold data 220 may be any other parameter that can be detected by the system 100 and that can signal that residue has been detected or that the system 100 needs to be updated. In one or more examples, the trigger threshold data 220 includes a surface profile of the interior surface of the dump body.

[0046] The trigger threshold data 220 may be set for any data collected on the dump body of a haul truck. In some examples, the trigger threshold data 220 may account for noise and / or inaccuracies in the measurement system. Thus, the trigger threshold data 220 may be set to a value that exceeds the known variability of the system. For example, if the system has a known error of five percent, the trigger threshold data 220 may be set to any measurement value that exceeds five percent. In one or more examples, the system 100 may use a flatness score for the interior surface of the dump body. The system 100 may analyze previous scans or models of the truck body to define the boundaries of a portion of the planar shape. In one or more examples, a neural network or machine learning system may be used to automatically detect and define the boundaries of a portion of the planar shape on the dump body of a dump truck. The system 100 may acquire known images and system-captured scans of an empty dump body and generate a point cloud map of the stored images and system-captured scans. Each defined region of the point cloud map may be rectified so that the plane is aligned with the XY plane. Thus, the elevation difference between the highest and lowest points in the point cloud map of the known image of the empty dump body and the scan captured by the system may be the flatness score in the XY plane. Thus, the trigger threshold data 220 may be configured to reduce false alarms about detecting residue and / or determining that a new baseline scan is needed.

[0047] Truck type list 224 may be configured to store a list of known truck types (e.g., dump bodies having recognized interior surface geometries) stored in second controller 208. Truck type list 224 may be transmitted to another controller (e.g., first controller 200, computer 231, or any other controller in wireless communication with system 100) to share the stored truck types. For example, if system 100 scans a dump body and does not recognize the truck type, system 100 may prompt the driver to select a dump body type from truck type list 224.

[0048] Truck type 3D CAD model 226 may be stored in second controller 208 as a model of a known dump body type. In one or more embodiments, truck type 3D CAD model 226 may include truck types from known manufacturers or their partners. In one or more embodiments, a 3D model of a new truck may be created. In another embodiment, a new truck that is not a known truck type may be 3D modeled and added to truck type 3D CAD model 226. Truck type 3D CAD model 226 may provide accurate, substantially accurate, and / or relatively accurate dimensions of the dump body of the haul truck per 3D CAD model. In one or more embodiments, the accurate dimensions may be accurate to the same or similar level of manufacturing tolerances of the CAD model or manufacturing drawing, for example. The dimensions of the 3D CAD model stored in truck type 3D CAD model 226 may be used by scan comparator algorithm 222.

[0049] The scan comparator algorithm 222 may be implemented to compare scans captured from a first controller (e.g., first controller 200) with known scans stored in a database (e.g., baseline model repository 210 or track type 3D CAD models 226). The scan comparator algorithm 222 may include a neural network for continuously comparing new scans from the first controller with known scans previously stored in the system 100. The neural network may be a statistical analysis algorithm that compares the frequencies of different object primitives to define input and / or output correlations. Thus, the system 100 may capture images of different track types and classify the captured images with track boundaries, images, and labels for various track types. As the neural network receives and analyzes more tracks, it can generate a matrix of weights. For example, the neural network may generate coefficients in a large-scale n-dimensional curve fit. The n-dimensional curve fit of the known image or scan may then be compared to unlabeled data to determine whether the neural network's target accuracy has been achieved. In an embodiment, the neural network may be configured for a specific task. For example, a neural network may be configured for image recognition. A neural network configured for image recognition may be able to analyze a custom data set (e.g., labeled photographs of machinery or dump bodies) and perform a specific task (e.g., identifying the type of machinery or dump body in the photograph, outlining a portion of an image showing the dump body of a dump truck).

[0050] In another example, the scan comparator algorithm 222 can compare the new scan to a computer-aided design (CAD) model of a known dump body. If the known scan or known 3D CAD model matches the truck type, the scan comparator algorithm 222 can determine the volumetric (e.g., volume V in FIG. 3 ) difference between the scanned dump body and the scan of the known dump body or known 3D CAD model. If a CAD model is not available, the scan comparator algorithm 222 may include a two-sided comparator secondary test and a flatness comparator secondary test.

[0051] The two-sided comparator secondary test of the scan comparator algorithm 222 may use a neural network to divide the scanned area in half along the length of the truck body floor and compare each half of the divided scan area to the other half of the divided scan area. In some examples, as described above, the neural network may analyze captured images of the dump body and determine the area of ​​the captured image corresponding to the dump body of the haul truck. The area corresponding to the dump body of the haul truck may be extracted from the disparity map as a 3D point cloud and rectified to the XY plane. The neural network may then divide the 3D point cloud along the center on the y-axis and compare the height of each point in one half to the height of each corresponding point from the other half. If the difference between the divided scanned areas exceeds a threshold stored in the trigger threshold data 220, the second controller 208 may identify the dump body as containing residue and / or flag the dump body as scraped.

[0052] The flatness comparator secondary test of the scan comparator algorithm 222 can identify geometric features of the dump body (e.g., corners, planes, or intersections) and calculate a flatness score for each planar feature (e.g., floor 66, sidewall 64, or truck wall 68). In one or more embodiments, the system 100 can analyze the scan for flatness scores using a neural network. For example, the neural network can take a captured image and define planes on the interior of the dump body. The neural network can then create a 3D cloud map on the plane and orient the 3D cloud map on the X and Y axes. The neural network can then compare maximum and minimum height values ​​to generate a flatness score for each of the planes on the interior of the haul truck dump body. If the flatness score of any of the dump body's planar features is greater than a maximum threshold or less than a negative threshold stored in the trigger threshold data 220, the second controller 208 can identify the dump body as containing residue and / or flag the dump body as scraped.

[0053] Therefore, by comparing two scans, comparing a scan to a 3D CAD model, a two-sided comparator secondary test, or a flatness comparator secondary test, the scan comparator algorithm 222 can predict the presence of residue within the dump body of the dump truck.

[0054] Inventory control program 228 may be configured to receive inputs from various systems of system 100. For example, inventory control program 228 may use baseline model repository 210, truck ID list 212, driver ID list 214, driver app location 216, and excavator ID 218 to track the location of equipment on worksite 10 and calculate the theoretical amount of material to be moved by the machine. Inventory control program 228 may also use trigger threshold data 220, scan comparator algorithm 222, truck type list 224, and truck type 3D CAD model 226 to automatically remove the calculated volume of residue from the theoretical amount of material to be moved by the machine to more accurately account for the material on worksite 10. In one or more embodiments, inventory control program 228 may also alert an excavator (e.g., excavator 20) that residue is in a haul truck (e.g., haul truck 50) to prevent the excavator from overloading the haul truck. Thus, the inventory control system 228 may help ensure that haul trucks are not overloaded (eg, within the weight limits of the transportation department) before leaving the work site 10 .

[0055] The modem 230 may be configured to allow the second controller 208 to communicate wirelessly with the first controller 200 and the computer 231. The modem 230 may include a gateway control module that allows aggregated information to be shared between the second controller 208 and other controllers or computers in the system 100.

[0056] Computer 231 may be configured to help the truck driver communicate with other components of system 100. Computer 231 may be a cell phone, tablet, laptop, or any other type of computer that can be located in the truck or at work site 10 that the driver can access to communicate with other components of system 100. For example, the driver may communicate with second controller 208 to provide a driver ID, a truck ID, and the location of the truck they are driving. For example, computer 231 may include a driver ID 232, a truck ID 234, a location 236, and a modem 238.

[0057] The driver ID 232 may be configured to communicate the unique identification of the driver to the second controller 208 so that the second controller 208 can download and upload information about the driver. The driver ID 232 may be assigned to the driver before the driver enters a work site (e.g., work site 10) for the first time or when the driver enters the work site for the first time.

[0058] Truck ID 234 may be configured to communicate the unique identification of the truck to second controller 208 so that second controller 208 can download and upload information about the truck. Because a driver may drive multiple trucks, truck ID 234 may also be tracked. Truck ID 234 may be assigned to a truck either before the truck first arrives at a job site or after the truck has operated at a job site.

[0059] Location 236 may be the location of the computer at any given time. Location 236 may be measured by a global positioning sensor or some other location device. Location 236 can communicate the location of computer 231 whether computer 231 is on or off the job site.

[0060] The modem 238 may be configured to allow the computer 231 to wirelessly communicate with the first controller 200 and the second controller 208. The modem 238 may include a gateway control module that allows aggregated information to be shared between the computer 231 and other controllers or computers in the system 100.

[0061] In one or more embodiments, the first controller 200 and the second controller 208 can communicate work information to the computer 231. For example, the second controller 208 can communicate the amount of payable cargo to be loaded onto the haul truck 50 and the time it will take to load the haul truck 50. In another embodiment, the second controller 208 can communicate the total amount of payable cargo to be moved by the dump truck 50 at the work site 10. In yet another embodiment, all of the information of the first controller 200 and the second controller 208 can be shared with the computer 231. [Industrial Applicability]

[0062] In one or more operational embodiments of the disclosed system, the system 100 may include a truck onboarding program 500 and a residue detection program 600 .

[0063] 5 is a flowchart illustrating a truck onboarding program 500. When a new truck (e.g., haul truck 50) enters a worksite (e.g., worksite 10), system 100 can execute truck onboarding program 500. Truck onboarding program 500 is configured to collect and communicate information to one or more controllers while the truck is at the worksite so that the information can be stored and used in various calculations. In one embodiment, onboarding program 500 can be installed on a back-office controller (e.g., second controller 208). In another embodiment, onboarding program 500 can be installed on a machine (e.g., a controller on an excavator, truck, or any other machine on the worksite).

[0064] In step 502, the program can generate a new truck ID for a new truck at the job site. The new truck ID is a unique truck ID that can be tied to the truck for the life of the truck or for the duration of the truck's presence at the job site. The new truck ID can be appended with any additional information collected by the truck onboarding program 500.

[0065] In step 504, the program may initiate a truck scan. In one embodiment, a controller (e.g., second controller 208) may wirelessly communicate with another controller (e.g., first controller 200) to capture a scan of the dump body of the truck. As described above, the scan of the dump body may be performed by a controller mounted on the excavator. In another embodiment, the scan of the dump body may be performed by a controller mounted on a post (e.g., post 170). In yet another embodiment, the scan of the dump body may be performed by a controller remote from the job site.

[0066] In step 506, a controller (e.g., first controller 200) can scan the dump body of the truck, and the controller can aggregate the scan and wirelessly transmit the scan to a back office. The back office can store this scan in a database (e.g., baseline model repository 210) and reference it for all future scans. Once stored in baseline model repository 210, the baseline scan is linked to the truck identification pin (e.g., truck identification pin 211) and driver identification pin (e.g., driver identification pin 213) of the haul truck and driver.

[0067] In step 508, the back office may compare the scan received from the controller with scans of known truck beds stored in a repository (e.g., baseline model repository 210) to detect the truck type of the truck scanned by the controller.

[0068] In step 510, a controller (e.g., second controller 208) can compare the received scan of the haul truck dump body to a 3D CAD model stored in the controller (e.g., truck type 3D CAD model 226). If a 3D CAD model exists, the controller can execute a comparator function (e.g., scan comparator algorithm 222) to compare the scan of the dump body to the 3D CAD model of the dump body. If a 3D CAD model of the dump body type does not exist, the controller can use one or more secondary threshold tests.

[0069] In step 512, the controller may perform a secondary threshold test (e.g., the two-sided comparator and / or the flatness comparator of the scan comparator algorithm 222). The controller may include a prompt that allows the operator, shop floor supervisor, or engineer to select between the two-sided comparator or the flatness comparator secondary test. In another example, the controller may perform both the two-sided comparator and the flatness comparator secondary test and combine and / or compare the results. In yet another example, a neural network installed in a controller (e.g., the second controller 208) may scan dump bodies to find similar dump bodies and determine whether to perform the two-sided comparator and / or the flatness comparator secondary test.

[0070] In step 514, the controller may perform a two-sided comparator secondary test by dividing the scan in half along the length of the dump body floor and comparing each half of the divided scan with the other half of the divided scan. The two-sided comparator secondary test may then determine whether any residue is present by detecting differences between the halves of the scan.

[0071] In step 516, the controller may perform a flatness comparator secondary test by identifying geometric features of the dump body (e.g., corners, planes, or intersections) and calculate a flatness score for each planar feature of the dump body (e.g., floor, sidewall, or track wall). A lack of flatness detected by the flatness comparator secondary test may indicate the presence of residue within the dump body.

[0072] In step 518, if the comparator function outputs a volume number above a threshold, the controller can send a signal to a computer or controller on the haul truck (e.g., computer 231) or on the excavator (e.g., first controller 200) that residue on the dump body needs to be scraped off. In one example, the controller can compare the known scan (or 3D CAD model) to the new scan, and the comparator can output a positive value above a positive threshold, indicating residue is present. In another example, the controller can compare the new scan to the known scan (or 3D CAD model), and the comparator can output a negative value below a threshold, indicating residue is present.

[0073] In step 520, if the comparator function outputs a positive or negative value above or below a threshold, the controller can send a signal to a computer or controller on the haul truck (e.g., computer 231) or excavator (e.g., first controller 200) that the dump body needs a recapture of a baseline scan. For example, if the comparator function compares a known scan to a new scan and the new scan appears to have a larger volume than the known scan, the controller can send a signal to a computer or controller on the haul truck or excavator that the dump body needs a new baseline scan.

[0074] 6 is a flow chart illustrating a residue detection program 600. The residue detection program can be configured to detect residue within a known or unknown dump body (e.g., dump body 54) of a haul truck (e.g., haul truck 50).

[0075] In step 602, a computer (e.g., computer 231) can transmit a truck ID and a driver ID (e.g., truck identification pin 211 and driver identification pin 213) to a back office (e.g., second controller 208). The computer can transmit the truck ID and the driver ID before the haul truck enters the work site or after the haul truck enters the work site (e.g., work site 10).

[0076] In step 604, the computer can transmit the truck location to a back office. The truck location can be used to track the truck's location on and off the job site. For example, the truck's location can be determined when the truck is a set distance from the job site. In another example, the back office can determine if the truck is in (or approaching) a position for loading or unloading by an excavator. The truck's location can be stored in a database on the controller (e.g., driver app location 216).

[0077] In step 606, the back office can compare the truck's location to the excavator's location fence. By instructing the excavator how far away it is from the excavator fence, the excavator can be informed of how long it has until it needs to load or unload a haul truck. This can give the excavator an estimate of how long it can continue working on its current task before it needs to go load or unload a haul truck.

[0078] In step 608, the back office can alert the excavator when a haul truck arrives. An excavator location fence can be set by the back office to automatically notify the excavator that the truck needs to be loaded or unloaded as it crosses the fence. Alerting the excavator allows it to drive to a location before the truck arrives to maximize job site efficiency.

[0079] In step 610, the back office can alert a computer (e.g., computer 231) and a controller (e.g., second controller 208) that the truck does not have a baseline scan, and the controller can capture and send a baseline scan to the back office. Alternatively or additionally, the baseline scan can be captured, for example, while the truck is traveling between a fill location and a dump location.

[0080] At step 612, the excavator is enabled for a new truck fill operation, which may record data regarding the truck fill (e.g., fill time, fill volume (based on a measurement system attached to the excavator), fill location, fill pattern, or any other data that may be useful in knowing how the haul truck was loaded by the excavator).

[0081] In step 614, a neural network residing in one of the controllers (e.g., first controller 200 or second controller 208) continuously analyzes the scans received from scanning device 102 to determine the location of the dump body of the haul truck. In one or more embodiments, the truck or excavator may include a QR code or April tag to help system 100 identify the truck type.

[0082] At step 616, a scanning device 102 on the boom of an excavator (eg, excavator 20) continuously scans the dump body as the excavator loads material into the dump body.

[0083] In step 618, the controller (e.g., the first controller 200) may send a scan of the dump body (e.g., a scan of the dump body or a scan of the loading operation from the excavator) to the back office (e.g., the second controller 208).

[0084] In step 620, the back office can compare the scan of the dump body (e.g., a scan of the dump body or a scan of a loading operation from an excavator) to a stored baseline scan, a 3D CAD model of a known dump body type, a bilateral comparator, or a flatness comparator to compare the scans and obtain volumetric difference values.

[0085] In step 622, the back office can send an alert to the dump truck and excavator to scrape off residue within the dump body if any of the comparison functions output a value above or below a threshold. For example, the comparator can output a positive value indicating residue within the dump body of the haul truck (e.g., when the volume of a scan of the dump body is subtracted from the volume of a known scan or 3D model). In another example, the comparator can output a negative value indicating residue within the dump body of the haul truck (e.g., when the volume of a known scan or 3D model is subtracted from the volume of a scan of the dump body).

[0086] In step 624, if any of the comparison functions output a value above or below a threshold, the back office can send a recapture baseline scan message to the dump truck and excavator. For example, the comparator can output a positive value indicating the need for a new baseline scan of the dump body of the haul truck (e.g., if the volume of a known scan or 3D model is to be subtracted from the volume of the scan of the dump body). In another example, the comparator can output a negative value indicating the need for a new baseline scan of the dump body of the haul truck (e.g., if the volume of a scan of the dump body is to be subtracted from the volume of a known scan or 3D model).

[0087] The foregoing detailed description is intended to be illustrative, not limiting. Accordingly, the scope of the present disclosure should be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled.

Claims

1. 1. A system for detecting residue within a dump body of a haul truck, comprising: an excavator configured to lift and dump material into the dump body, the excavator including a boom member configured to extend at least a portion of the boom member above the dump body; a scanning device configured to generate a scan of an interior surface of the dump body; a first controller, receiving the scan of the interior surface of the dump body; detecting the presence of residue within the dump body by comparing the scan of the interior surface of the dump body with scans of interior surfaces of known dump bodies; calculating a difference in volume between the scan of the interior surface of the known dump body and the scan of the interior surface of the dump body; a first controller configured to alert an operator of the excavator of the amount of residue present in the dump body of the haul truck, enabling the operator to avoid overloading the dump body of the haul truck.

2. The first controller is mounted on the excavator, and the system comprises:

10. The system of claim 1, further comprising a second controller located at the work site, wherein both the first controller and the second controller are configured to wirelessly communicate with each other.

3. The first controller is mounted on the excavator, and the system comprises: The system of claim 1 , further comprising a remotely located second controller, wherein both the first controller and the second controller are configured to wirelessly communicate with each other.

4. and a computer configured to receive information from the haul truck driver, the computer comprising: Driver identification pin, a track identification pin; and the location of the haul truck via a global positioning sensor.

5. 5. The system of claim 4, wherein the computer is a cell phone having an application that the driver executes to enter the driver identification pin and the truck identification pin.

6. The second controller: a baseline model repository configured to store said scans of interior surfaces of known dump bodies; a truck identification pin list including all known haul trucks that have entered the work site; and a driver identification pin list including all known drivers who have entered the work site; a location of a driver at the work site, the location tracking the location of all trucks that have been at the work site; 5. The system of claim 4, further comprising an excavator identification pin list, the excavator identification pin list including all excavators at the work site.

7. The second controller: receiving from the computer the driver identification pin, the truck identification pin, and the location of the haul truck; determining a dump body type of the haul truck with the truck identification pin by referencing the truck identification pin list; obtaining the scan of the dump body type associated with the track identification pin from the baseline model repository; receiving the scan of the interior surface of the dump body from the first controller; calculating the residue using at least one scan comparison algorithm to compare the scan of the dump body type associated with the truck identification pin with the scan of the interior surface of the dump body for transporting material; and The system of claim 6 , configured to send an alert to the computer when the residual is calculated.

8. The second controller includes an inventory control system, and the inventory control system calculating an estimated amount of material that can be loaded onto the dump body of the haul truck; Subtracting said calculated amount of residue to find the actual amount of material moved; and The system of claim 7 , configured to store the actual amount of material moved for updating work site accounting.

9. The first controller includes a new track program, which, when executed, assigning a unique truck identification pin to new trucks at said work site; communicating the unique track identification pin to the second controller, the second controller storing the unique track identification pin in the track identification pin list; transmitting a signal to the scanning device to capture a scan of the new truck at the work site; and The system of claim 6 , further comprising: communicating the scan of the new truck at the work site to the second controller; and the second controller storing the scan in the baseline model repository.

10. The second controller includes a new truck prompt program, which, when executed, 10. The system of claim 9, wherein when not finding the track identification pin from the first controller in the track identification pin list, sending an alert to the computer to execute the new track program.

11. The second controller: a two-sided symmetric scan comparator algorithm configured to detect residue within a dump body of the dump truck that cannot be identified by the system, wherein when executed, the two-sided symmetric scan comparator algorithm: transmitting a signal to the scanning device to capture a scan of the dump body; analyzing the captured scans to determine an area of ​​the dump truck corresponding to the dump body; extracting a disparity map as a three-dimensional point cloud from the captured scans; rectifying the three-dimensional point cloud from the captured scan onto an XY plane; Dividing the acquired scan in half at the midpoint of the X plane to generate a first half of points and a second half of points; comparing the height of each point in the first half of the points to a correlated point in said second half of the points to generate a bilateral symmetric score; and compiling said two-sided symmetric scores and comparing them to a threshold number; The system of claim 6 , wherein if the two-sided symmetry scan comparator algorithm determines that the two-sided symmetry score is above a threshold, it alerts an operator that residue may be present within the dump body.

12. The second controller: a flatness scan comparator algorithm configured to detect residue within a dump body of the dump truck that cannot be identified by the system, and implementing the flatness scan comparator algorithm; transmitting a signal to the scanning device to capture a scan of the dump body; analyzing the captured scans to determine an area corresponding to the dump body of a dump truck; extracting a disparity map as a three-dimensional point cloud from the captured scans; rectifying the three-dimensional point cloud from the captured scan onto an XY plane; Calculate a flatness score by comparing the highest and lowest points on the flat surface, and Compiling the flatness scores and comparing them to a threshold number; The system of claim 6 , wherein if any of the flatness scan comparator algorithms determines that the flatness score is above a threshold, the system alerts an operator that residue may be present within the dump body.

Images