Identify residues in an unknown track bed

Abstract

A system for detecting residues within the dump body of a transport truck may include at least one camera and at least one controller. The at least one camera is configured to generate a scan of the inner surface of the dump body. The at least one controller is configured to receive a scan of the inner surface of the dump body and determine the type of the dump body by comparing the scan of the inner surface of the dump body with at least one scan of the inner surface of a known dump body.

Description

Technical Field

[0001] The present disclosure relates to a transport truck. More specifically, the present disclosure relates to detecting and calculating residues within the dump body of a transport truck.

Background Art

[0002] Machines such as transport trucks or transport machinery are used in various industries to transport or move materials from one location to another. When using a transport truck to transport materials, under certain conditions, a portion of the materials can adhere or attach to the inner surface of the dump body of the transport truck after each dumping operation. The materials remaining within the dump body may be referred to by different terms such as residues, residual loads, or dead beds.

[0003] The residual materials remaining within the dump body are undesirable as they reduce the productivity of the machine. More specifically, the residual materials reduce the effective capacity (e.g., volume) of the dump body and, therefore, require a greater number of transport cycles to move a desired quantity of materials from the loading site to the dumping site or can cause misunderstandings about the quantity of materials being transported. Further, if the transport truck is loaded based on its volumetric transport capacity, the truck may become overloaded. The increase in the weight of the dump body due to residual materials can overload the transport truck, increase the axle load on the road, and / or reduce the fuel efficiency of the transport truck. Each of the above-mentioned problems can reduce the efficiency of the material movement process and can cause damage or excessive wear to the road or the transport truck itself.

[0004] U.S. Patent Application No. 20180179732A1 by Barsch et al., filed on December 22, 2016, discloses a toll cargo optimization system that is coupled to a handling machine and includes one or more vision sensors configured to scan and generate a toll cargo body data set. The system may further include a loading machine that includes a toll cargo bucket configured to load toll cargo onto a toll cargo body. Additionally, a loading system controller may be communicatively coupled to each of the handling machine and the loading machine and may be configured to identify the handling machine and the loading machine using a set of machine identifiers. Further, the controller may receive a toll cargo body data set from the one or more vision sensors, generate a toll cargo body map, and program a loading sequence of the toll cargo body based on the toll cargo body map. The loading system controller may transmit and display a loading sequence configured to guide a loading cycle between the handling machine and the loading machine.

Summary of the Invention

[0005] In one embodiment, a system for detecting residues within a dump body of a haul truck may include at least one camera and at least one controller. The at least one camera may be configured to generate a scan of an interior surface of the dump body. The at least one controller may be configured to receive a scan of the interior surface of the dump body and determine a type of the dump body by comparing the scan of the interior surface of the dump body to at least one scan of an interior surface of a known dump body.

[0006] In another embodiment, a method for calculating residues of a dump body of a haul truck may include comparing a scan of an interior surface of the dump body taken by a camera to at least one scan of a known dump body. The method may also include determining a type of the dump body captured by the camera when the scan of the interior surface of the dump body matches one of at least one scan of an interior surface of a known dump body.

Brief Description of the Drawings

[0007] The drawings are not necessarily drawn to scale, and like numbers may represent similar components in different figures. Like numbers with different letter suffixes may represent different instances of similar components. The drawings generally illustrate, by way of example, various embodiments discussed in this specification, but are not limiting.

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

DETAILED DESCRIPTION OF THE INVENTION

[0009] FIG. 1 is a perspective view of a system 100 for scanning the dump body 54 of a transport truck 50. The work site 10 may include a plurality of locations designated for a specific purpose. For example, the work site 10 may include a loading position (not shown) where at least one excavator 20 (hereinafter, "excavator 20") can load materials onto one or more transport trucks 50 (hereinafter, transport truck 50). The work site 10 may also include one or more dumping positions (not shown) where the transport truck 50 can load and unload the materials carried by the transport truck 50, with or without the assistance of the excavator 20. In another embodiment, the transport truck may be loaded or unloaded at a location not on the work site 10.

[0010] The excavator 20 can be configured to be loaded onto or unloaded from the transport truck 50. The excavator 20 can include a platform 22, at least one ground engaging unit (hereinafter, "ground engaging unit 24"), and an excavation and dropping system 25. In one or more embodiments, the excavator 20 can be any type of machine used to excavate materials at a work site.

[0011] The platform 22 can be configured to hold an operator of the excavator 20 who controls the excavator 20. As shown in FIG. 1, the platform 22 can extend in the longitudinal direction between and away from the ground engaging units 24. The ground engaging unit 24 and the excavation and dropping system 25 can be attached to the platform 22.

[0012] The ground engaging unit 24 can be configured to move the excavator 20 in the longitudinal direction along the ground. As shown in FIG. 1, the ground engaging unit 24 can be a tracked assembly or a crawler. In another embodiment, the ground engaging unit 24 can be a wheel such as an inflatable or rigid tire, or any other ground engaging device used in the operation of a construction vehicle.

[0013] The excavation and dropping system 25 can be configured to excavate and drop materials at the work site 10. The excavation and dropping system 25 can include a boom 26, a stick member 30, a bucket 34, and a bucket cylinder 38.

[0014] The boom 26 can be attached to and extend from the platform 22. The boom 26 can mechanically couple the platform 22 and the stick member 30. The boom 26 can include at least one boom cylinder 28 (hereinafter, "boom cylinder 28"). The boom cylinder 28 can be attached to the boom 26 at one end and to the platform 22 at the other end. The boom cylinder 28 can expand and contract to move the boom 26 relative to the platform 22.

[0015] The stick member 30 can be pivotally attached to the boom 26. The stick member 30 can extend from the boom 26. The stick member 30 can mechanically couple the boom 26 and the bucket 34. The stick member 30 can include at least one stick member cylinder 32 (hereinafter, "stick member cylinder 32"). The stick member cylinder 32 can be attached to the stick member 30 at one end and to the boom 26 at the other end. The stick member cylinder 32 can be extended and retracted to move the stick member 30 relative to the boom 26.

[0016] The bucket 34 can be configured to penetrate the surface and pick up materials at the work site 10. The bucket 34 can be pivotally attached to the stick member 30 at a location opposite to where the stick member 30 is attached to the boom 26. The bucket 34 can include at least one bucket cylinder 38 (hereinafter, "bucket cylinder 38"). The bucket cylinder 38 can be attached to the stick member 30 at one end and to the bucket 34 at the other end. The bucket cylinder 38 can be extended and retracted to move the bucket relative to the stick member 30.

[0017] The boom 26, boom cylinder 28, stick member 30, stick member cylinder 32, bucket 34, and bucket cylinder 36 can be controlled by an operator using an operator controller (not shown) to move the position of the bucket 34 and pick up and drop materials.

[0018] The platform 22 can include a power source 40. The power source 40 can be provided in any number of different forms including, but not limited to, an internal combustion engine, an electric motor, a hybrid engine, or any power source used to supply power to construction equipment. The power from the power source 40 can be transmitted to various components and systems of the excavator 20 such as the ground engaging unit 24 or the excavation and dropping system 25.

[0019] The excavator 20 can be controlled by one or more controllers (hereinafter, "controller 42"). The controller 42 can include one or more processors, microprocessors, microcontrollers, electronic control modules (ECMs), electronic control units (ECUs), programmable logic controllers (PLCs), or any other suitable means for electronically controlling the functions of the excavator 20.

[0020] The haul truck 50 can be configured to transport and load and unload materials within or outside the work site 10. The haul truck 50 can include a frame 52, a dump body 54, a platform 56, and a ground-engaging unit 58.

[0021] The frame 52 can be configured to provide structural support and rigidity to the other components of the haul truck 50. For example, the dump body 54, the platform 56, and the ground-engaging unit 58 can all be attached to the frame 52. Thus, the frame 52 can be subjected to many stresses and strains while the haul truck 50 is being filled, while the haul truck 50 is operating around or outside the work site 10, and while the haul truck 50 is being loaded and unloaded.

[0022] The dump body 54 can be configured to receive materials from the excavator 20 or any other equipment capable of loading materials into the dump body 54. The dump body 54 can be attached to at least one haul body cylinder (not shown). The dump body 54 can be pivotally attached to the frame 52. The haul body cylinder can be attached to the frame 52 and the dump body 54 such that when the haul body cylinder extends, the dump body 54 can rotate around its connection to the frame 52 to tilt the dump body 54 upward, and when the haul body cylinder contracts, the dump body 54 can rotate around its connection to the frame 52 to level the dump body 54 with the frame 52.

[0023] The ground engagement unit 58 can be configured to move the transport truck 50 along the ground in the front - rear direction. As shown in FIG. 1, the ground engagement unit 58 can be a wheel such as an inflatable or rigid tire. In another embodiment, the ground engagement unit 58 can be a tracked assembly or crawler, or any other ground engagement device used to operate a construction vehicle.

[0024] The platform 56 can be configured to hold an operator of the transport truck 50 and a control (e.g., a human - machine interface) for controlling the transport truck 50. As shown in FIG. 1, the platform can extend in the longitudinal direction between and away from the ground engagement units 58. The platform 56 can include a power source 70 and a controller 72.

[0025] The power source 70 can be configured to supply power to various components of the transport truck 50. The power source 70 can be provided in any number of different forms, including, but not limited to, an internal combustion engine, an electric motor, a hybrid engine, or any power source used to supply power to construction equipment. Power from the power source 70 can be transmitted to various components and systems of the transport truck 50, such as the ground engagement unit 58 or the transport body cylinder 60.

[0026] The controller 72 can be configured to control various components of the transport truck 50. The controller 72 can include one or more processors, microprocessors, microcontrollers, electronic control modules (ECMs), electronic control units (ECUs), programmable logic controllers (PLCs), or any other suitable means for electronically controlling the functions of the transport truck 50.

[0027] As shown in FIG. 1, a system for measuring residues (hereinafter, "system 100") can be used at a work site 10 or other locations, such as between a loading position and a disposal position. System 100 can be configured to identify the type of dump bed, calculate the amount of residues in the dump bed, and update the accounting of the materials moved to the work site 10. System 100 can include at least one scanning device 102, such as a 3D scanner, a camera, or other device for capturing three-dimensional information regarding the interior of the dump body. As shown in FIG. 1, the scanning device 102 can be installed on the boom 26 of the excavator 20 such that the scanning device 102 can have the dump body 54 within the field of view 104. In one or more embodiments, the scanning device 102 can be installed on the stick member 30 of the excavator 20 such that the scanning device 102 can have the dump body 54 within the field of view 104. In yet another embodiment, the scanning device 102 can be installed anywhere else on the excavator 20 such that the scanning device 102 can have the dump body 54 within the field of view 104. The scanning device 102 and the field of view 104 will be described in more detail below in connection with FIG. 4.

[0028] FIG. 2 is a perspective view of the scanning device 102 of the system 100 that scans the dump body 54 of the transport truck 50. As shown in FIG. 2, the scanning device 102 can be mounted on the post 170 (for example, in addition to or instead of a scanning device installed on an excavator). In one or more embodiments, the post 170 may be tall enough such that when the transport truck 50 parks, passes under, or passes under or beside the scanning device 102, the scanning device 102 can have the dump body 54 of the transport truck 50 within the field of view 104. In one or more embodiments, the post 170 can be located at the work site 10. In another embodiment, the post 170 can be located outside the work site 10 at any location where the transport truck 50 can drive. For example, the post 170 can be located on the side of a road or other route of one or more transport trucks. In one or more embodiments, the post 170 is positioned at a set distance from the work site 10 such that the driver can have his or her truck scanned by the scanning device 102 and the system 100 can warn the work site that a transport truck (for example, the transport truck 50) is near (for example, far away or approaching) the work site.

[0029] FIG. 3 is a perspective view of the dump body 54 of the transport truck 50. The dump body 54 can include an inner surface 62, at least one side wall 64 (hereinafter, "side wall 64"), a floor 66, and a truck wall 68. As shown in FIG. 3, each of the side walls 64 can extend from opposite ends of the floor 66. The truck wall 68 can extend from the floor at the end closest to the transport truck (not shown). Thus, the truck wall 68 can be configured to extend away from the floor 66 until it reaches the height of the side wall 64 and then extend away from the side wall 64 to help protect the platform 56 of the transport truck 50. The inner surface 62 is defined by the side wall 64, the floor 66, and the truck wall 68. The inner surface 62 can define the volume V of the dump body 54.

[0030] The dump body 54 shown in FIG. 3 is merely an example of a dump body. However, the dump body is provided in any different shapes and sizes. In one or more embodiments, the dump body may have higher or lower sidewalls. In one or more embodiments, the dump body may have higher or lower track walls. In one or more embodiments, the dump body may have a longer or shorter floor. In one or more embodiments, the dump body can combine any combination of higher or lower sidewalls, higher or lower track walls, or longer or shorter floors. Therefore, as one or more of these dump body parameters change, the internal surface and volume of the dump body may also change. Accordingly, the system 100 (shown in FIGS. 1 and 2 and discussed in more detail below) can detect the dump body type and determine its volume V. Still further, although a dump body of a transport truck is contemplated and shown, residues can occur in a variety of other devices and systems.

[0031] In one or more embodiments, the system 100 can be configured to identify any type of track, define a set of criteria for determining the geometric parameters of the dump body, compare with one or more stored scans, and find residues within the dump body.

[0032] FIG. 4 is a schematic diagram of a system 100 used to detect residues within a dump body 54 (FIGS. 1-3) of a transport truck 50 (FIGS. 1-2). The system 100 can include a first controller 200, a second controller 208, and a computer 231. In one or more embodiments, the first controller 200, the second controller 208, and the computer 231 can be combined in various combinations. For example, the first controller 200 and the second controller 208 can be combined to interact with the computer 231. In another embodiment, the first controller 200 can be combined with the computer 231 to interact with the second controller 208. In yet another embodiment, the second controller 208 can be combined with the computer 231 to interact with the first controller 200.

[0033] The first controller 200 may include one or more processors, microprocessors, microcontrollers, electronic control modules (ECMs), electronic control units (ECUs), programmable logic controllers (PLCs), or any other suitable means for processing images or three-dimensional data captured by the scanning device 102 and communicated to the second controller 208 and the computer 231. The first controller 200 may include a storage medium or memory accessible by the controller 200, for example, in the form of a floppy disk, hard drive, optical medium, random access memory (RAM), read-only memory (ROM), or any other suitable computer-readable storage medium commonly used in the art (each referred to as a "database"), which may be in the form of a physical and non-transitory storage medium. As shown in FIGS. 1 and 2, the first controller 200 may be installed within the system 100. In one or more embodiments, the first controller 200 may be a controller mounted on the excavator 20 (e.g., controller 42 shown in FIG. 1). If the scanning device 102 is disposed on the post 170, the controller 200 may be present, for example, on the post 170, or may be present, for example, at a back-office location.

[0034] The first controller 200 can communicate electrically with one or more scanning devices 102. In one or more embodiments, the first controller 200 can communicate electrically with two scanning devices 102. The scanning device 102 can be configured to capture images (videos (i.e., continuous video images), still images (i.e., photos taken at a set frequency)) or three-dimensional data within the field of view 104 and retransmit those images and / or data to the first controller 200. In one or more embodiments, the scanning device 102 may be a stereo camera (i.e., two monochrome and one color camera module). In another embodiment, the scanning device 102 may be any other type of camera that can be used to detect residues within the dump body of a dump truck. In yet other embodiments, the scanning device 102 can include a three-dimensional scanner or other surface capture system.

[0035] Each scanning device 102 can define a field of view 104. To increase or decrease the field of view 104, the scanning device 102 may be adjusted (e.g., moved further away or closer to the object) to increase the overall field of view 104 of the system 100, or additional scanning devices 102 may be added. In yet more embodiments, the scanning device 102 can be adjusted to increase the field of view 104. For example, a lens (not shown) of the scanning device 102 can be added (or changed) to the scanning device 102 to increase the width of the field of view 104. There are many other modifications (such as zooming, focus adjustment, etc.) that can be made to the scanning device 102 to improve the field of view 104 and / or adjust the clarity, accuracy, and / or precision of the captured images or data.

[0036] In the embodiments shown in FIGS. 1 and 2, one scanning device 102 can be installed on the boom (e.g., boom 26) of an excavator (e.g., excavator 20). In another embodiment, one of the scanning devices 102 can be installed on the boom (e.g., boom 26), and the other of the scanning devices 102 can be installed on the stick member (e.g., stick member 30) of the excavator (e.g., excavator 20). In yet another embodiment, one or both of the scanning devices 102 can be installed on the boom and / or stick member of the excavator. In one or more embodiments, any number of scanning devices 102 can be added to the boom or stick member of the excavator to provide image capture quality and improve the field of view 104. Additionally or alternatively, one or more scanning devices 102 can be installed on one or more posts 170.

[0037] The first controller 200 may also include a vision processing electronic control module 202. The vision processing electronic control module 202 may be configured to receive images or data captured from any and / or all of the scanning devices 102 and process the captured images or data for transmission to the second controller 208 and computer 231. The vision processing electronic control module 202 can communicate with any of the scanning devices 102 of the system 100 to receive and aggregate images or data captured from the various scanning devices 102. The vision processing electronic control module 202 may also be electrically connected to the gateway electronic control module 204.

[0038] The gateway electronic control module 204 may enable communication of the aggregated captured images or data from the vision processing electronic control module 202 to the modem 206. Accordingly, the gateway electronic control module 204 may enable wireless communication of the aggregated captured images or data to either the second controller 208 and computer 231 via the modem 206.

[0039] The second controller 208 may include one or more processors, microprocessors, microcontrollers, electronic control modules (ECMs), electronic control units (ECUs), programmable logic controllers (PLCs), or any other suitable means for processing images or three-dimensional data communicated from the first controller 200 or computer 231. The second controller 208 may include a storage medium or memory accessible by the second controller 208 in the form of, for example, a floppy disk, hard drive, optical medium, random access memory (RAM), read-only memory (ROM), or any other suitable computer-readable storage medium commonly used in the art (each referred to as a "database"), which may be in the form of a physical non-transitory storage medium. The second controller 208 is configured to at least receive the captured image or data aggregated from the first controller 200, process the aggregated captured image or data, identify the type of dump body captured in the image, complete accounting based on the processed aggregated captured image or data, and communicate with the first controller 200 and computer 231. The second controller 208 can process the aggregated captured image from the first controller 200 during a known scan of a known dump body of a known haul truck. As described above, the second controller 208 may include a computer-readable storage medium. The computer-readable storage medium may store various information types including a baseline model repository 210, a track id list 212, a driver id list 214, a driver app location 216, an excavator ID 218, trigger threshold data 220, a scan comparator algorithm 222, a track type list 224, a three-dimensional computer-aided design model of the track type (hereinafter, "track type 3D CAD model 226"), an inventory management program 228. The second controller 208 may also include a modem 230.

[0040] The baseline model repository 210 can be configured to store known scans of known track body types. The aggregated scans of the known dump body and the haul truck from the second controller 208 are stored in the baseline model repository 210 and can be used as reference scans for other operations performed by the second controller 208. Further, the second controller 208 can update the scans stored in the baseline model repository 210 if the known dump body has a scan that results in a volume higher or lower than a previous baseline scan. Each scan stored in the baseline model repository 210 can be identified by the track identification pin 211 or the driver identification pin 213.

[0041] The track id list 212 can be configured to store information about one or more trucks operating at the worksite 10 (Figure 1). The track identification pin 211 of the truck can be stored in the track ID list 212. When a truck operates at the worksite 10, the system 100 can store all information (e.g., the location of the truck, the amount of material moved by the truck, the time the truck was at the worksite, or any other information related to the truck on the worksite) in the second controller 208.

[0042] The driver id list 214 can be configured to store information about one or more drivers operating at the worksite 10. The driver identification pin 213 of the driver can be stored in the driver ID list 214. When a driver operates at the worksite 10, the system 100 can store all information (e.g., the location of the driver, the amount of material moved by the driver, the time the driver was at the worksite, or any other information related to the one or more drivers on the worksite) on the second controller 208.

[0043] The operator app position 216 can be configured to store the position of the operator inside or outside the work site 10. In one or more embodiments, the position of the operator can be shared with the second controller 208 via the computer 231. In another embodiment, the position of the operator can be shared with the second controller 208 by one or more controllers on the transport truck or other work machinery (e.g., the controller 72 mounted on the transport truck 50). The second controller 208 can use the position of the operator to automatically warn the operator of other equipment (e.g., the operator of the excavator 20) when the transport truck is ready for loading or unloading.

[0044] The excavator ID 218 can be configured to store information regarding one or more excavators on the work site 10. The information stored in the excavator ID 218 can include the loading capacity (e.g., the size of the boom and bucket), and the position of the excavator.

[0045] The trigger threshold data 220 can be collected and stored for each type of dump body to send a signal to the second controller 208 when a residue is detected, when a new scan of the dump body should be performed, or when a new baseline should be stored in the baseline model repository 210. The trigger threshold data 220 can include various shapes of the dump body. For example, the trigger threshold data 220 can be the length of the floor or the height of the side wall of the dump body. In another embodiment, the trigger threshold data 220 can be the volume of the inner surface (e.g., the inner surface 62 of the dump body 54). In yet another embodiment, the trigger threshold data 220 can be any other parameter that can be detected by the system 100, and the system can signal that a residue has been detected or that the system 100 needs to be updated. In one or more embodiments, the trigger threshold data 220 includes the surface profile of the inner surface of the dump body.

[0046] The trigger threshold data 220 may be set for any data collected by the dump body of the transport truck. In some embodiments, the trigger threshold data 220 may account for the noise and / or inaccuracies of the measurement system. Thus, the trigger threshold data 220 may be set to a value that exceeds the known variations of the system. For example, if the system has a known error of five percent, the trigger threshold data 220 may be set to any measured value that exceeds five percent. In one or more embodiments, the system 100 may use the flatness score of the inner surface of the dump body. The system 100 may analyze a previous scan or model of the truck body to define a boundary of a portion of the planar shape. In one or more embodiments, a neural network, or machine learning system, may be used to automatically detect and define a boundary of a portion of the planar shape on the dump body of the dump truck. The system 100 may obtain a known image of the empty dump body and a scan captured by the system, and generate a point cloud map of the saved image and the scan captured by the system. Each defined region of the point cloud map may be modified such that the plane is aligned with the XY plane. Thus, the altitude difference between the highest point and the lowest point of the point cloud map of the known image of the empty dump body and the scan captured by the system may be the flatness score of the XY plane. Thus, the trigger threshold data 220 may be configured to reduce false alarms for detecting residues and / or determining that a new baseline scan is required.

[0047] The truck type list 224 can be configured to store a list of known truck types (e.g., dump bodies having a recognized internal surface shape) stored in the second controller 208. The truck type list 224 can be transmitted to another controller (e.g., the first controller 200, the computer 231, or any other controller in wireless communication with the system 100) to share the stored truck types. For example, if the system 100 scans a dump body and does not recognize the truck type, the system 100 can prompt the driver to select the dump body type from the truck type list 224.

[0048] The truck type 3D CAD model 226 can be stored in the second controller 208 as a model of a known dump body type. In one or more embodiments, the truck type 3D CAD model 226 can include truck types from known manufacturers or their partners. In one or more embodiments, a 3D model of a new truck can be created. In another embodiment, a new truck that is not a known truck type can be 3D modeled and added to the truck type 3D CAD model 226. The truck type 3D CAD model 226 can provide accurate, substantially accurate, and / or relatively accurate dimensions of the dump body of the transport truck for each 3D CAD model. In one or more embodiments, the accurate dimensions can be accurate at a level the same as or similar to the manufacturing tolerances of the CAD model or manufacturing drawing, for example. The dimensions of the 3D CAD models stored in the truck type 3D CAD model 226 can be used by the scan comparator algorithm 222.

[0049] The scan comparator algorithm 222 can be executed to compare a scan captured from a first controller (e.g., the first controller 200) with known scans stored in a database (e.g., the baseline model repository 210 or the track type 3D CAD model 226). The scan comparator algorithm 222 can include a neural network for continuously comparing a new scan from the first controller with known scans previously stored in the system 100. The neural network may be a statistical analysis algorithm that compares the frequencies of different object primitives to define input and / or output correlations. Thus, the system 100 can capture images of different track types and classify the captured images by track boundaries, images, and labels of various track types. As the neural network receives and analyzes more tracks, the neural network can generate a weight matrix. For example, the neural network may generate coefficients in a large-scale n-dimensional curve fitting. Then, the dimensional curve fitting of a known image or scan may be compared with unlabeled data to determine whether the target accuracy of the neural network has been achieved. In an embodiment, the neural network can be configured for a specific task. For example, the neural network may be configured for image recognition. A neural network configured for image recognition may be able to analyze a custom dataset (e.g., labeled photographs of a machine or a dump body) and may be able to perform a specific task (e.g., identification of the type of machine or dump body in the photograph, outlining a part of an image showing the dump body of a dump truck).

[0050] In another embodiment, the scan comparator algorithm 222 can compare a new scan to a computer-aided design (CAD) model of a known dump body. If the known scan or known 3D CAD model matches the truck type, the scan comparator algorithm 222 can determine the volume (e.g., volume V in FIG. 3) difference between the scanned dump body and the scan of the known dump body or known 3D CAD model. If no CAD model is available, the scan comparator algorithm 222 may include a bilateral comparator secondary test and a flatness comparator secondary test.

[0051] The bilateral comparator secondary test of the scan comparator algorithm 222 can use a neural network to divide the scanned area in half along the length of the floor of the truck body and compare each half of the divided scan area to the other half of the divided scan area. In some embodiments, as described above, the neural network can analyze the captured image of the dump body and determine the area of the captured image corresponding to the dump body of the transport truck. The area corresponding to the dump body of the transport truck can be extracted from the disparity map as a 3D point cloud and corrected to the XY plane. Next, the neural network can divide the 3D point cloud along the center by the y-axis and compare the height of each point in one half to the height of the corresponding point from the other half. If the difference between the divided scanned areas exceeds the threshold stored in the trigger threshold data 220, the second controller 208 can identify that the dump body contains residue and / or set a scraping flag for the dump body.

[0052] The flatness comparator secondary test of the scan comparator algorithm 222 can identify the geometric features (e.g., corners, planes, or intersections) of the dump body and calculate the flatness score for each planar feature (e.g., floor 66, side wall 64, or truck wall 68). In one or more embodiments, the system 100 can analyze the scan for the flatness score using a neural network. For example, the neural network can acquire the captured image and define the planes on the interior of the dump body. Next, the neural network can create a 3D cloud map on the plane and modify the 3D cloud map on the XY axis. Next, the neural network can compare the highest and lowest height values to generate a flatness score for each of the planes inside the dump body of the haul truck. If the flatness score of any of the planar features of the dump body is greater than the maximum threshold stored in the trigger threshold data 220 or less than the negative threshold, the second controller 208 can identify that the dump body contains residues and / or flag the dump body for scraping.

[0053] Accordingly, by comparing two scans, comparing a scan with a 3D CAD model, a bilateral comparator secondary test, or a flatness comparator secondary test, the scan comparator algorithm 222 can predict the presence of residues within the dump body of the dump truck.

[0054] The inventory management program 228 can be configured to receive inputs from various systems of the system 100. For example, the inventory management program 228 can use the baseline model repository 210, the track id list 212, the driver id list 214, the driver app location 216, and the excavator ID 218 to track the location of equipment on the job site 10 and calculate the theoretical amount of materials moved by the machine. The inventory management program 228 can also use the trigger threshold data 220, the scan comparator algorithm 222, the track type list 224, and the track type 3D CAD model 226 to automatically remove the calculated volume of residues from the theoretical amount of materials moved by the machine in order to more accurately grasp the materials on the job site 10. In one or more embodiments, the inventory management system 228 can also warn the excavator (e.g., excavator 20) that the residue is in the transport truck (e.g., transport truck 50), preventing the excavator from overloading the transport truck. Thus, the inventory management system 228 can help ensure that the transport truck is not overloaded (e.g., within the weight limit of the transportation department) before leaving the job site 10.

[0055] The modem 230 can be configured to enable the second controller 208 to communicate wirelessly with the first controller 200 and the computer 231. The modem 230 can include a gateway control module that enables sharing of aggregated information between the second controller 208 of the system 100 and other controllers or computers.

[0056] Computer 231 can be configured to assist the truck driver in communicating with other components of system 100. Computer 231 can be a mobile phone, tablet, laptop, or any other type of computer that can be located within the truck or at the work site 10 and is accessible to the driver for communicating with other components of system 100. For example, the driver can communicate with the second controller 208 to provide a driver ID, a truck ID, and the location of the truck being driven. For example, computer 231 can include a driver ID 232, a truck ID 234, a location 236, and a modem 238.

[0057] Driver ID 232 can be configured to communicate the driver's unique identification to the second controller 208 so that the second controller 208 can download and upload information regarding the driver. Driver ID 232 can be assigned to the driver before the driver first enters the work site (e.g., work site 10) or when the driver first enters the work site.

[0058] Truck ID 234 can be configured to communicate the unique identification of the truck to the second controller 208 so that the second controller 208 can download and upload information regarding the truck. Since the driver can drive multiple trucks, truck ID 234 can also be tracked. Truck ID 234 can be assigned to the truck either before the truck first arrives at the work site or after the truck has operated at the work site.

[0059] Location 236 can be the location of the computer at any given time. Location 236 can be measured by a global positioning sensor or some other location device. Location 236 can communicate the location of computer 231 regardless of whether computer 231 is at the work site or outside the work site.

[0060] The modem 238 can be configured to enable the computer 231 to wirelessly communicate with the first controller 200 and the second controller 208. The modem 238 may include a gateway control module that enables sharing of aggregated information between the computer 231 of the system 100 and other controllers or computers.

[0061] In one or more embodiments, the first controller 200 and the second controller 208 can communicate work information to the computer 231. For example, the second controller 208 can communicate the amount of toll goods loaded onto the transport truck 50 and the time taken to load the transport truck 50. In another embodiment, the second controller 208 can communicate the total amount of toll goods moved by the dump truck 50 at the work site 10. In yet another embodiment, all of the information of the first controller 200 and the second controller 208 can be shared with the computer 231.

Industrial Applicability

[0062] In one or more operational embodiments of the disclosed system, the system 100 may include a track onboarding program 500 and a residue detection program 600.

[0063] FIG. 5 is a flowchart showing a track onboarding program 500. When a new track (e.g., haul truck 50) enters a work site (e.g., work site 10), system 100 can execute the track onboarding program 500. The track onboarding program 500 is configured to collect information and communicate it to one or more controllers so that the information can be stored and used in various calculations while the track is at the work site. In one embodiment, the onboarding program 500 can be installed on a back office controller (e.g., second controller 208). In another embodiment, the onboarding program 500 can be installed on a machine (e.g., a controller on an excavator, a truck, or any other machine on the work site).

[0064] In step 502, the program can generate a new track ID for the new track at the work site. The new track ID can be a unique track ID that is associated with the track for the life of the track or while the track is present at the work site. The new track ID can be appended to any additional information collected by the track onboarding program 500.

[0065] In step 504, the program can start a track scan. In one embodiment, a controller (e.g., second controller 208) can wirelessly communicate with another controller (e.g., first controller 200) to capture a scan of the dump body of the track. As described above, the scan of the dump body can be performed by a controller attached to an excavator. In another embodiment, the scan of the dump body can be performed by a controller attached to a post (e.g., post 170). In yet another embodiment, the scan of the dump body can be performed by a controller remote from the work site.

[0066] In step 506, the controller (e.g., the first controller 200) can scan the dump body of the truck, and the controller can aggregate the scan and wirelessly transmit the scan to the back office. The back office can save this scan in a database (e.g., the baseline model repository 210) for reference in all future scans. Once saved in the baseline model repository 210, the baseline scan is linked to the haul truck and the truck identification pin (e.g., the truck identification pin 211) and the driver identification pin (e.g., the driver identification pin 213) of the driver.

[0067] In step 508, the back office can compare the scan received from the controller with the scan of the known truck bed saved in the repository (e.g., the baseline model repository 210) to detect the truck type of the truck scanned by the controller.

[0068] In step 510, the controller (e.g., the second controller 208) can compare the received scan of the dump body of the haul truck with the 3D CAD model (e.g., the truck type 3D CAD model 226) saved in the controller. If a 3D CAD model exists, the controller can execute a comparator function (e.g., the scan comparator algorithm 222) to compare the scan of the dump body with the 3D CAD model of the dump body. If there is no 3D CAD model of the dump body type, the controller can use one or more secondary threshold tests.

[0069] In step 512, the controller can perform a secondary threshold test (e.g., the bilateral comparators and / or flatness comparator of the scan comparator algorithm 222). The controller may include a prompt to allow the operator, on-site supervisor, or engineer to select the bilateral comparator or flatness comparator secondary test. In another embodiment, the controller can perform both the bilateral comparator and flatness comparator secondary tests, combine, and / or compare the results. In yet another embodiment, a neural network installed in the controller (e.g., the second controller 208) can scan the dump body to find similar dump bodies and determine whether to perform the bilateral comparator and / or flatness comparator secondary tests.

[0070] In step 514, the controller can perform a bilateral comparator secondary test by splitting the scan in half along the length of the floor of the dump body and comparing each half of the split scan with the other half of the split scan. The bilateral comparator secondary test can then determine whether any residue is present by detecting the difference between the halves of the scan.

[0071] In step 516, the controller can perform a flatness comparator secondary test by identifying the geometric features (e.g., corners, planes, or intersections) of the dump body and calculate the flatness score for each planar feature (e.g., the floor, side walls, or track walls) of the dump body. The lack of flatness detected by the flatness comparator secondary test may indicate the presence of residue within the dump body.

[0072] In step 518, when the comparator function outputs a volume number exceeding the threshold value, the controller can send a signal indicating that it is necessary to scrape off the residue in the dump body to a computer or controller on the transport truck (for example, computer 231) or the excavator (for example, the first controller 200). In one embodiment, the controller can compare a known scan (or 3D CAD model) with a new scan, and the comparator can output a positive value exceeding the positive threshold value to indicate the presence of residue. In another embodiment, the controller can compare a new scan with a known scan (or 3D CAD model), and the comparator can output a negative value below the threshold value to indicate the presence of residue.

[0073] In step 520, when the comparator function outputs a positive or negative value exceeding or falling below the threshold value, the controller can send a signal indicating that the dump body needs to recapture the baseline scan to a computer or controller on the transport truck (for example, computer 231) or the excavator (for example, the first controller 200). For example, when the comparator function compares a known scan with a new scan and the new scan appears to have a larger volume than the known scan, the controller can send a signal indicating that the dump body needs a new baseline scan to a computer or controller on the transport truck or the excavator.

[0074] FIG. 6 is a flowchart showing a residue detection program 600. The residue detection program can be configured to detect residues in a known or unknown dump body (for example, dump body 54) of a transport truck (for example, transport truck 50).

[0075] In step 602, the computer (e.g., computer 231) can send the truck ID and the driver ID (e.g., the truck identification pin 211 and the driver identification pin 213) to the back office (e.g., the second controller 208). The computer can send the truck ID and the driver ID before the transport truck enters the work site or after the transport truck enters the work site (e.g., work site 10).

[0076] In step 604, the computer can send the truck position to the back office. The position of the truck can be used to track the position of the truck inside and outside the work site. For example, the position of the truck can be determined when the truck is at a set distance from the work site. In another embodiment, the back office can determine whether the truck is at (or approaching) the position for loading or unloading by the excavator. The position of the truck can be stored in a database on the controller (e.g., the driver app position 216).

[0077] In step 606, the back office can compare the position of the truck with the position fence of the excavator. By instructing the excavator of the distance away from the excavator fence, the excavator can be informed of the time until the transport truck needs to be loaded or unloaded. Thereby, an estimate of how long the excavator can continue to work on the current task until it needs to go to load or unload the transport truck can be given to the excavator.

[0078] In step 608, the back office can warn the excavator when the transport truck arrives. The excavator position fence can be set by the back office to automatically notify the excavator that the truck needs to be loaded or unloaded when the truck crosses the fence. By warning the excavator, it becomes possible to drive the excavator to the position before the truck arrives to maximize the efficiency of the work site.

[0079] In step 610, the back office can warn a computer (e.g., computer 231) and a controller (e.g., the second controller 208) that the truck does not have a baseline scan, and the controller can capture the baseline scan and send it to the back office. Alternatively or additionally, the baseline scan can be captured, for example, while the truck is moving between the filling position and the dumping position.

[0080] In step 612, the excavator enables filling of a new truck. The filling operation of the new truck can record data related to filling the truck (e.g., filling time, filling volume (based on a measurement system attached to the excavator), filling location, filling pattern, or any other data that may be useful for knowing how the haul truck is loaded by the excavator).

[0081] In step 614, a neural network provided in one of the controllers (e.g., the first controller 200 or the second controller 208) continuously analyzes the scan received from the scanning device 102 to determine the position of the dump body of the haul truck. In one or more embodiments, the truck or the excavator may include a QR code or an April tag to assist the system 100 in identifying the truck type.

[0082] In step 616, the scanning device 102 on the boom of the excavator (e.g., excavator 20) continuously scans the dump body when the excavator loads materials into the dump body.

[0083] In step 618, the controller (e.g., the first controller 200) can send the scan of the dump body (e.g., the scan of the dump body or the scan of the loading operation from the excavator) to the back office (e.g., the second controller 208).

[0084] In step 620, the back office can compare the scan of the dump body (e.g., the scan of the dump body or the scan of the loading operation from the excavator) with the saved baseline scan, the 3D CAD model of the known dump body type, the bilateral comparator, or the flatness comparator to compare the scans and obtain the volume difference value.

[0085] In step 622, if any of the comparison functions outputs a value that exceeds or falls below the threshold, the back office can send an alarm to the dump truck and the excavator to scrape off the residue in the dump body. For example, the comparator can output a positive value indicating the residue in the dump body of the transport truck (e.g., when the volume of the scan of the dump body is subtracted from the volume of the known scan or 3D model). In another embodiment, the comparator can output a negative value indicating the residue in the dump body of the transport truck (e.g., when the volume of the known scan or 3D model is subtracted from the volume of the scan of the dump body).

[0086] In step 624, if any of the comparison functions outputs a value that exceeds or falls below the threshold, the back office can send a recapture baseline scan message to the dump truck and the excavator. For example, the comparator can output a positive value indicating the need for a new baseline scan of the dump body of the transport truck (e.g., when the volume of the known scan or 3D model is subtracted from the volume of the scan of the dump body). In another embodiment, the comparator can output a negative value indicating the need for a new baseline scan of the dump body of the transport truck (e.g., when the volume of the scan of the dump body is subtracted from the volume of the known scan or 3D model).

[0087] The above detailed description is intended to be illustrative and not limiting. Accordingly, the scope of the present disclosure should be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled.

Claims

1. A system for detecting residues within the dump body of a haul truck, comprising: a scanning device configured to generate a scan of the inner surface of the dump body; and a first controller configured to: receive the scan of the inner surface of the dump body; and determine the type of dump body by comparing the scan of the inner surface of the dump body with a plurality of scans of the inner surfaces of known dump bodies.

2. The system of claim 1, further comprising an excavator configured to lift and dump materials into the dump body, the excavator comprising a boom member configured to extend at least a portion of the boom member over the dump body.

3. The first controller is configured to: calculate a volume difference between the scan of the inner surface of a known dump body and the scan of the inner surface of the dump body; and warn an operator of the excavator of the amount of residue present within the dump body of the haul truck, enabling the operator to avoid overloading the dump body of the haul truck.

4. The first controller is mounted on the excavator, and the system further comprises a second controller located at a work site, wherein both the first controller and the second controller are configured to communicate wirelessly with each other.

5. The first controller is mounted on the excavator, and the system further comprises a second controller located remotely, wherein both the first controller and the second controller are configured to communicate wirelessly with each other.

6.

7. The system of claim 4, further comprising a computer configured to receive information from an operator of the haul truck, the computer being configured to collect: an operator identification pin; a truck identification pin; and the position of the haul truck via a global positioning sensor.

7. ​ The system according to claim 6, wherein the computer is a mobile phone having an application for the driver to input the driver identification pin and the truck identification pin.

8. The second controller A baseline model repository configured to store the scan of the inner surface of a known dump body, A truck identification pin list including all known transport trucks that have entered the work site, A driver identification pin list including all known drivers that have entered the work site, The position of the driver at the work site, which tracks the positions of all trucks that were at the work site, A system according to claim 6, comprising: an excavator identification pin list including all excavators at the work site.

9. The second controller Receives from the computer the driver identification pin, the truck identification pin, and the position of the transport truck, Determines the dump body type of the transport truck with the truck identification pin by referring to the truck identification pin list, Obtains the scan of the dump body type related to the truck identification pin from the baseline model repository, Receives from the first controller the scan of the inner surface of the dump body, Calculates the residue using at least one scan comparison algorithm to compare the scan of the dump body type related to the truck identification pin with the scan of the inner surface of the dump body for transporting materials, and The system according to claim 8, configured to transmit an alarm to the computer when the residue is calculated.

10. The second controller includes an inventory management system, and the inventory management system Calculates an estimated amount of materials that can be loaded onto the dump body of the transport truck, Subtracts the amount of the calculated residue to find the actual amount of materials moved, and The system according to claim 9, configured to store the actual amount of materials moved to update the accounting at the work site.

11. The first controller comprises a new track program which, when executed, assigns a track identification pin unique to the new track at the work site, communicates the unique track identification pin to the second controller, which stores the unique track identification pin in the track identification pin list, sends a signal to the scanning device to capture a scan of the new track at the work site, and communicates the scan of the new track at the work site to the second controller, which stores the scan in the baseline model repository. The system according to claim 8. **Claim 12** The second controller comprises a new track prompt program which, when executed, sends an alert to the computer to execute the new track program when the track identification pin cannot be found from the first controller in the track identification pin list. The system according to claim 11. **Claim 13** The second controller comprises a symmetric scan comparator algorithm configured to detect residues in the dump body of a dump truck that cannot be identified by the system. When executed, the symmetric scan comparator algorithm sends a signal to the scanning device to capture a scan of the dump body, analyzes the captured scan to determine the area corresponding to the dump body of the dump truck, extracts a disparity map from the captured scan as a three-dimensional point cloud, modifies the three-dimensional point cloud from the captured scan on the XY plane, divides the captured scan in half at the midpoint of the X plane to generate a first half of points and a second half of points, compares the height of each point in the first half of points with the corresponding points in the second half of points to generate a symmetric score, and compiles the symmetric scores and compares them with a threshold number, and when the symmetric comparator algorithm determines that the symmetric score exceeds the threshold, warns the operator that there may be residues in the dump body. The system according to claim 8. **Claim 14** The second controller It is equipped with a flatness scan comparator algorithm configured to detect residues in the dump body of a dump truck that cannot be identified by the said system. Implementing the said flatness scan comparator algorithm is to send a signal to the scanning device to capture the scan of the dump body, analyze the captured scan to determine the area corresponding to the dump body of the dump truck, extract a disparity map as a three-dimensional point cloud from the captured scan, correct the three-dimensional point cloud from the captured scan onto the XY plane, compare the highest point and the lowest point on the plane to calculate a flatness score, and compile the said flatness scores and compare them with a threshold number, When any of the said flatness scan comparator algorithms determines that the flatness score exceeds the threshold, warn the operator that there may be residues in the dump body. The system according to claim 8.

Images